JEV-as-a-Judge measures a typed decision service that returns a verdict and label probabilities. Across 5,172 base judgments and 17 configurations, JEV is a cheap first pass with clear capability limits. A frozen JEV → GPT-6 policy reaches 93.4% accuracy at 41.4% of GPT-6’s fee on 1,610 held-out pairs. On 570 pairs from new workloads, locally selected thresholds match GPT-6’s accuracy while saving 24.5% of its fee.
Updated to the 28 September 2026 author manuscript; first posted to arXiv on 22 September. The single-judge timing panel, frozen offline policy, live replication, and prospective test are separate experiments. Fees are estimates from reported token usage at collection-time prices, not invoices; configurations differ in model size and reasoning effort and are not compute-matched.
The LLM judges share a decision-only contract: a discrete judgment plus label probabilities, without a rationale. The study retains failures, distinguishes supplied labels from targeted human adjudication, and reports source-cluster bootstrap intervals.
JEV ranks 8th of 15 LLM judges on the 4,850 public-task judgments, 1.7 points behind GPT-6 and 2.8 behind GPT-5.6. It matches GPT-6 on RewardBench (92.5%) and trails it on HaluEval (87.3% vs 88.4%). On a separate matched timing panel, JEV costs $0.044 per 1,000 judgments and returns in 0.152 seconds at the median.
On JudgeBench, JEV scores 78.6% against GPT-6’s 93.1%, a 14.6-point gap concentrated in reasoning and coding. On 500 RM-Bench prompts, making the rejected answer more elaborate reduces JEV’s accuracy by 8.8 points relative to matched styles; GPT-6 drops 1.7 points. Fast, valid output does not ensure a correct decision.
On 200 balanced responses without evidence, JEV, GPT-4.1 mini, and GPT-5.4 agree with only 53.5–56.0% of existing hallucination labels, despite mean maximum probabilities of 0.91–0.96. On a separate 400-example document-grounded summary task, agreement is 69.0–69.8%. Confidence must be validated for the actual task.
The frozen offline policy saves 58.6% of GPT-6’s fee on 1,610 held-out preference pairs. On two new correctness workloads, 100 selection labels per workload set stricter thresholds: the live cascade matches GPT-6’s 90.2% accuracy on 570 held-out pairs while escalating 74.2%. Savings depend on how much the first stage can safely accept.
Seventeen configurations, base-order accuracy, invalid outputs counted as errors. The public tasks now include 1,500 RewardBench pairs, 350 JudgeBench pairs, and 3,000 HaluEval decisions. Hosted reasoning models use low effort (Qwen3.6 at its default); local Qwen uses non-thinking inference. These settings are not compute-matched. Switch tasks to compare the rankings.
Percentage points, 95% source-cluster bootstrap interval, 2,000 resamples.
On supplied labels, RewardBench is tied (95% CI −1.4 to +1.4 points), HaluEval favors GPT-6 by 1.1 points, and JudgeBench favors it by 14.6 points. No single ranking captures all three workloads.
Accuracy by domain, JEV vs GPT-6 (n per domain).
Reasoning and coding drive the gap; knowledge differs least. Solving difficult comparisons is a distinct capability from ordinary preference judging.
The targeted blinded audit covers 183 items from the original 990-item public-task subset: 163 where either judge disagreed with the supplied label, plus 20 controls. One human annotation, an LLM flagging pass, and author adjudication give a label-sensitivity analysis; this is not an independent multi-annotator consensus over the expanded evaluation. The human-adjudicated JEV − GPT-6 differences, normalized over the original 400 RewardBench, 240 HaluEval, and 350 JudgeBench items, are −3.0, −2.5, and −16.0 points, respectively.
The appendix evaluates Laya English and Typed-decisions as released, without fine-tuning; they are independent models, not JEV’s weights. Their RewardBench / JudgeBench / HaluEval accuracies are 47.3 / 48.9 / 68.9% and 49.6 / 49.1 / 71.3%. Valid outputs and fast local inference do not by themselves reproduce JEV’s accuracy. Context-complete subsets and confidence diagnostics are reported in the paper.
A separate frozen 120-decision panel, one model group at a time, eight workers, 0.12-second minimum starts. Outcome latency includes network, provider queueing, and retries after the first attempt. Fees use reported usage at collection-time prices; missing usage keeps a conservative reservation.
Median (dot) and p95 (bar end), seconds, log scale.
Reported usage (dot); lighter extension = conservative reservation where usage was missing. USD, log scale.
The roughly 9-fold and 277-fold fee differences describe this workload and these services. Different model capacities, reasoning budgets, caches, and providers prevent attributing the gap solely to a non-generative architecture.
Follow-up samples probe harder selection and presentation effects: 100 RewardBench 2 prompts with four-way top-1 selection, and 500 RM-Bench prompts with all nine style pairings in both orders (9,000 judgments per judge). More elaborate writing can distract a judge from which answer is correct.
Accuracy (%), 3,000 judgments per condition. Hard: the less elaborate answer is the preferred one.
Hard-minus-normal change: JEV −8.8 points [−10.6, −7.1], GPT-6 −1.7 [−3.2, −0.3]. On RewardBench 2 four-way selection, JEV / GPT-6 / Skywork score 73.0 / 75.0 / 79.0%; the 100-prompt sample leaves wide uncertainty.
Rows: preferred answer's style. Columns: rejected answer's style. Each cell: 1,000 judgments.
Above the diagonal the rejected answer is more elaborate. JEV’s top-right cell (concise preferred vs. Markdown rejected) scores 72.1%, compared with 90.2% for GPT-6. The full matrix shows how content and presentation interact.
The paper uses the maximum label probability q for every judge. JEV’s separate native confidence field correlates with it (Spearman 0.958–0.999 across the public tasks), but they are distinct quantities. For pairwise routing, probabilities are aligned to response identity and averaged across both presentation orders before computing q.
Bar: label agreement (%). Tick: mean maximum probability. 200 balanced general responses, no evidence supplied.
JEV’s Brier score is 0.821 and error-detection AUROC is 0.498; GPT-5.4’s Brier score is 0.818. On 400 document-grounded summaries, label agreement is 69.8 / 69.0 / 69.3% for JEV / GPT-4.1 mini / GPT-5.4.
Brier score on supplied labels (lower is better), valid outputs only. Error-detection AUROC below.
On the full HaluEval sample, JEV has a lower supplied-label Brier score (0.196 vs 0.213). But replacing 28 labels in the original 240-item audit subset reverses that subset’s ranking: JEV 0.062 vs GPT-6 0.032. This sensitivity analysis does not relabel all 3,000 decisions.
One temperature per task, fitted on the pilot selection set (64 / 32 / 32 judgments), evaluated on the disjoint extension. NLL, lower is better.
The fitted temperatures differ seven-fold across tasks. Task-specific validation beats a universal temperature.
Averaging both orders gives a +0.76-point accuracy change [0.00, 1.57] on the public preference pairs. The diagnostics show that most JEV order flips already occur below high confidence. Four-way rotations keep the same choice on 87/100 prompts; averaging their probabilities does not improve top-1 accuracy.
JEV judges each preference pair in both orders and averages the aligned probabilities. If its maximum probability reaches the threshold, its verdict stands; otherwise GPT-6 judges the pair independently. A threshold of 0.9, selected on 96 pilot pairs and frozen before evaluation, was tested on 1,610 held-out pairs. These results are offline simulations from retained outputs; the live tests below measure the complete sequence.
The pool is 83% RewardBench. Savings and accuracy changes differ by workload, so the split matters more than the pooled headline.
| Held-out workload | JEV alone | Cascade | GPT-6 | Δ (pp) [95% CI] | Escalated | Fee ratio reported / upper |
|---|---|---|---|---|---|---|
| RewardBench · 1,340 pairs | 92.8% | 93.7% | 92.4% | +1.27 [0.45, 2.03] | 24.8% | 0.267 / 0.308 |
| JudgeBench · 270 pairs | 81.3% | 92.2% | 93.0% | −0.74 [−2.22, 0.74] | 64.8% | 0.668 / 0.668 |
| All · 1,610 pairs | 90.9% | 93.4% | 92.5% | +0.93 [0.24, 1.66] | 31.5% | 0.414 / 0.440 |
JEV-alone accuracy uses the two-order average; GPT-6 uses one call. Fees are relative to GPT-6 on the same pairs. On JudgeBench, escalation closes 94% of JEV's accuracy gap while saving about one third of the fee.
A retrospective comparison applied the same selection rule to seven other inexpensive or open-weight first stages. GPT-4.1 mini stayed within one point of GPT-6 on both benchmarks, but its cascades used 62.2% and 98.3% of GPT-6's fee on RewardBench and JudgeBench, versus JEV's 26.7% and 66.8%. GPT-OSS 120B, Gemini 3 Flash, and Skywork saved fees but lost 7.41, 4.44, and 6.67 points on JudgeBench. Local-model fee ratios exclude GPU costs; this comparison does not establish a global cost–accuracy optimum.
The fallback also changes the threshold. With GPT-5.6 Sol, the frozen point rule selected τ = 0.7: it gained 0.82 points on RewardBench but lost 4.81 on JudgeBench. A two-point tolerance on a small selection set is not a held-out guarantee.
The τ = 0.9 policy was rerun on 510 previously analyzed held-out pairs: 240 RewardBench and 270 JudgeBench. Its live gate agreed with the offline gate on 98.6% of pairs.
| Live replication · 510 pairs | Cascade | GPT-6 alone |
|---|---|---|
| Accuracy | 92.5% | 93.1% |
| Fee relative to GPT-6 | 57.2% | 100% |
| Median latency | 0.27 s | 2.10 s |
| p95 latency | 4.26 s | 4.49 s |
The live accuracy difference is −0.59 points [−1.95, 0.59], with 46.5% escalated. Accepted pairs return in 0.18 s at the median; escalated pairs take 2.50 s. This replication confirms the offline estimates on familiar items; it does not test generality to a new workload.
Across 4,850 public items, the 3,744 with JEV confidence at least 0.9 give nearly tied accuracy: 94.7% for JEV and 94.4% for GPT-6. On the remaining 1,106 items, GPT-6 leads 75.0% to 67.7%.
This is a descriptive analysis of the full public sample. Confidence generally tracks correctness, but its signal weakens on style-adversarial RM-Bench pairs: error-detection AUROC falls from 0.839 on matched styles to 0.764 on hard pairs. On reference-free prose it is 0.498, and confidence routing does not help.
Order instability mostly falls below the gate: only 3 of JEV's 95 reversal flips have base-order confidence ≥ 0.9. Averaging both orders is a robustness step; its measured accuracy gain is small and not statistically significant.
Before any model call, the study froze the protocol, split, and threshold-selection rule for PPE correctness and JudgeBench's Claude response split. Each workload used 100 local selection pairs; the cascade then ran live on the remaining 400 and 170 pairs. Here JEV alone trails GPT-6 by 14 points, so maintaining accuracy requires more escalation.
| Held-out workload | Frozen τ | JEV alone | Cascade / GPT-6 | Escalated | Fee ratio |
|---|---|---|---|---|---|
| PPE correctness · 400 pairs | 0.95 | 78.2% | 88.2% / 88.2% | 67.8% | 0.693 |
| JudgeBench, Claude split · 170 pairs | 0.99 | 70.9% | 94.7% / 94.7% | 89.4% | 0.910 |
| Both workloads · 570 pairs | Per workload | 76.1% | 90.2% / 90.2% | 74.2% | 0.755 |
PPE covers MMLU-Pro, MATH, GPQA, MBPP-Plus, and IFEval. JudgeBench's candidate responses were new to the study, but some questions were not: its held-out split contains 57 seen and 113 unseen questions. Cascade parity holds on both subsets.
With most pairs escalated, median latency is nearly unchanged: 1.87 s for the cascade versus 1.91 s for GPT-6; p95 is 4.95 versus 5.58 s. Exact accuracy parity on these samples is an observed result, not a guarantee for future workloads.
The prospective test selected τ = 0.95 for PPE and 0.99 for JudgeBench. A point-estimate rule would have selected 0.9 for both. Thresholds depend on the workload and fallback; 0.9 is not a universal setting.
In an exploratory simulation with 2,000 draws and 96 selection labels in total, a shared threshold chosen by the point rule lost more than two accuracy points in 44.0% of draws. The lower-bound rule reduced that share to 5.2%; separate thresholds for the two workloads reduced it to 0.6%.
Those 96 labels are drawn across RewardBench and JudgeBench, not 96 per workload. The simulation repeatedly reuses one benchmark pool and does not model distribution shift. It motivates local validation; it is not a formal risk guarantee.
For each labeled pair, let d be cascade correctness minus fallback correctness. A threshold qualifies when mean(d) − 1.645 × SD(d) / √n ≥ −0.02. Among qualifying thresholds, choose the one accepting the most pairs, with ties favoring the stricter threshold.
Primary accuracy counts every failed call as an error. Provider-side structured-generation failures, HTTP-200 contract violations, and transport failures are separated so that service availability is not mistaken for reasoning quality.
| Hosted judge | Valid / 5,172 | Generation | Contract | Transport | Breakdown | JudgeBench acc. (all / valid only) |
|---|
JEV returns valid outputs on all 5,172 base judgments. Qwen3.6 at default effort scores 87.5% on valid JudgeBench outputs, but has 174 failed outcomes across the base evaluation: 24 generation, 117 contract, and 33 transport failures. Valid-only accuracy answers a different question from end-to-end accuracy.
The paper documents frozen requests, retained failures and probabilities, separate timing measurements, and pre-specified routing protocols. Its appendix describes an offline supplementary package for the principal analyses; historical calibration, repeat/paraphrase, interface, and rubric summaries are retained tables rather than fully regenerated pipelines. Prompts, schemas, and an exact JEV request/response are included in the paper.
Scope: one proprietary JEV version, heterogeneous model settings, and supplied benchmark labels supplemented by a targeted blinded human audit. API fees are estimates from usage, not invoices; local GPU costs are excluded where stated. The latest author manuscript is dated 28 September 2026. The primary paper link opens arXiv and its public version history.
@misc{li2026jevjudge,
title = {{JEV}-as-a-Judge: Accept When Confident,
Escalate When Unsure},
author = {Li, Yubo and Miao, Yidi and Krishnan, Ramayya
and Padman, Rema},
year = {2026},
eprint = {2609.26550},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2609.26550}
}