Safe, because it looked everything up

Somebody on your team reads judge verdicts. Maybe it's a fixed sample every Friday, maybe it's whichever ones looked odd, and either way the number you'd most like to know, how many of the verdicts nobody read are wrong, is a guess. Sher Badshah, Ali Emami and Hassan Sajjad offer to replace the guess with a sentence you could put in a design doc: let the judge keep the verdicts it is sure of, make it look things up when it isn't, hand the rest to a person, and with probability at least 95%, no more than one kept verdict in ten will be wrong.

Their paper, Judge, Retrieve, or Abstain, was posted on 18 August and accepted at COLM 2026. I rebuilt its calibration and its router around a 9-billion-parameter Qwen on my laptop, gave it 1,972 factual questions whose answers come from tables that ship with Python and macOS, and wrote down, before collecting a single verdict, what would count as the promise failing.

It didn't fail. In a thousand calibration draws, the router as the paper prints it never once kept more wrong verdicts than it promised. I had half the write-up done as a clean pass when I noticed where its verdicts came from, which surprised me more than a failure would have: it had looked something up for 99.9% of the items. A judge that always checks its sources never has to decide when to trust its memory, and that decision is why the paper has two modes at all.

So I changed one tie-break, so it would answer from memory whenever that cost no coverage. Retrieval fell by half, and the promise still held on the four kinds of question it had been calibrated on. Then I calibrated it on three of the kinds and scored it on the fourth. On Unicode code points, 44% of the verdicts it kept were wrong, against a target of 10%, in every draw.

What this piece doesn't cover: subjective or pairwise judging, the paper's trained probe and LoRA uncertainty scores, live web search and how it drifts, and any judge larger than 9B. My search is a BM25 index over the same tables the answers came from, which is far kinder than the web, and the last result section is what happened when I made it less kind.

The working rule I'd take into any eval pipeline: calibrate per kind of question, not per judge, and keep a small labelled stream from live traffic whose only job is to tell you when the kinds have changed.

A judge that is allowed to say it doesn't know

Some model answers factual questions, and a second model judges each answer True or False without seeing a gold answer. The paper calls this reference-free judging. The judge works from what it knows, so it is sometimes wrong in a way the verdict doesn't show, and the paper gives it two ways out: it can look something up, and it can refuse.

Four terms carry the rest.

• A verdict is the judge's True or False on one candidate answer. It is wrong when it disagrees with the label.
• Uncertainty is the binary entropy of the judge's own probability of True against False, read at the token where it writes the verdict. One forward pass, no sampling (Section 4, Appendix B.2). Zero is certain; 0.69 is a coin flip.
• Risk is the share of kept verdicts that are wrong. The paper calls it the false discovery rate, and you choose its ceiling, α. At α = 0.10, at most one kept verdict in ten may be wrong.
• Coverage is the share of all verdicts that get kept. Everything else goes to a person.

In Mode 1 the judge decides from memory, and the verdict stands if its uncertainty is at or below a threshold t1. Otherwise, in Mode 2, the top three search results go into a second prompt and the judge decides again, under a second threshold t2. Otherwise a person decides (Algorithm 2).

The thresholds are set once, offline, on labelled verdicts. For any pair (t1, t2) you count how many calibration verdicts it keeps and how many are wrong, and the Clopper-Pearson bound turns that count into an upper limit on the true risk that holds with probability 1 − δ (Eq. 7). Algorithm 1 tries every pair of observed uncertainty values and keeps the one that keeps the most verdicts with its bound at or below α (Eq. 14). With δ = 0.05, you get the sentence from the top of this piece (Eq. 4, Eq. 15).

I had assumed that gap between the proof and the search would be the story. It turned out to matter less than I expected, and a different line of the same algorithm mattered more.

Thirteen lines, and one of them sets the bill

Every check in the search asks one question: with m kept verdicts and w wrong, is the Clopper-Pearson bound at or below α? That holds exactly when the binomial probability of w or fewer errors at a true rate of α is at most δ, so I precompute the largest allowed w for every m and each check becomes a table lookup. The core of my Algorithm 1:

for t1 in t1s:                        # ascending, as printed
    # ...move every item with u1 <= t1 out of the Mode 2 pool...
    j = len(t2s) - 1
    while j >= 0:                     # largest t2 first
        m = m1 + kept2[j]             # verdicts kept by this pair
        if m < best_m or (m == best_m and not fewer_retrievals):
            break                     # cannot beat the pair we have
        if e1 + wrong2[j] <= wmax[m]: # Clopper-Pearson bound <= alpha
            while j > 0 and kept2[j - 1] == kept2[j]:
                j -= 1                # smallest t2 that keeps the same set
            best_m, best = m, (t1, t2s[j])
            break
        j -= 1

test_joint_matches_literal_algorithm_one checks it against a slow, line-by-line transcription of Algorithm 1 on 25 random calibration sets, tie-breaks included, and they agree on all 25. Another pins the bound: zero errors first certifies a 10% risk at 29 verdicts.

The line that matters is the break on equal coverage. As printed, Algorithm 1 scans t1 upwards and replaces its best pair only when a new one keeps strictly more verdicts. If Mode 2 can certify every item by itself, the very first t1, the single most confident Mode 1 verdict, already reaches full coverage, and nothing later can beat it. You get a router that keeps one verdict from memory and retrieves for the rest. fewer_retrievals=True is my change, not the paper's: on a tie, prefer the larger t1, which answers more from memory.

The paper's chart shows averages, and the promise is about the tail

The authors test four open-domain question sets (TriviaQA, Natural Questions, HotpotQA, PopQA), 2,000 questions each, with candidate answers from Qwen3-8B and Llama-3.1-70B, judged by four open models from 4B to 14B, and average every result over 100 random 50/50 splits into calibration and test halves (Section 4).

The comparison is fair, and Table 12 adds the control I'd have asked for: a second pass with no new evidence barely helps, so the evidence, not a second look, does the work.

What Figure 1 can't show is the number the promise is about. Eq. 4 bounds the probability that one calibration draw produces thresholds whose true risk is above α. A mean over splits can sit under α while one draw in five breaks it, and the paper doesn't say how many of its 100 splits went over. I don't expect that hides a problem, but it's the wrong summary for the claim, and it's the gap my test is built around.

The authors are candid about the search, and I want to give them credit for it. Appendix D.4 describes a conservative variant that runs every check at δ divided by the number of pairs, and Table 13 prices it.

Two smaller things. The labels come from another model, Qwen2.5-7B-Instruct reading the reference answers (Appendix B.4), so the promise is about agreement with that evaluator; with exact-match labels, coverage on Natural Questions at α = 0.15 drops to zero (Table 7). And I found no code or data release.

Why I passed on the construct-validity paper

I screened six judge papers first posted in the six months to 8 October on claim support, baseline strength, causal evidence, production transfer and testability. Trust or Escalate (July 2024), which this paper builds on, and the Judge Reliability Harness (March 2026) were out on date. I read the selected paper in full and the others only as far as their abstracts, so the shortlist is a judgement call; weigh it accordingly.

The construct-validity paper was the closest call, and its finding is probably the more important one, but it tells you what to measure rather than what to run. Badshah and colleagues hand you a component with a promise attached, and a promise is something a prototype can try to break.

Questions whose answers nobody can argue with

A risk promise is only as good as its labels, and I didn't want a model grading the grader, so every question has its answer in a table. Elements (472): atomic numbers, names and symbols. Ports (500): the TCP port assigned to a named service, from the IANA registry in macOS's /etc/services. Unicode (500): the code point of a named character, from Python's unicodedata, half from blocks like Greek and half from scripts like Cherokee and Vai. Time zones (500): a zone's UTC offset on 15 January or 15 July 2025, from tzdata 2026c.

Labels are exact matches after normalisation. The search corpus is 15,772 rows from the same tables, and the answering row comes back in the top three for 99.8% of questions, which is the weak point of the design.

The examinee and the judge are the same model, qwen3.5:9b at Q4_K_M in Ollama 0.34.4 on an Apple M4 Pro, temperature 0, thinking off. One model in both seats is one of the paper's own configurations, and probably what a team with one local model would run. The prompts follow Appendix B.3. Uncertainty comes from the top 20 log-probabilities at the first True or False token, renormalised over the two, which the paper doesn't specify.

The examinee got the elements right 99.6% of the time, the time zones 80.4%, Unicode 10.8% and ports 4.0%. Judging from memory, it said True to 84.4% of all candidates, and of its 802 wrong verdicts, 760 were True on a wrong answer. It agreed with itself. Its most confident mistake was agreeing, with probability 0.997, that the port for imap4-ssl is 993. The registry says 585, with a comment that reads, in full, "use 993 instead". I counted that against the judge, and I wouldn't argue hard with anyone who counted it the other way.

A guarantee about true risk needs a truth, so the 1,972 recorded items are the population. Each calibration set is 1,000 items drawn from it with replacement, and the thresholds chosen on that draw are scored exactly on all 1,972. The paper's i.i.d. assumption holds by construction, so if the promise breaks, the procedure broke it.

The thresholds went into PREREGISTRATION.md before the first verdict, with six predictions. Three matter here: the printed search would overshoot in 8% to 25% of draws, because it keeps the best of several hundred thousand pairs each checked at 5%; the grid correction would give up at least half the coverage gain; and on a kind of question the calibration never saw, the router would overshoot in most draws.

git clone https://github.com/shravan1996/ship-the-paper-abstaining-judge
cd ship-the-paper-abstaining-judge
python3 -m unittest discover -s tests -t .
python3 eval.py       # the pre-registered analysis, from the committed verdicts
python3 explore.py    # the exploratory analyses below

Collection needs a local Ollama; everything after it is standard-library Python reading the committed verdicts. The full implementation and reproducibility notes are in the prototype repository.

A thousand draws, and the promise never broke

The pre-registered rule says Continue: no draws over target, and a coverage gain of 99.9 points over Eq. 8 as written, or 60.8 over the more generous reading. The paper's claim survived the test I designed to break it.

Why does Eq. 8 as written keep nothing? It asks for the largest threshold such that every smaller one also certifies, and with 1,972 distinct uncertainty values the smallest threshold holds one verdict, which can't certify anything. The sentence introducing Eq. 8, "the largest threshold whose UCB stays below the risk level," works, and my read is that it's what the authors ran. It's a search over many thresholds too, and it held.

Then the grid correction, which I predicted would cost at least half the coverage gain. It cost nothing, and that surprised me. So did the printed search, which I expected to overshoot in 8% to 25% of draws and which overshot in none. I was wrong on three of six predictions, all for the same reason.

It never broke because it never had to choose

The judge with evidence was right on 1,958 of 1,972 items, 99.3%, because my retriever nearly always handed it the answering row. With Mode 2 that good, every item can be kept at a risk far below any target I tried, and the strict tie-break settles on the smallest t1. The router keeps one verdict from memory and looks up everything else.

That's a perfectly safe system, and the expensive one the paper exists to improve on. If you retrieve for everything, the thresholds do no work, and the promise held because nothing near its edge was tested.

Then I let it answer from memory

The change is the one line in the code above: on a tie in coverage, prefer the pair that answers more from memory. Same objective, same check, same calibration draws.

The promise held, right up against the line: the 95th percentile of risk across draws was 9.9% at a 10% target. My read is that this is what a risk-controlled router should look like, spending exactly the risk you allowed to buy back half the retrieval bill. It also has no slack left for traffic that differs from the calibration set.

Unicode was the kind of question nobody calibrated on

So take the slack away. Calibrate on three of the four families, score the thresholds on the fourth. Appendix A.2 says a change of task calls for recalibration; I wanted to know what happens if you skip it.

On elements, the judge's confident verdicts are right: its median uncertainty there is 0.0016, a probability of True above 0.999. On Unicode it said True to a wrong code point 379 times, with a median probability of 0.89. Calibrated on every item of the other three families, the cheaper router's t1 lands at 0.38, and 222 of those 379 confident mistakes sit under it. The judge was 98.6% sure that U+03C0 is the GREEK PI SYMBOL. It is U+03D6.

I expected the unseen family to hurt. I didn't expect it to be invisible from inside the router. Coverage on Unicode was 99.6%, the retrieval share looked normal, and nothing went to a person. Every number on the dashboard said the system was working.

A hundred labels is a small price. The catch is knowing the family is new. Figure 2 puts these rows side by side, with one from the next section, where even the router as printed breaks.

What happens when search misses?

On familiar questions it mostly still holds, but the printed search finally overshoots at a tight target, and on a new family even the router as printed breaks. To get there I removed the answering row from the results for a fixed share of items, so BM25 returned its three nearest neighbours instead.

I expected that to make the judge hesitate. I did not expect it to make the judge decisive in the wrong direction. Shown "Helium, atomic number 2; Lithium, atomic number 3; Indium, atomic number 49" and asked whether hydrogen's atomic number is 1, it said False. Without the answering row it rejected 468 of 472 correct element answers, and on time zones it got 183 of 244 wrong, 139 of them at an uncertainty under 0.1. It read "the results don't mention it" as "the results contradict it."

So the gap I set out to measure is real and modest, and it needs a Mode 2 that is sometimes confidently wrong plus a tight target or a small calibration set. The grid correction closes it at a price that matches the paper's Table 13. The split version, which picks candidate pairs on half the calibration set and certifies them in a fixed order on the other half, closed it everywhere I looked and kept six times as much as the grid correction at a 5% target. It pays where labels are scarce: with 100 time-zone labels and a quarter of searches missing, it kept 0.6% of verdicts, where the printed search kept 34.5% by breaking its promise in one draw in seven. I'd still take that trade. It isn't in the paper, and it's the fix I'd use.

Four ways your eval set stops looking like your traffic

Every failure I found is one failure in different clothes: the calibration set stopped being a fair sample of what the router was scoring. I expect it to arrive in a real pipeline in one of these four ways.

A product team adds a new kind of question

The calibration set was built in March from March's questions. In June the assistant starts answering about a new product line, and the judge is confidently wrong about it in a way it never was about the old ones. No dashboard moves. The control is boring: tag every item with a family, keep thresholds per family, and send items from a family with no thresholds straight to a person. In my test, 100 labels from the new family were enough; labels borrowed from the other three were not.

Someone upgrades the model being graded

The paper lists a new candidate model as a reason to recalibrate, and my data suggests why. Almost every mistake my judge made from memory was a True on a wrong answer, so its risk depends on how many wrong answers it is shown, and a stronger or weaker examinee changes that mix. I didn't run a second examinee, so I can't say how big the move is. Make the examinee's version part of the threshold table's key.

Search finds the right table and not the right row

The Mode 2 instruction tells the judge to say False when the results contradict the answer, and my judge read neighbouring rows as contradiction. Watch the rate of Mode 2 False verdicts per family: a jump in a family where the examinee is usually right is this failure, and it shows up before any audit label does. A Mode 2 prompt that separates "contradicts" from "doesn't mention" might fix it. I haven't tried one.

The labels were another model's opinion

A promise about risk is a promise about agreement with whoever wrote the labels, and the paper's were written by Qwen2.5-7B. Table 7 shows how much that choice moves things, and my imap4-ssl row is the small version of the same problem. If your calibration labels come from a model, audit a slice by hand before you put α in a design doc.

Five rows say go, and three say slow down

The three slow rows have one cause, and it isn't the paper's statistics. Failure modes scores 3 because the failure that matters is silent: 44% wrong with every dashboard number normal. Data gravity scores 3 because the fix is labels, per family, for as long as the families keep changing, and operations scores 3 because somebody has to keep collecting them. The other five would let you run the paper's router tomorrow.

One calibration set per kind of question

The request path is short: a family tag, that family's thresholds, Algorithm 2, and a verdict that goes to the eval report or the review queue. An item whose family has no row never reaches the judge's memory, the one synchronous rule I'd defend hardest, because it turns the Unicode failure from silent into a queue that grows. Auditing runs beside the path: a random sample of kept verdicts goes to a person, and a family whose rolling bound crosses α stops keeping verdicts until it's recalibrated.

Start with the families you already grade

1. 01

   List the kinds of question your judge actually sees

   Owner

   Applied science + the eval owner

   Artifact

   A family tag on every graded item, with weekly volumes per family and the examinee model that produced it

   Signal

   Every item in last month's eval traffic maps to a family; the unmapped share is known

2. 02

   Label 250 verdicts per family, per examinee model

   Owner

   Eval owner

   Artifact

   Calibration sets drawn at random from each family's traffic, labelled by people or against references you have audited

   Signal

   Each family has its own set; none is borrowed from a neighbour

3. 03

   Calibrate per family and replay last month

   Owner

   Platform

   Artifact

   A versioned threshold table built with Algorithm 1, the tie-break toward memory, and a held-out half that certifies the chosen pair

   Signal

   Coverage, retrieval share and abstention share per family on replayed traffic

4. 04

   Run in shadow with an audit stream

   Owner

   Platform + eval owner

   Artifact

   Router decisions logged beside today's grading, with 5% of kept verdicts sent for a human label

   Signal

   Rolling Clopper-Pearson upper bound on risk per family, plus the rate of Mode 2 False verdicts

5. 05

   Turn it on for one family with a kill switch

   Owner

   Eval owner

   Artifact

   Kept verdicts from one family feed the eval report; everything else unchanged

   Signal

   The family's audit bound stays under α for four weeks

6. 06

   Widen family by family, or stop

   Owner

   Eval owner + platform

   Artifact

   A decision record per family with coverage, retrieval share, audit bound and the date of its last recalibration

   Signal

   No family goes live on another family's evidence

Ship gates, per family:

• Its own calibration set holds at least 250 labelled verdicts from its own traffic.
• Replaying last month keeps at least half its verdicts; a router that abstains on most of a family is an expensive review queue.
• The shadow audit's upper bound on risk sits below α.

Kill criteria, per family:

• The rolling audit bound crosses α.
• The examinee, the judge or the search index changes version and the family hasn't been recalibrated since.
• Mode 2 False verdicts jump while audited examinee accuracy hasn't moved, which is the right-table, wrong-row failure showing up days before the audit does.

Human review policy: abstentions always go to a person; that's what they're for. Each family's first two weeks live are audited at 20% instead of 5%. An item from a family with no thresholds reaches the review queue without the judge's opinion attached, so nobody is anchored by a verdict that was never calibrated.

Worth running this month, one kind of question at a time

Ship, as a shadow trial on offline grading of factual answers, one kind of question at a time.

The scorecard averages 3.6, which the rubric reads as Wait, and I'm calling it Ship anyway, for a narrow reason. Ship here means a bounded shadow trial under named controls, and that trial is also the cheapest experiment that would settle my remaining doubts. I've seen each control it needs work at least once: per-family calibration took Unicode from 44% wrong to 4%, the split check closed the overshoot, and the audit bound is ordinary statistics. Nothing touches a user; the worst case is a wrong number in an eval report, caught by an audit you'd run anyway.

The paper and my prototype agree more than they disagree, which hasn't happened often in this series. They part on what the promise covers. The paper shows average error under α wherever it looked; the promise is about each calibration draw, and it holds only while tomorrow's questions look like the calibration set's. I expect most teams to find that condition fails more quietly than they think, so I'd let my results set the controls and the paper's set the decision to try it.

What I wouldn't do is the broad version: one calibration set for the whole judge, and a design doc promising "at most 10% wrong" about traffic nobody has audited.

I would change this verdict if:

1. A shadow audit of a real family, calibrated on that family's own labels, puts the upper bound above α after 1,000 audited verdicts. The promise would then be failing where its assumptions hold, and I'd move to Wait.
2. The authors release code whose tie-break already prefers memory, and per-split results that rarely go over α. My tie-break finding would become a note about the pseudocode, and I'd raise confidence to High.
3. Live web search, in place of my BM25, produces the right-table, wrong-row failure on more than a few percent of a family's items. That family's Mode 2 prompt needs rewriting first, and its verdict becomes Wait.

What I can't tell from a laptop is how often a real eval set changes kind without anyone noticing. My families were separate tables with separate names. In a real assistant, a new kind of question arrives as a product launch, a new customer segment, a dropdown someone added to a form. I suspect some of that drift happens inside families no tag would separate, and I'm not sure an audit at 5% would see it in time. Is a family tag enough, or does the next Unicode arrive one confident mistake at a time?

Acknowledgements

This article builds on Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees by Sher Badshah, Ali Emami and Hassan Sajjad.

The prototype reimplements the calibration in Section 3 and Algorithms 1 and 2 from the paper's text; I found no official code. It uses:

• Qwen3.5 9B, from the Qwen team at Alibaba Cloud, served locally by Ollama, as both the examinee and the judge
• Reference answers from Python's unicodedata module (the Unicode Character Database, version 16.0), the IANA service name and port number registry as it ships in macOS's /etc/services, and the IANA time zone database (tzdata 2026c)
• A periodic table written into the repository by hand, checked by a test for completeness and uniqueness

The questions, prompts and evaluation are my own. No third-party question set was used.

References

Primary research

• Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees. Sher Badshah, Ali Emami and Hassan Sajjad, 2026. First public version 18 August 2026; accepted at COLM 2026.

Supporting and contrary evidence

• Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement. Jaehun Jung, Faeze Brahman and Yejin Choi, 2024. Selective judging with a guarantee on agreement with people; background, outside the six-month window.
• Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan and Lihua Lei, 2021. Treats choosing a threshold as multiple hypothesis testing; the basis for the split-and-certify variant.
• A Judge Should Know What Changed: Construct Validity for LLM-as-a-Judge Evaluation. Jianlin Chen, Wenhui Chen, Ziyao Lin and Chi Man Vong, 2026. Judges stay stable under edits that should not matter and miss most edits that should.
• When Trust Meets Truth: Trust-Truth Separability in LLM-as-Judge. Xin Sun, Di Wu, Yuchen Guo, Jiahuan Pei, Isao Echizen, Abdallah El Ali and Saku Sugawara, 2026. A source label alone moves a judge's truth verdicts.
• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability. Zongyou Yang, Yinghan Hou and Xiaokun Yang, 2026. Replacing the judge is a measurement change.
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias. Justin D. Norman, Michael U. Rivera and D. Alex Hughes, 2026. High test-retest reliability alongside severe position bias.
• Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations. Manan Gupta and Dhruv Kumar, 2026. Conformal prediction sets as a per-item reliability signal for scored judgments.
• Judge Reliability Harness: Stress Testing the Reliability of LLM Judges. Sunishchal Dev, Andrew Sloan, Joshua Kavner, Nicholas Kong and Morgan Sandler, 2026. Excluded on date: first posted 5 March 2026, outside the six-month window.

Prototype

• Source, data, setup and evaluation
• Exact tested commit

Sources last checked: 2026-10-08