For two years the argument about AI in legal work has been conducted as if hallucination were a bug: an immaturity the next model would grow out of. In September 2025 the company that makes the most-used models published a paper saying it is not a bug. It is a consequence of how the models are trained and scored, and no amount of scale fixes it [1].
That paper is worth reading in full, because it changes what a careful lawyer should ask of any tool that writes.
What the paper says
The authors are four researchers at OpenAI and Georgia Tech. Their claim is stated in the first sentence of the abstract: language models hallucinate “because the training and evaluation procedures reward guessing over acknowledging uncertainty” [1]. A model that says “I don’t know” scores zero on a multiple-choice benchmark. A model that guesses scores sometimes. Across thousands of such benchmarks, the guessing model wins, so the training pressure runs one way. In their phrase, models “are optimized to be good test-takers, and guessing when uncertain improves test performance” [1].
The paper then does something the field had not done cleanly: it shows that hallucination on a factual question reduces to a classification error. If a model cannot reliably tell a true statement from a plausible false one, it will produce false ones at a rate the mathematics predicts. Pretraining on more data lowers that rate for things that appear often and leaves it where it is for things that appear once, like a specific docket number or the holding of an obscure case. Those are exactly the things a legal filing is made of.
The remedy the paper proposes is not a better model. It is a change in scoring: penalize a confident wrong answer more than an admission of uncertainty, and give partial credit for saying so. That is a proposal about benchmarks, not a feature you can buy today.
The numbers OpenAI put beside the argument
The research post that accompanied the paper carries a table from the GPT-5 system card, on the SimpleQA factual-question benchmark [2][3]. Two models are compared. The older one, o4-mini, abstained on 1 percent of questions, answered the rest, and was wrong 75 percent of the time. The newer one, gpt-5-thinking-mini, abstained on 52 percent of questions and was wrong 26 percent of the time. Its accuracy was slightly lower than the older model’s: 22 percent against 24 percent [2].
Read those four numbers together. The error rate fell from 75 to 26 percent, but it fell because the model stopped answering half the questions, not because it got better at the ones it answered. The improvement is abstention, and abstention is the only lever a model has. The post says as much: the better behavior is “acknowledging uncertainty,” and the way to get it is to stop rewarding the guess.
For a lawyer the practical reading is this. When a model declines, that is the system working as intended. When it answers, nothing in the architecture has checked the answer against anything. The answer and the confidence come out of the same operation.
Why abstention is not enough for a filing
Abstention is a real improvement and it is not the same thing as verification, for three reasons the paper’s own framing makes plain.
First, the model cannot tell you why it abstained. “I’m not certain” could mean the authority does not exist, or that it exists and the model has seen it once. A lawyer needs the first case distinguished from the second, because the first is a cite to strike and the second is a cite to go read.
Second, the model cannot be trusted to abstain when it should. Abstention is itself a learned behavior, learned from the same scoring that the paper says is miscalibrated. A model tuned to abstain more will abstain on some questions it could have answered and answer some it should have declined. There is no line inside it that separates the two.
Third, and this is the part that matters for anyone who signs a document: an abstention is a failure to produce text. It is not a computed result with a reason attached. A court does not accept “the tool was unsure” as a reason a citation was not checked. The duty to verify, which ABA Formal Opinion 512 restates for generative tools, sits with the lawyer regardless of what the tool declined to say [4].
What this changes about buying a tool
If hallucination is structural, then the question for a vendor is not “how often does your model hallucinate” but “what, outside the model, checks what it wrote before I see it.” The paper is clear that a model has no internal checker; the thing that samples the claim is the thing that samples the citation and the thing that samples the confidence. A check has to be a separate operation, with a separate standard, that a person can re-run.
That is what a citation gate is: code, not a model, that resolves every cited source, confirms the quoted words are in it, and refuses to render a line that fails [see related]. It is also why the right output for an unproved claim is not softer wording. It is a line that says, in plain words, that this could not be grounded, and why. The paper’s authors would call that acknowledging uncertainty. We call it a held line, and it is the only honest answer a system can give when the record does not contain the proof.
The sentence to keep
The paper’s authors argue that some questions are unanswerable from what a model has seen, so a generative model’s accuracy has a ceiling below perfect no matter how it is trained [1]. That is a design statement from the people who make the thing. It is not an argument against using models. It is an argument for never letting a model’s sentence reach a court without a check that is not a model.