Retrieval-augmented generation is the standard answer to the fabrication problem, and it is a real improvement. It is not a solution, and the difference matters when you are the one signing the filing.
What the architecture does, precisely
A retrieval system takes your question, searches a corpus, and puts the documents it found into the model’s context. Then the model writes an answer. Those are two separate steps, and only the first one touches the corpus [2].
The second step is the same next-token prediction it always was. It has better material in front of it, so it is wrong less often. But nothing in the pipeline compares the sentence that comes out against the document that went in. The citation at the end of that sentence is a claim the model made about its own sources, not a measurement of them.
What the measurement found in law
Stanford’s RegLab and the Institute for Human-Centered AI built a preregistered set of more than 200 open-ended legal queries and ran them against the leading retrieval-based legal research products [1]. General-purpose chatbots had already been measured on legal questions by the same group: hallucination rates between 58 percent with ChatGPT 4 and 88 percent with Llama 2 on specific, verifiable questions about randomly selected federal cases [3].
The purpose-built products did better, and still hallucinated between 17 and 33 percent of the time [1]. The write-up is blunt about the reason, under a heading that reads “RAG Is Not a Panacea”: any of the subsidiary steps may introduce error into the generated response [2].
They named three mechanisms, none of which a better index fixes.
Retrieval fails. Law is not a set of verifiable facts sitting in a database; it is built up over time by judges writing opinions, so the document that definitively answers a query may not exist as a discrete thing to find [2].
The retrieved document is inapposite. Rules and precedent differ across jurisdiction and time. A document can be semantically similar to the query and still be the wrong authority. One system recited the “undue burden” standard for abortion restrictions as good law after Dobbs overruled it [2]. The retrieval worked. The result was wrong anyway.
Sycophancy. Asked a question with a false premise, a system agreed that Justice Ginsburg had dissented in Obergefell and explained that she did so on the basis of her views on international copyright [2]. Both halves are invented, and neither came from a retrieval failure.
The same result outside law
This is not a quirk of legal corpora. RAGTruth, a word-level hallucination corpus built from nearly 18,000 generated responses across domains and tasks in standard RAG frameworks, exists precisely because, as its authors put it, despite the integration of retrieval, models may still present unsupported or contradictory claims relative to the retrieved contents [4]. The benchmark had to be built because the failure kept occurring in ordinary use.
What “grounded” is actually measuring
When a vendor says a system is grounded, ask what the word covers. Usually it means: the answer was generated in the presence of source documents. Sometimes it means: every citation in the answer resolves to a document that exists. Almost never does it mean: every proposition in the answer was checked against the passage cited for it, and anything that failed was withheld.
Those are three very different guarantees, and only the third one changes what you have to do before you sign. ABA Formal Opinion 512 makes the consequence explicit — using a generative AI tool without an appropriate degree of independent verification or review of its output could violate the duty of competence — and the opinion cites the Stanford study for the proposition that the leading legal products hallucinate between 17 and 33 percent of the time [5].
The question to put to a vendor
Not “do you use RAG.” Everyone uses RAG. Ask instead:
- After the answer is written, what compares each sentence to the source it cites?
- What happens to a sentence that fails that comparison — is it deleted, flagged, or printed?
- What does the system do when the corpus does not contain an answer?
If the honest reply to the last question is that the model writes something anyway, retrieval has improved the odds and left the architecture alone.
The alternative is to move the guarantee out of the model entirely: let the model draft freely, then hold every line to its source in code that the model does not control, and publish only what survives. That is how Apodicta is put together, and it is why it will return a shorter answer with the gaps named rather than a complete-looking one it can’t ground.