Hallucination
A hallucination is a fabricated statement delivered in convincing form — a court decision that never existed, a contract clause that is not in the document. It happens because a language model produces the most likely continuation and draws no distinction between recalled and looked up. What makes it dangerous is not frequency but shape: it does not look like a gap, it looks like an answer.
Why does it happen at all?
A model writes by repeatedly choosing the most probable next piece of text. It has no notion of whether a statement is true, only of whether it is plausible. A fabricated case number is formally identical to a real one.
The expectation of fluency makes it worse. A system that is not permitted to say "I do not know" will fill gaps in its knowledge, because the task it was given leaves it no alternative.
What actually prevents it?
Three measures, in this order. Answer from supplied material rather than model memory — that is RAG. Require a citation for every statement, so an unsupported claim cannot be emitted. And permit "not found" as a valid, expected answer.
The third is skipped most often and works best. As long as a system must always answer, it will produce exactly the statements that later surface as invented citations.
Why is one incident so expensive?
Not because of the single error but because of what it does to adoption. A fabricated quotation that someone internally notices ends trust in the system permanently, and with it the willingness to attempt further use cases.
That is why citation is not a refinement but the entry condition for deployment anywhere statements carry consequences.
What it is not to be confused with
An error in the source
If a wrong figure is in the document and the system reproduces it with a citation, that is a data quality problem rather than a hallucination — and it is traceable, because the source is named.
An outdated answer
A superseded but real version is not an invention. It is still dangerous, and it is caught by surfacing the validity date, not by hallucination controls.
Frequently asked
What is the hallucination rate?+
The question misleads, because the rate depends on the design rather than the model. In free answering from model memory it matters; in answering exclusively from supplied material with mandatory citation it is largely moot — there is either a supported answer or none.
Can hallucinations be detected automatically?+
Only partially. The dependable route is prevention by construction rather than detection: where every statement must carry a source, an unsupported statement cannot be produced in the first place.
Are newer models not much more reliable?+
They are, but the improvement concerns frequency, not the nature of the problem. As long as a model may compose freely, the possibility remains — and in work with liability attached, a rarer invention is still an invention.
Does a human reviewer solve it?+
Only if the reviewer can realistically check. A person approving forty outputs an hour is not reviewing, and an interface that offers nothing but a confirm button produces the appearance of oversight rather than oversight. That distinction matters under Article 14 of the EU AI Act as well as in practice.
The test that settles it
Ask a question whose answer demonstrably is not in your documents. A usable system says so. That single exchange tells you whether it can be deployed where statements have consequences.
Request a test