When a language model might be making things up, the standard fix is to add a verifier: a retrieval system that checks the answer against a source, or a second model asked to judge whether the evidence supports the claim. The verifier is the safety net. So we asked a blunt question about the net itself: what does it do when the evidence has nothing to do with the claim?
We handed verifiers passages that did not bear on the statement being checked, and watched what they did.
What we found
A verifier that reasons over text without first checking whether the text is even about the claim will, often enough to matter, return a confident verdict anyway — across legal citations, software vulnerabilities and contracts. The structured verifiers that kept a simple internal relevance test — does the cited source even mention the right entities? the right norm? — almost never did. The failure is not "hallucination" in the abstract. It is ruling without first asking whether the evidence is on topic.
That contrast is the whole point: relevance is not a nice-to-have bolted on after the fact, it is the check that decides whether a verdict means anything. A net that never asks "is this even the right evidence?" will confidently catch the wrong fish.
Knowing when to abstain
A safety net that always answers is not a safety net. So the constructive half of the work is calibrated abstention: a verifier that can decline when the evidence falls short, with a finite-sample guarantee on how often it is allowed to be wrong. Where relevance can be scored cleanly — legal citations are the clean case — adding that gate turns a verifier that had to answer everything into one that answers when it can and steps back when it can't, without letting unrelated evidence through as a confident verdict.
This is the empirical side of a theme in our book El fantasma sin ego: an inventory of the ways these systems fail epistemically — hallucination, sycophancy, miscalibration — and what it would take for a system to know the edge of its own competence.