Large language models sometimes generate fluent answers that are not supported by the provided context. By reading both the context and another model's response—even when the generator is proprietary—a compact frozen proxy model can detect such hallucinations. Attention‑based detectors rely on summary statistics of the proxy's attention weights, but conventional pipelines materialize the full attention matrix, causing memory to grow quadratically with sequence length. We compute these statistics without materializing the matrix by accumulating them during the FlashAttention pass, performing sparse lookups afterward, or executing a second tiled pass over the score matrix, categorizing the resulting paths into virtually free and modest‑cost variants.
On an H100 GPU, the statistics of five published detectors add only 16% latency to a Qwen3‑8B forward pass at 2K tokens, dropping to 1.4% at 35K tokens, and require merely 22.5 KiB of storage per response token. We trained probes on frozen Qwen3 proxies ranging from 0.6B to 8B parameters and evaluated them on three benchmarks. A 4B proxy outperformed the strongest of nine zero‑shot 4‑9B LLM judges in F1 on two benchmarks; the proxies matched or exceeded specialized grounding checkers up to 7B parameters; probing each proxy beat prompting it directly as a judge; and adding attention features significantly improved F1 in half of the 12 benchmark‑proxy pairs, never causing a notable drop.
Overall, leveraging FlashAttention to accumulate attention statistics provides a low‑overhead solution for hallucination detection, offering a practical path for real‑world deployment. Review