Verifying numerical claims in papers written by AI agents is both important and challenging. Recent benchmarks show many agent‑generated papers fail integrity checks, yet those verdicts are often produced by a single LLM without explaining the nature of the failure, and existing checks only test internal consistency or whether a reported value appears somewhere in the agent's output.
We re‑examined all 540 released judgments from a recent public benchmark (nine agents, fifteen open research problems) and manually labeled 256 numerical claims from 18 submissions. Failures concentrate on the alignment between the paper and the underlying work rather than overt cheating; about four in five unsupported claims correspond to values in the agent's own files that come from the wrong experiment, metric, or run.
To address this we introduced a deterministic claim‑to‑evidence checker. Its rules are fixed before labeling, and it scores whether the surrounding context identifies the claimed result while calibrating each claim against numerical decoys. Against blinded annotators it achieves over 90% precision (78 of 85) with a Cohen's kappa of 0.89. The checker verifies only a subset of claims, but most unverifiable claims are still traceable, so failure to verify is not evidence of falsehood.
An LLM verifier given the same evidence recovers more traceable claims but accepts about three times as many misattributed ones, making the two approaches complementary.
Applied to 99 submissions, the checker reveals large differences in traceability across agents. A second judge from a different vendor preserves the official agent ranking while doubling the pass rate.
These findings motivate provenance‑first auditing of AI‑produced research.
Review