In high-stakes domains, AI-generated answers are often fluent yet difficult to verify, particularly when they involve multi-step reasoning instead of a single final answer.
We propose a reasoning-based, reference-free framework for auditing LLM-generated outputs. The method decomposes a generated reasoning trace into segments, labels local premise-target relations using Natural Language Inference (NLI), and organizes these relations into a hypergraph.
A deterministic backward AND-OR search then assigns segment-level audit labels that indicate how each segment is grounded within the generated response. We evaluate the framework in two settings: deductive mathematical reasoning with Hard2Verify, and open-ended medical reasoning with UroReason, a new physician-annotated benchmark of LLM reasoning traces from real clinical cases.
Across these settings, our NLI-hypergraph audit provides a more reliable reference-free evaluation signal than direct LLM-as-judge baselines. In the clinical setting, state-of-the-art LLM judges often fail to identify problematic reasoning segments, over-accepting fluent but weakly grounded responses.
Our results show that QA evaluation should account for how inferential relations compose across a reasoning trace, rather than relying only on final answers or LLMs as verifiers. UroReason will be made available through an API, and our code will be released as open source.
Blogger's Review: This paper proposes a reference-free reasoning auditing framework, offering a new perspective on assessing the credibility of AI-generated content in the open-ended question answering domain. By meticulously analyzing the reasoning process, it provides a better way to identify potential reasoning flaws, ensuring more reliable support for AI systems in high-stakes applications. The release of open-source code and API will undoubtedly foster further research in this area.