Evaluating the multi-hop reasoning capabilities of large language models remains a significant challenge. Although current models achieve strong results on existing multi-hop question answering datasets, such performance often masks two critical vulnerabilities:
- Reliance on internal parametric knowledge rather than adherence to the provided context;
- Exploitation of dataset shortcuts, such as single-document cues or type-matching, that diminish the need for genuine evidence aggregation across multiple documents.
To address these limitations, we introduce CRiT-QA (Counterfactual Reasoning with Traps), a dataset explicitly designed for this purpose. CRiT-QA transforms factual reasoning chains with counterfactual entities to neutralize reliance on memorized knowledge and enforce strict context dependency. Furthermore, it injects multi-anchor distractor chains, which are plausible but incorrect reasoning paths that diverge at different hops. These traps require models to follow the entire reasoning process rather than exploiting shallow heuristics.
Our experiments show that LLMs exhibit substantial performance degradation on CRiT-QA compared to standard datasets, exposing their vulnerability to counterfactual conditions and distractor traps. Thus, CRiT-QA serves as a rigorous diagnostic tool for evaluating genuine multi-hop reasoning and provides a foundation for developing more reliable, evidence-grounded LLMs.
Blogger's Review: The introduction of CRiT-QA offers a fresh perspective on assessing the reasoning capabilities of language models, particularly in multi-hop scenarios. By setting counterfactual conditions and distractor paths, researchers can better identify and understand the limitations of models, which is crucial for advancing stronger and more reliable AI systems.