ARCANA (Abstraction and Reasoning Corpus of ANAlogy) is a benchmark designed to test far analogical reasoning, i.e., reasoning between domains that appear superficially unrelated but share deep relational or structural similarities. The tasks require mapping real‑world objects or physical phenomena onto abstract grid‑based transformations, contrasting with the ARC‑AGI benchmark that mainly leverages innate human priors.
A human study confirmed that participants can reliably solve ARCANA instances, while evaluations of various large language models (LLMs) revealed a pronounced performance gap. The models exhibited systematic failure modes on several tasks, indicating that current LLMs struggle with deep analogical reasoning.
To bridge this gap, we introduce the ConceptSIR framework, which combines concept slippage, introspection, and reflection. Experiments show that ConceptSIR guides standard LLMs to solve several ARCANA challenges that remain unsolved by direct prompting.
Overall, far analogical reasoning remains an open problem for contemporary LLMs. All materials are available at sites.google.com/view/arcanacogtest, and the full codebase, benchmark, and ConceptSIR implementation are publicly released on GitHub.
Review