NeFut Logo NeFut
中 Admin Login

[CS.AI] Math Reasoning in LLMs is Organized by Approach, Not Topic

Published at: 2026-09-24 22:00 Last updated: 2026-09-28 00:49
#AI #Machine Learning #LLM

Mathematical reasoning benchmarks are usually arranged by topic, yet language models may internally structure computation by reusable reasoning approaches. This paper investigates whether open math‑capable LLMs organize themselves by topical sub‑skill or by reasoning approach, and presents evidence that the latter is the decisive factor.

We introduce a generation‑replay protocol: the model first produces a solution, after which we replay the exact prompt‑plus‑generation trajectory and extract activation‑importance signatures for each reasoning token. These signatures are clustered in an unsupervised manner across eight models and five mathematical reasoning sources, and the resulting structure is evaluated with structural, semantic, and intervention tests.

Across all 40 model‑source cells, the recovered clusters consistently outperform size‑matched random baselines. Two independent frontier‑LLM judges report approach‑level coherence in 77‑82 % of real clusters, compared with only 6‑11 % for within‑source controls. Clusters that are pure‑topic are usually labeled with a finer‑grained reasoning approach than the topic itself.

In an approach‑controlled prompting experiment, changing the requested reasoning approach shifts cluster assignment in seven of eight model conditions, whereas paraphrasing the prompt largely preserves the original cluster. These findings indicate that math‑capable LLMs organize internal computation by reasoning approach rather than benchmark topic.

Implication: Topic‑stratified benchmarks and topic‑balanced training corpora can still miss the crucial axis; even deliberately topic‑balanced corpora may remain imbalanced with respect to reasoning approaches. Future evaluation and data collection should prioritize diversity of approaches to avoid hidden biases.

Review

Original Source: https://arxiv.org/abs/2609.27041

[h] Back to Home