Large language models (LLMs) achieve strong performance on mathematical reasoning benchmarks, yet the mathematically meaningful skills behind their reasoning remain underexplored. We introduce SHAPE, a framework that examines Chain-of-Thought (CoT) trajectories from two perspectives rooted in mathematics education: (1) semantic spaces – the model's evolving mathematical interpretation of a problem (e.g., algebraic, geometric); (2) heuristics – the specific mathematical actions taken within those spaces (e.g., simplifying, working backward).
Applying SHAPE to analyze the reasoning paths of various models reveals that the choice of heuristics explains answer correctness better than traditional CoT features. Moreover, models tend to concentrate their reasoning within a few semantic spaces rather than exploring many disparate ones, a pattern consistent with human learning behavior.
We then investigate whether post‑training truly enhances mathematical proficiency. Findings show that reinforcement learning induces mode‑seeking in heuristic usage, causing models to rely on a limited set of frequent heuristics and reducing diversity.
Motivated by this, we incorporate a mechanism that promotes heuristic diversity during post‑training and demonstrate its effectiveness in boosting accuracy. Overall, SHAPE offers a theoretically grounded diagnostic tool for decoding LLM reasoning and points to a new avenue for improving math reasoning via post‑training. The code is publicly available: https://github.com/holi-lab/SHAPE-of-CoT
Blogger's Review: SHAPE bridges educational theory with model interpretability, highlighting the synergy between semantic spaces and heuristics. It provides a concrete direction for post‑training enhancements and opens promising research opportunities.