Abstract
Large language models (LLMs) are increasingly capable of mathematical problem solving and can even assist with research-level proofs. However, we still lack a scalable and reproducible way to measure step-level reasoning in long proofs across diverse sources. This evaluation gap limits trustworthy AI assistance in proof-certified scientific progress.
Existing evaluations often emphasize final answers or rely on costly expert grading, while end-to-end proof generation remains open-ended and hard to verify automatically.
We introduce Mask-Proof, a pipeline that turns real proofs into automatically checkable masked-step tasks. It masks key formula steps, provides the necessary surrounding context, and evaluates model reconstructions with an LLM-based equivalence judge using repeated votes for stability. The resulting Mask-ProofBench contains 292 curated problems across diverse research areas.
Experiments with 17 models show that reasoning-enhanced models outperform standard models by 12% to 27%. Our evaluator achieves 96.8% agreement with expert annotators, enabling faithful, reproducible, and comparable measurement of step-level mathematical reasoning. Benchmark, annotations, and code are available at GitHub.
Blogger's Review: The introduction of Mask-Proof not only fills the gap in evaluating mathematical proofs but also provides a novel approach for LLMs in scientific research. By introducing checkable masked-step tasks, it enhances the credibility of mathematical reasoning, which could have profound implications for scientific validation and education in the future.