Large language models can now produce complex multi‑step mathematical proofs, yet reliably judging their correctness and pinpointing early logical errors remains a core challenge. Existing evaluation methods largely rely on model‑based natural‑language judgments, which often miss local reasoning gaps. Formal theorem provers such as Lean offer rigorous verification, but applying them to informal text faces locality and semantic mismatch issues: a prover may bypass a local flaw by proving an overly broad target, or validate an auto‑formalized statement that drifts from the original mathematical intent.
To address this, we introduce FaithSieve, a Lean‑assisted framework for fine‑grained evaluation of natural‑language proofs. FaithSieve first decomposes coarse proof steps into local reasoning units, extracts typed proof obligations, and verifies them via a formal evaluation agent. Formal validation is gated by a semantic alignment score, ensuring that Lean evidence is incorporated only when the formal statement faithfully preserves the context, objects, and logical form of the original claim.
We build two expert‑verified datasets—ProofLoc‑Olympiad (350 problems) and ProofLoc‑University (200 problems across six advanced domains)—to benchmark first‑error localization. Experiments show that with a GPT‑5.4 backbone, FaithSieve achieves 81.43% exact first‑error accuracy on the Olympiad set, outperforming the direct‑judging baseline of 72.29%. On the University benchmark, FaithSieve reaches 84.5% exact accuracy versus 75.0% for the direct judge.
These results demonstrate that decomposing proofs into fine‑grained units and grounding them with faithful formal evidence markedly improves the reliable evaluation of natural‑language mathematical reasoning.
Blogger's Review: FaithSieve presents a compelling approach by filtering formal evidence through semantic alignment, effectively bridging the gap between pure language judgments and rigorous proof verification, and offering a more trustworthy assessment of LLM‑generated mathematics.