Abstract
Chain-of-thought (CoT) prompting fails severely on constraint-dense logical reasoning tasks, where unverified errors accumulate silently across steps. We introduce SymStep: an LLM makes one atomic claim at a time (e.g., DEDUCE: Alice, pet, Cat), followed by a lightweight constraint propagator that checks the claim for consistency with prior accepted deductions, rejects contradictions, and cascades implied facts automatically.
SymStep+G additionally provides MRV guidance after each accepted step, directing the LLM toward the most constrained unresolved variable. On a 35-puzzle retained subset of ZebraLogicBench, a benchmark of 1,000 Einstein-style logic puzzles, both Direct and CoT achieve 0%, while SymStep+G reaches 97%. On AR-LSAT analytical reasoning problems, SymStep achieves 100% compared to CoT's 87%. On LGP-14, SymStep+G attains 100% versus 0% for CoT and Logic-LM, the strongest prior symbolic+LLM baseline we compare against.
Ablation studies reveal that MRV guidance is a key mechanism for reducing directionless cycling, while consistency checking provides a safety net against explicit contradictions. Across six benchmarks spanning five task domains, SymStep variants match or exceed every baseline on constraint-dense and arithmetic tasks. Experiments on AQUA-RAT algebra confirm that the advantage is constraint-density-specific.
Blogger's Review: SymStep's groundbreaking advancements in logical reasoning demonstrate the immense potential of combining symbolic reasoning with language models. By incorporating MRV guidance and consistency checking, SymStep not only enhances reasoning accuracy but also offers new approaches for tackling complex logical tasks, warranting deeper exploration in future research.