Abstract
Autoregressive multimodal large language models (MLLMs) suffer from error snowballing: a single incorrect inference early in a chain-of-thought (CoT) trace corrupts all downstream reasoning. We find that in state-of-the-art open-source MLLMs, once the first error occurs, the reasoning cascades into failure across all remaining steps in 65% of such cases (a metric we term the snowball rate). Existing mitigations—sampling multiple chains, post-hoc self-verification, or full program synthesis—either lack symbolic grounding, catch errors too late, or sacrifice the flexibility of natural language reasoning.
We propose Constraint-Anchored Reasoning Traces (CART), a neuro-symbolic framework that trains MLLMs to interleave natural language reasoning steps with symbolic constraint assertions: lightweight, machine-checkable statements about visual content (e.g., count(red_objects) = 3). A dual-pronged Constraint Propagation Module—combining a learned neural grounding head with Boolean Constraint Propagation—continuously verifies these anchors against extracted visual features and checks their mutual logical consistency. When a contradiction is detected, a backtrack controller halts generation and reverts to the last consistent checkpoint, preventing error propagation.
A variable-frequency emission mechanism allows the model to adaptively control anchor density, avoiding trace bloat. We construct 218K training instances by augmenting GQA, CLEVR-CoGenT, and VCR with ground-truth constraint annotations derived from scene graphs, and fine-tune open-source MLLMs (LLaVA-NeXT, Qwen2-VL) via LoRA. On five benchmarks, CART reduces the snowball rate from 0.65 to 0.14, improves GQA accuracy by +4.6 percentage points over training-only baselines, and achieves 89.1 F1 on POPE-all with at most 18% inference overhead.
Blogger's Review: The Constraint-Anchored Reasoning Traces (CART) framework significantly mitigates the risk of error propagation in autoregressive models by integrating neural networks with symbolic logic. This innovative approach showcases the potential to enhance accuracy and stability in multimodal reasoning, opening new avenues for future MLLM research.