This paper documents a failure class observed in frontier large language models—exception chain collapse—primarily in eligibility evaluation under nested conditional rules such as “A is required UNLESS B applies, UNLESS C overrides B”. The failure reproduces on first observation, yet its empirical surface is unstable: between March and April 2026, several failure cells closed silently under the same model alias, without version bumps (e.g., GPT-5.4 on construction insurance moved from 96.6% to 100% using the same prompt and harness). For regulated workflows, frontier model accuracy is a shifting compliance boundary that can change without notice.
We present the Aethis Eligibility Module, a neuro-symbolic architecture where LLMs author rules from authoritative sources, and an SMT-based layer executes them deterministically, consistent with the authored specifications regardless of model drift, reasoning-effort defaults, or prompt formats.
Three evidence bases are provided: (i) a controlled benchmark of 225 scenarios across four regulatory domains documents the pattern and the drift that partially closed it; (ii) a 20-scenario adversarial extension on construction insurance, where the engine scores 20/20, as does one of four frontier configurations (GPT-5.4 at low reasoning effort), while the other three, including Anthropic’s strongest model at evaluation time, fail the same coverage-gap edge case; (iii) external validation on nine peer-reviewed LegalBench tasks, with 949 held-out cases, showing the engine is significantly more accurate than all three frontier models (combined McNemar’s p value).
Blogger’s Review: By documenting the phenomenon of exception chain collapse, this paper highlights the instability of frontier large language models in compliance evaluations. The Aethis Eligibility Module provides a solution that combines neural networks and symbolic logic, showcasing potential for future rule execution. Stability and accuracy of models are crucial, especially in regulated environments.