Introduction
This paper investigates the behavior of large language models (LLMs) under anomalous conditions, specifically how their explanatory engagement (including length, specificity, and self-reported confidence) changes as failures become asymptotically rarer in a workflow with a low, controllable failure rate.
Experimental Design
The authors built a local, zero-cost harness on three open-weight models (qwen3:8b, llama3.1:8b, mistral:7b) running a repeated tool-call task where one call fails at probability $p$, swept across eight rates from 0.2 to 0.0001, under five elicitation conditions from immediate prompting to none.
Results
The results show that as failures become asymptotically rarer, the models' explanatory engagement does not change in a monotonic pattern as expected. Instead, it presents different patterns under different conditions. Specifically, under immediate forced explanation, the model's explanation length peaks at 28.4 words at $p=0.05$ and then settles to 17.4-19.0 words at the rarest rates, with confidence rising unevenly from about 53% to the 70s-90s.
Conclusion
The results of this paper suggest that elicitation structure is a first-class moderator of collapse observability. Future work directions include capturing behavior between discrete rate points. Blogger's Review: This paper experimentally studies the change in explanatory engagement of large language models under anomalous conditions, revealing the importance of asymptotic rarity and elicitation structure. It has significant implications for understanding and improving LLMs' ability to handle anomalies.