In this paper, we introduce the concept of 'prolepsis' to explore when transformers make decisions and what prevents them from correcting those decisions. Prolepsis refers to a transformer committing early, with task-specific attention heads sustaining that commitment and no layer able to correct it.
We replicate Lindsey et al.'s (2025) finding on planning sites in open models (Gemma 2 2B, Llama 3.2 1B) and pose five questions:
- Planning is invisible to six residual-stream methods; among those tested, only CLT-based steering succeeds.
- The single-site spike replicates in shape at the final prompt token (Anthropic's site is the newline; see the Note added).
- Specific attention heads route the decision to the output, filling a gap flagged as invisible to attribution graphs.
- The evidence is consistent with search within at most 16 layers and commitment beyond, forming a two-model hypothesis.
- Factual recall shows the same motif at a different network depth, with zero overlap between recurring planning heads and the factual top-10.
Prolepsis recurs across tasks in the decoder-only models tested: the template is shared, while the routing substrates differ. All experiments are run on a single consumer GPU (16 GB VRAM).
Blogger's Review: This study offers a new perspective on understanding the decision-making mechanisms of transformer models, particularly how they maintain consistent commitments across various tasks. By investigating the behaviors of different models' attention heads, the authors reveal the complex decision-making processes within the models, providing significant insights for future model design and optimization.