Attention collapse in autoregressive language models manifests as repetitive token loops where the model becomes trapped in self-reinforcing attractors, representing a persistent issue that existing decoding-time heuristics fail to address. We present a principled framework that penalizes or compensates anomalous confidence arising from collapsed generation patterns by comparing a token's observed frequency against its corpus prior through an adjacent-conditional probability construction.
The resulting self-normalizing penalty ratio $$R=f(m,n,p)/f(np,n,p)$$ requires no ad hoc standardization and admits a closed-form logit offset with zero approximation error. The correction is isolated from the loss gradient and accumulated into a frozen output-layer bias via exponential moving average, enabling deployment as a repair mechanism for models that have already collapsed without requiring intrusive modifications to standard training pipelines. Experimental validation on a 1.5B-parameter model demonstrates that the frozen-bias mechanism can rescue a model already trapped in a collapsed attractor, reducing 2-gram repetition from 0.073 to near 0 while preserving generation quality.
Blogger's Review: This paper introduces the Bayesian repetition penalty mechanism, offering a novel approach to tackle attention collapse in autoregressive language models. By quantifying and adjusting the confidence of generation patterns, it demonstrates superior generation quality and flexibility, warranting further research and application.