NeFut Logo NeFut
中 Admin Login

[CS.AI] The Intrusion Pathway: How Non-Evidential Context Degrades Language Model Performance

Published at: 2026-11-06 22:00 Last updated: 2026-11-07 12:32
#Machine Learning #optimization #LLM

Language models lose accuracy when a prompt contains non‑evidential context—irrelevant statements or random off‑distribution tokens that share no semantic overlap with the query and provide no competing answer. Such noise is common in retrieval‑augmented, tool‑using, and long‑context deployments, yet a mechanistic explanation has been missing. We attribute the degradation to an "Intrusion Pathway": MLP layers first enrich representations of both the query subject and irrelevant tokens; “parroting heads” then copy prefix‑token content to the prediction position; and relational readers largely avoid unrelated sources. Although subject enrichment and selective reading are preserved, the broader computation remains vulnerable to intrusion. By localising the effect we identify a sparse set of attention heads—called intrusion heads—that drive most of the interference. Their prefix‑derived writes promote distractor content, while the reduced support for the correct answer stems mainly from weaker non‑prefix contributions rather than direct suppression by the prefix. These heads transfer across tasks and distractor types and are largely weight‑disjoint from heads that integrate relevant evidence, enabling a class‑conditional intervention. An oracle probe that replaces the corrupted last‑token residual with its clean counterpart recovers on average 97.6% of the distractor‑induced drop across four downstream tasks. A deployable mask that knocks out about ten heads (≈1% of all attention heads) restores 57% of the lost performance with limited cost to helpful context.\n\nReview: The work pinpoints a tiny, transferable subset of attention heads as the source of noise sensitivity and offers a practical, low‑overhead mitigation strategy.

Original Source: https://arxiv.org/abs/2611.07778

[h] Back to Home