Abstract
Fine-tuning large language models (LLMs) to inject new knowledge faces a critical challenge: while LLMs can quickly memorize new facts, they fail to effectively use these facts in downstream reasoning tasks. We formalize this failure as the Knowing--Using Gap, characterized by an accuracy gap and a temporal lag between memorization and generalization.
To understand this phenomenon, we fine-tune LLMs with unseen knowledge and monitor the spatial permeation dynamics of the knowledge internally using a novel intervention technique called self-patching. Self-patching identifies activation locations where relocating representations substantially improves failed generalization cases.
These results support a knowledge-circuit misalignment hypothesis: memorized representations can exist internally but may not be routed to computation-effective layers. To demonstrate the practicality of this diagnostic finding, we design a simple heuristic strategy which recovers 58% to 75% of the oracle headroom in generalization failure. Experiments are conducted cross-domain to ensure the robustness of this finding.
Blogger's Review: This paper delves into the Knowing-Using Gap in large language models through the self-patching technique, revealing potential internal representation issues. Understanding this mechanism is crucial for optimizing LLM applications in real-world scenarios.