Abstract
Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge but suffers from knowledge conflicts: when retrieved information contradicts parametric memory, the shared self-attention pathway produces unpredictable outputs. We present TokenMem, a lightweight memory system that injects knowledge into frozen LLMs through a dedicated cross-attention channel, bypassing competition with parametric memory in the residual stream.
TokenMem trains only a thin gating adapter (approximately 3-7M parameters) via a two-phase curriculum: first learning general knowledge utilization, then strengthening faithful compliance under counterfactual knowledge. In controlled experiments on five models spanning three families (Qwen3-4B/8B/14B, LLaMA-3.1-8B, OLMo-3-7B), TokenMem achieves 69-70% Knowledge Compliance (KC) on counterfactual benchmarks, compared to 20-52% for vanilla RAG, a gap of up to 49 percentage points.
Ablation studies show that the two-phase curriculum is critical: removing Phase 2 collapses KC to near-zero. Mechanistic analysis reveals that the gate adapter learns a conflict-aware, layer-specific injection strategy without explicit supervision.
Blogger's Review: The introduction of TokenMem provides a novel and efficient approach for knowledge injection in frozen LLMs, showcasing its unique advantages, especially when dealing with knowledge conflicts. The effectiveness of this method is validated not only by experimental results but also offers new insights for future model designs. Its lightweight design enhances the flexibility of large model applications, warranting further exploration.