Efficient long-context inference is not only about reducing memory costs but also about keeping useful contextual evidence accessible as generation proceeds. Existing compression-oriented methods, such as key-value (KV) cache compression and context compression, often make early decisions on which contextual information to retain or rely on external compressors. This limits the adaptability of compressed contexts for subsequent reasoning steps.
This paper introduces PReM (Preserve and Refresh Memory), a context-compression framework that maintains long contexts as the model's internal layer-wise KV memory and learns what to preserve and when to refresh it. Specifically, PReM employs a dedicated memory layer for memory-selection decisions and a special memory token to trigger refreshes during generation. To train this behavior, PReM introduces Phase-Separated Refresh Training, aligning memory selection with memory-conditioned generation while preserving continuity across refreshes.
Experiments with 32K-token contexts show that PReM outperforms strong baselines under both 16x and 32x compression, maintaining a favorable balance between answer quality and inference efficiency.
Blogger's Review: The innovation of PReM lies in its dynamic management of context, particularly in long-context inference. By implementing an adaptive memory selection and refresh mechanism, it significantly enhances the performance and efficiency of generative models, showcasing remarkable application potential. Its design philosophy offers valuable insights for future context processing technologies.