As language models remain frozen in their training state, the world continues to evolve. Knowledge editing has emerged as a key alternative to full retraining, but it faces core capability erosion: mathematical and programmatic reasoning collapse while encyclopedic recall remains intact. We trace this asymmetric degradation to a distributional mismatch.
Covariance-based editors preserve only the subspaces spanned by their reference corpus and fail to capture the operative distribution shaped by post-training techniques like SFT and DPO. Static external corpora, including Wikipedia and even the original pretraining mixture, cannot recover this shifted manifold.
We propose Moir, which estimates the preservation covariance $C$ directly from the model itself by sampling from its decoding distribution. By seeding generation with a single random vocabulary token, we bypass the instruction-following templates that usually dominate sampled outputs, exposing the broader subspaces the model has internalized. Moir requires no external data and serves as a drop-in component for any covariance-based editor, providing a practical advantage given that the pre- and post-training corpora of most modern LLMs are not publicly accessible.
Across OLMo-2, Llama-3.1, and Qwen-3 (7-8B), Moir consistently extends preservation in the most vulnerable domains under both MEMIT and AlphaEdit, particularly on Qwen3-8B, retaining 79.9% GSM8K accuracy after 20,000 AlphaEdit batch edits compared to just 10.9% with the Wikipedia baseline. These results suggest that aligning the preservation distribution with the model's operative distribution is a key factor in non-destructive editing, and that the model itself may be the most accessible source of that distribution for deployed systems.
Blogger's Review: The introduction of Moir presents a fresh perspective on knowledge editing by allowing models to self-direct, reducing reliance on external data, and showcasing adaptability in dynamic environments. This approach not only enhances the robustness of editing but also opens new avenues for future LLM applications, warranting further research and exploration.