We introduce Mechanistic Shaping, a framework that constrains a model’s internal representations during training so that it obeys specifications about which concepts it should express (e.g., no harmful content) and which concepts it may rely on for decisions (e.g., not using gender when screening resumes). Across vision, medical imaging, and language tasks, up to 32 B parameters, and training paradigms such as classification, SFT, DPO, and GRPO, Mechanistic Shaping markedly reduces targeted behaviors while preserving task performance.
Since training against a representation can itself alter that representation, we ask whether shaped concepts remain monitorable. To answer this we define a hierarchy of monitor classes that separates a concept that merely evades a probe from one that is truly gone. We identify three geometric circumvention mechanisms: the concept drifts away from the monitored layer, it is erased from the layer, or it persists in higher‑order statistics as a nonlinear encoding (explained via a product‑code construction). Each mechanism can be detected by a matching monitor.
Empirically, refitted linear probes recover the concept in most settings, and the original probe stays reliable whenever gradients do not flow through it. Together, these findings highlight the promise of interpretability as a training tool and call for a rigorous science of interpretability‑based training. Review