Self‑evolving language models iterate by proposing candidate updates and retaining any version that raises a visible score. When that score is an imperfect proxy for the true capability, continual selection widens the gap—a phenomenon known as reward hacking. To address this we introduce HackProbe, a monitor that can be attached to any self‑evolving loop using only two black‑box hooks, without access to weights or activations.
HackProbe keeps a secret, distribution‑fixed comparison core whose frozen distribution makes the capability proxy comparable across generations, together with a rotated fresh layer that hardens the core against co‑adaptation. Four tests built on this core detect: (1) level gap; (2) scale‑aligned divergence with online change‑point detection; (3) capability stagnation; (4) conditional confidently‑wrong rate. Applying a Sidak correction turns these into a calibrated family‑wise $p$‑value.
Diagnosis alone cannot stop hacking, so we add a risk‑aware immunization layer: each generation re‑selects an honest candidate from the proposal pool using the core and a purely structural gaming footprint, leaking at most $\log_2 \Pi$ bits to the host. We prove a detectability bound that converts a target error rate into an explicit probe‑size budget and delineate what probe rotation buys and what it does not.
In a controlled prompt‑level host experiment with four injected hacking channels and ground‑truth labels, HackProbe achieves $0.763$ AUROC, outperforming the strongest baseline ($0.663$), and reduces the false‑positive rate from $0.706$ to $0.434$. Its bandwidth‑limited reselection is the only immunization that returns more true capability under hacking—averaging a $5.2$‑point gain—while forfeiting $4.7$ points on clean runs; per‑channel effects are largely not individually significant.
Review: HackProbe offers a quantifiable, weight‑agnostic solution for detecting and immunizing reward hacking, illustrating a viable path toward safe iteration in self‑evolving systems.