This paper investigates emergent misalignment in reward models after narrow harmful fine‑tuning. We fine‑tuned two state‑of‑the‑art reward models, Skywork‑Reward‑V2‑Llama‑3.1‑8B and Skywork‑Reward‑V2‑Qwen3‑8B, to prefer harmful outputs in a single domain (medical or financial). The preference leaked to seven held‑out domains, while overall scores dropped only 4.8 and 5.3 points respectively, still outperforming the untouched RM‑Qwen. We then fine‑tuned the underlying language models Llama‑3.1‑8B‑Instruct and Qwen3‑8B using DPO and SFT on the same data; reward models proved more susceptible to preference inversion than their language‑model counterparts. Mechanistic analysis shows two effects localized to the final scored token: (1) geometric compression of representations, reducing the angle between safe and harmful responses; (2) content‑specific rotation, where harmful representations form a smaller angle with the scalar head than safe ones, yielding higher rewards. Both effects appear only in harmful‑preference fine‑tunes, not in control experiments.
Review