Preference-based post-training is often understood through endpoint behavior, yet the learned update that produces this behavior remains largely opaque. We investigate RLHF and related preference optimization through the spectral structure of their induced parameter updates.
By decomposing effective LoRA updates and reloading their spectral components as plug-in modules, we transform preference-induced updates into objects that can be isolated, recomposed, and directly intervened on.
Across model families, optimization algorithms, and supervision regimes, these updates consistently develop a spectral head-tail organization. A compact head emerges early and carries the dominant endpoint shift, while a heterogeneous residual tail remains.
This split is functional rather than merely descriptive. Plug-in intervention shows that the head accounts for the visible behavioral departure from the base model, while the tail is weak in isolation.
Cross-run recomposition further indicates that mixed adapters follow the source of the head, suggesting that the head carries run-level solver bias. This endpoint dominance does not imply learning sufficiency.
Head-only learning is non-vacuous but fails to recover the full solution, especially on out-of-distribution behavior. Tail-only learning yields little visible gain, yet the full solution is not recovered without the tail.
These findings recast preference post-training as structured update reorganization rather than a monolithic behavioral correction, and suggest that alignment gain and coverage loss are tied to how the learned update itself is organized.
Blogger's Review: This paper presents a novel perspective on spectral update reorganization for preference tuning, highlighting the complexity and diversity of models in post-training processes, which merits further exploration of its potential and limitations in practical applications.