Direct Preference Optimization (DPO) has become a standard reward‑model‑free method for aligning language models with preference data. As the preference margin expands during training, the logistic DPO loss grows less sensitive to further updates. By examining the loss from a geometric perspective, we identify the sigmoid factor as a learning signal that quantifies the local sensitivity of the objective. Leveraging this insight, we introduce Learning‑Signal‑Controlled Direct Preference Optimization (LSC‑DPO), which dynamically regulates the learning signal to stay within a target regime, preserving effective gradient information. A log‑space analysis derives conditions under which the target learning‑signal regime can be stably tracked. Experiments on AlpacaEval 2, MT‑Bench, and Anthropic‑HH demonstrate that LSC‑DPO consistently outperforms vanilla DPO and strong preference‑optimization baselines. Moreover, we find that different coefficient initializations generate distinct transient learning‑signal trajectories, even though their long‑term signal levels converge. From this observation we formulate a signal‑budget compensation rule that adjusts the target learning signal to offset transient differences, markedly reducing performance variance across initializations.
Review