Direct Alignment Algorithms (DAAs) like DPO have become common post-training methods for aligning LLMs with human preferences. However, DAAs have been observed to over-optimize their implicit reward models, resulting in decreased likelihood of preferred responses. This leads to a reduction in the total likelihood assigned to responses in the preference dataset, potentially resulting in undesirable behavior. To counteract this undesired side effect, we examine objectives that incorporate a regularization term to maintain the total length-normalized probabilities of chosen and rejected responses.
To better understand over-optimization, we investigate how changes in response likelihood are distributed over tokens with and without regularization. We find that a significant portion of likelihood changes comes from a small set of outlier tokens, explaining how DAAs can improve generation quality despite decreasing the likelihoods of chosen responses. We apply the proposed regularization to both reference-based (DPO) and reference-free (SimPO) methods and find (1) improved trade-offs between generation quality and general benchmark capability, and (2) enhancements in reward modeling across datasets. For instance, on Llama-3.1-8B-Instruct, we observe a 20% relative increase in AlpacaEval2 scores and a 9% relative performance gain on general benchmarks. Additionally, the added regularization term effectively mitigates the displacement within preferred responses overall, particularly for outlier tokens, by utilizing low-likelihood tokens.
Blogger's Review: By introducing a regularization term, the researchers effectively address the over-optimization issue in DAAs, enhancing the generation quality and preference alignment of LLMs. This approach offers new insights for future model optimization, particularly highlighting the significance of regularization when handling preferred responses.