Abstract
Direct Preference Optimization (DPO), an efficient alternative to PPO-based RLHF, struggles with knowledge-intensive generation: standard preference signals from human annotators or LLM judges exhibit a systematic verbosity bias that rewards fluency over logical correctness. This blindspot creates a logical alignment gap—SFT models achieve only 0.05-0.22 in NLI entailment despite producing fluent text.
We propose RLearner-LLM with Hybrid-DPO: an automated preference pipeline that fuses a DeBERTa-v3 NLI signal with a verifier LLM score, eliminating human annotation while overcoming the "alignment tax" of single-signal optimization.
Evaluated across five academic domains (Biology, Medicine, Law) with three base architectures (LLaMA-2-13B, Qwen3-8B, Gemma 4 E4B-it), RLearner-LLM yields up to 6x NLI improvement over SFT, with NLI gains in 15 of 15 cells and consistent answer-coverage gains. On Gemma 4 E4B-it (4.5B effective params), Hybrid-DPO lifts NLI in four of five domains (+11.9% to +2.4x) with faster inference across all five, scaling down to compact base models without losing the alignment-tax mitigation.
Our Qwen3-8B RLearner-LLM wins 95% of pairwise comparisons against its own SFT baseline; conversely, GPT-4o-mini wins 95% against our concise output—alongside the 69% win the same judge gives a verbose SFT over our DPO model, this replicates verbosity bias on a frontier comparator and motivates logic-aware metrics (NLI, ACR) over LLM-as-a-judge for knowledge-intensive generation.
Blogger's Review: RLearner-LLM effectively addresses the conflict between fluency and logic by introducing Hybrid-DPO, showcasing its advantages in knowledge-intensive generation tasks. The strategy based on DeBERTa-v3 offers significant insights for further optimization of large language models, making it worthy of attention and in-depth study.