In the post-training of large language models, evaluator feedback on policy-induced samples serves as a key mechanism for enhancing model performance. As policies improve, the quality of these sampled responses tends to converge, creating a bottleneck for policy optimization: reduced relative evaluator score gaps result in weak or misleading policy supervision.
We theoretically clarify why these gaps matter through a probability allocation perspective, demonstrating that the directional gain of shifting probability mass from one response to another is exactly the evaluator score gap between them. This identifies relative score gaps as the optimization signals guiding policy updates.
Motivated by this view, we propose DynamicRubric, a response-set-conditioned evaluator-policy co-evolution framework that generates weighted binary rubric items for each candidate set and aggregates the resulting judgments into response-level scores.
In experiments with 8B backbone models, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines using a 70B reward model or a 235B static rubric generator. Policies optimized by DynamicRubric also show improvements on verifiable reasoning and coding tasks.
A DynamicRubric-optimized model has been fully deployed in WeChat Search's AI answering scenario, serving all online traffic across tens of millions of requests per day and improving key online metrics. These results suggest a principle for evaluator-guided post-training: evaluators should evolve with the policies they supervise.
Blogger's Review: The introduction of DynamicRubric presents a novel approach to optimizing large language models by addressing the bottleneck of closely related response quality through dynamic co-evolution of evaluators and policies. Its successful implementation in WeChat Search highlights its practical significance and potential for broader application in the future.