Model failures often expose gaps in the behavioral criteria that guide training. RubricOpt introduces a framework that lets a policy revise these criteria during preference optimization. The policy first diagnoses its rubric‑free outputs, retains requirements already covered by the rubric, and edits clauses that repeatedly cause failures. A shared semantic representation links task demands, preference‑data contrasts, and rubric clauses, while a deterministic gate checks evidence references and fixed semantic anchors. During training, the rubric guides Direct Preference Optimization (DPO) with preference pairs, labels, and the objective kept fixed; the rubric is removed at evaluation. The analysis relates the local effect on rubric‑free success to semantic alignment and a residual beyond measured factors. Experiments on capability and open‑ended generation benchmarks show that conditioning DPO on a rubric can improve outcomes over ordinary DPO, though gains and regressions vary across models and datasets. A matched continuation isolates the contribution of revision from extra training and finds mixed incremental effects for the tested edit, whose feedback overlaps the evaluation set. RubricOpt thus brings the behavioral specification itself into the policy’s feedback loop.
Review