Rubricon is a pipeline that creates highly discriminative rubrics without human preference labels, manual annotation, or dedicated generator‑model training. It works with any large language model (LLM) by sampling multiple rubric candidates, selecting the most discriminative one, adding criteria to capture missed requirements or response flaws, and pruning criteria that are misaligned or superficial. The system also leverages LLM self‑critique and test‑time scaling, using observed pass rates of criteria on example model outputs to guide further refinement. Across five benchmarks covering professional, consumer, clinical, and general‑purpose LLM tasks, rubrics produced by Rubricon achieve about a 9% higher alignment with labor‑intensive expert‑authored rubrics compared to alternatives such as RRD, QSRG, OpenRubrics, and agentic generation via Codex. On HealthBench, fine‑tuning a Qwen3.5‑9B model with Rubricon rubrics yields roughly a 4.5% performance gain over training with rubrics from the strongest synthetic generator, RRD.
Review