Feedback noise is ubiquitous in preference data, stemming from subjective human judgments, annotation errors, and imperfect LLM judges. Training a reward model directly on such noisy data yields a biased estimate of the true risk. Classical denoising methods usually assume a uniform noise ratio across samples, which does not hold for preference feedback where noise is inherently sample‑wise. To tackle this, we introduce CleanRM. CleanRM requires only a few clean preference pairs and uses optimal transport to estimate per‑sample noise ratios, aligning the joint distribution of noisy feedback with that of clean feedback. The reward model is then trained on a noise‑corrected surrogate loss. We theoretically prove that the resulting objective is unbiased with respect to the clean risk in expectation. Extensive experiments show that CleanRM outperforms existing denoising baselines and achieves superior performance in downstream RLHF alignment tasks.
Review