Learning from human feedback has become a central paradigm for training modern AI systems, where models treat human utility as a reward function for policy learning. Existing approaches usually assume a \emph{universal utility} shared by the whole population and interpret annotator disagreement as random noise. This works for objective tasks but breaks down in subjective domains where preferences vary systematically across individuals.
We address subjective preference learning, assuming observed choices stem from heterogeneous yet internally consistent utility functions. Inspired by rational choice theory (RCT) \cite{tversky1981framing}, we introduce \emph{individuated utility} functions conditioned on both the individual and the decision context, and propose a multi‑stage architecture to estimate them from multimodal data.
Our framework is evaluated on a newly collected dataset containing over $575{,}000$ pairwise aesthetic judgments from $2{,}398$ participants comparing automotive wheel designs. Experiments show that individuated utility models substantially outperform universal utility models, including foundation‑model baselines. The findings demonstrate that annotator disagreement reflects meaningful preference heterogeneity rather than annotation noise.
More broadly, the results highlight the importance of gathering annotator attributes and learning individuated utility functions, enabling reward models that explicitly account for whose preferences they represent and faithfully capture human decision diversity.
Review