In modern AI systems, pairwise human comparisons serve as a primary interface for learning human preferences. Reinforcement Learning from Human Feedback (RLHF) and related alignment pipelines typically utilize the Bradley--Terry log-odds model, which assumes that choice probabilities are governed by latent reward differences. This paper examines what this assumption misses through a reduced-form model motivated by rational inattention, where each label is generated by a low-capacity evaluation channel.
The model separates two forms of ambiguity that standard reward modeling tends to conflate: a comparison may be difficult either because the two candidates are genuinely close in value or because the relevant distinction is hard to detect under limited attention. We show that limited attention can fundamentally distort what pairwise comparisons reveal. In particular, passive comparison data cannot generally distinguish reward, attention, and default tendencies, and heterogeneous attention can lead standard Bradley--Terry reward modeling to recover misleading rankings.
Our analysis indicates that learning is governed not by the raw number of labels but by the amount of attended information each label carries. A case study on human votes over language-model pairs from Chatbot Arena exhibits the predicted signature: a cyclic component of the comparison data that exceeds sampling noise and that no scalar reward can represent; a second case study on perceptual comparisons shows that response times and gaze carry gap information that the labels do not.
This perspective suggests that human feedback should be treated not as direct revealed preference but as an attention-limited measurement process: a weak preference signal may reflect hidden evaluation difficulty rather than genuine indifference.