This study investigates how looped transformers encode human preferences, training lightweight evaluator heads on frozen Ouro-2.6B loop-iteration states based on Anthropic HH-RLHF. A post-publication audit revealed that three headline results were inflated due to two independent evaluation errors.
The 95.2% pairwise evaluator accuracy was a canonical-ordering artifact: the data were correctly split, but the evaluator learned to prefer the first-presented argument; its strict antisymmetrized accuracy on the full 8,552-pair test set is 63.9%. The 84.5% pairwise probe and the below-chance 21.75% pointwise probe were due to source-item leaks (orientation rows and pair partners crossing the train/test split); corrected pair-disjoint values are 56.5% and 54.2% — above chance, leading to the withdrawal of the "reverse polarity" finding.
The central finding remains but is substantially reduced in magnitude: relative decoding is more accurate than pointwise decoding (pairwise +2.3 points, 95% confidence interval [+1.3, +3.3]), antisymmetrized evaluators still outperform linear probes, but no corrected readout matches the end-to-end reward model.
Methodological findings still hold (constant-output degradation, flip tests, swap protocol metric inflation), but with one correction: antisymmetrized accuracy, not antisymmetrized correlation, demonstrates relational discrimination. The two errors are mutually invisible — split audits cannot see ordering priors, antisymmetrization cannot see leaks — thus both checks are needed. Follow-up work will conduct a comprehensive audit (Kirin, 2026, in preparation).
Blogger's note: This study reveals the complexity of preference encoding in looped transformers, highlighting subtle evaluation pitfalls and potentially misleading results. The researchers' rigor in auditing and correcting errors underscores the importance of comprehensive review in model evaluation.