In chess positions assessed as essentially equal by strong engines (Stockfish 18 evaluation within 10 centipawns of zero and depth-stable), human results show imbalance. These positions carry outcome skews, which are the gaps between actual game results and player ratings' predictions, exhibiting stable directional properties: some favor White, others favor Black. We reproduce these skews across three different partitions—disjoint player-account sets, time, and disjoint rating bands—and in out-of-sample data eight months later.
On the primary split, we measure each position's skew, and the replication slope assesses how well one measurement predicts another after removing rating and opening-family effects: a slope of 1 indicates complete carry-over; 0 indicates no relation. We find a slope of 0.69 (family-clustered 95% CI [0.65, 0.74]), rising to 0.94 for the most popular and well-measured positions. The slope's value depends on the mix of positions. The invariance of existence survives every tighter evaluation band, search depth, calibration, and popularity cutoff we tested, replicating separately within blitz and rapid formats. The typical skew is small (median $| ext{δ}| ext{≈} 0.018$, about two percentage points of White score), yet it reproduces position by position across disjoint accounts. In these positions, the disadvantaged side also thinks longer. Even where the evaluation is most confident, it is not a sufficient statistic for human outcomes. This result is observational, leaving the causal question for a pre-registered randomized companion study.
Blogger's Review: This study highlights the significant disparities between engine assessments and human outcomes in chess, emphasizing the limitations of machine evaluations. While engines provide balanced assessments, human thought processes and psychological factors play crucial roles in actual gameplay, offering new directions for future research.