In cooperative AI evaluation, traditional benchmarks focus on explicit information, while human teamwork relies on implicit conventions—shared protocols that go beyond literal messages. We define the convention gap as the difference between the failure probability predicted from the literal content of communication and the observed failure rate, providing a metric for implicit communication.
In the card game Hanabi, the finite deck and deterministic hint rules make this posterior exactly computable. We replayed roughly 101,000 actions from three public datasets: human‑human (hanab.live), AI‑AI (HOAD), and human‑AI (HanabiData).
The gap was +26.2 percentage points (pp) for human pairs, –0.7 pp for AI pairs, and +16.4 pp for human‑AI pairs, with the largest contribution coming from cards that received no hints (+46 pp in human pairs).
Within human‑AI play, the literal information available to humans was similar across the three AI partners (predicted failure 38%‑41%), yet actual human failure rates ranged from 14.4% to 34.4%, and the gap varied from +24.1 pp down to +6.2 pp; the partner that produced the largest gap also yielded the fewest human errors.
Game scores carried different information, depending on each corpus’s roster composition, whereas the convention gap separated human from AI performance at the agent level.
As a known‑answer check, Off‑Belief Learning agents showed a gap of +1.6 pp at the convention‑free level, increasing monotonically to +21.7 pp as convention content grew.
These results suggest that convention compatibility may predict an AI’s effectiveness with human partners better than raw AI‑AI performance.
Review