NeFut Logo NeFut
Admin Login

[CS.AI] Language-model groups overstate consensus when replaying human deliberation on a reasoning task

Published at: 2026-09-19 22:00 Last updated: 2026-09-20 12:54
#AI #Machine Learning #LLM

Full-consensus rates are often taken as signals of collective cognition, yet they depend on how participation and final states are defined. We replayed one hundred held‑out human Wason groups and matched them with large language model (LLM) agent groups of the same size. For each participant we seeded a belief‑anchored agent using that participant's pre‑discussion answer, then scored humans and agents with identical code. Across human scoring definitions, full‑consensus ranged from 24.0% to 57.0%; about one‑fifth of participants never posted, whereas agents almost always did. Two post‑unblinding sensitivity analyses showed that agent groups remained more consensual: the submit‑based comparison (n = 98) yielded gaps of 34.0 and 43.9 percentage points for chat and reasoning modes, and the participation‑matched comparison (n = 45) yielded gaps of 34.1 and 44.4 points. These routes reduced different measurement asymmetries yet converged within 0.5 points. The gap persisted without early stopping and after re‑parameterizing to remove the memorizable answer; reasoning‑mode groups then agreed nearly unanimously, mostly on incorrect answers. Simulated consensus did not track collective accuracy, and belief‑anchored agent groups were biased estimators of the human outcome distribution in this setting. These analyses provide a scoring‑explicit basis for assessing simulated‑group estimates of human deliberative outcomes.

Review

Original Source: https://arxiv.org/abs/2609.20543

[h] Back to Home