Learning through exploration and using newly acquired knowledge to achieve goals is a core capability of general intelligence. Understanding this capability requires tracing how learners build, use, and revise world models during interaction.
We built the first dataset that records how humans discover and employ mechanisms in novel environments, using ARC-AGI-3 as a testbed. The dataset contains 144 trajectories from 64 participants across 25 public games, each pairing time‑synchronized actions with think‑aloud audio, plus 250 trajectories from large language model (LLM) agents with and without harnesses on the same games.
We annotated mechanism triggers and recognition, action purposes, errors and recovery, linking changes in understanding to the observations and actions that support them. Analysis shows that both humans and LLMs anticipate mechanisms and use familiar analogies. Qualitative representations appear in 50.0% of successful human trajectories versus 14.3% of successful LLM trajectories. Humans more often explore unfamiliar situations without a concrete hypothesis, while hypothesis testing is common in both groups.
We also traced how learners recover through belief revision, replanning, and execution correction. As LLMs with well‑designed harnesses reach saturated performance on public ARC-AGI-3 games, our cognitive analyses motivate next‑generation benchmarks that challenge familiar priors, require accumulated evidence and abstraction, and test planning through subgoals, experimentation, and recovery.
The dataset is publicly available at https://arc3-human-think-aloud.github.io/
Review