Emergence of Small Cognitive Models
Recently, large language models fine-tuned on human behavioral data have emerged as general-purpose cognitive proxies. However, the scale required for this and whether these models process task structure or exploit statistical shortcuts remain open questions. To explore these issues, we trained 14 models with parameters ranging from 135M to 14B across four architecture families on the Psych-101 dataset, which contains 10.7 million trial-level choices from 160 experiments.
Model Training and Evaluation
In-distribution evaluation showed that models of different scales performed similarly, as if they were under a performance ceiling. Models with only 0.6B to 1B parameters were sufficient to match a 70B baseline model on held-out participants. However, out-of-distribution evaluation revealed that larger models had a significant advantage in generalizing to novel task structures.
Diagnostic Analysis
To understand what information these models use, we conducted two diagnostic analyses. First, we progressively stripped four prompt channels—task instructions, experimental stimuli, outcome feedback, and choice history—across 27 experiments and permuted trial order. The results showed that masking the content of stimuli and feedback destroyed 75.7% of learned information and pushed models below chance, demonstrating that choice history alone does not account for performance. Second, permutation revealed invariance on tasks with independent trials but sensitivity where trial order was determined by prior responses.
Conclusion
In conclusion, small cognitive models show promise as noise ceiling estimators for psychological experiments, although their scope remains bounded by the paradigms seen in training. Blogger's Review: This article reveals the potential of small cognitive models in simulating human behavior but also emphasizes their limitations and the need for further research, providing a new perspective and research foundation for cognitive science and artificial intelligence.