Multimodal large language models (MLLMs) can provide scalable annotations of social and affective content, yet their temporal agreement with human judgments and their predictive value for neural signals require further validation. We compared diffusion‑based and autoregressive MLLMs against continuous human Theory of Mind (ToM) and affective valence ratings collected while watching three short cartoon movies. Behavioral analyses examined time‑aligned Pearson correlations, rating distributions, and the influence of preceding scene descriptions. Each rating set was then used in subject‑specific regression models to predict regional fMRI responses. Although model ratings differed systematically in distribution from human ratings, they captured aspects of human temporal variation, with generally stronger alignment for valence than for ToM. Contextual descriptions improved alignment in certain model‑movie combinations. Notably, in The Present more brain regions showed significant encoding of MLLM ratings than of human ratings, whereas this effect was limited for the other movies. Spatial overlap across rating sources was mainly observed among MLLMs; overlap between human and model ratings was confined to visual, dorsal‑attention, and posterior default‑mode areas. These findings support MLLMs as scalable tools for generating brain‑predictive social and emotional annotations, while highlighting systematic rating differences and model‑ and stimulus‑dependent alignment with human judgments.
Review