Abstract
Image descriptions represented with language models (LMs) predict human brain responses to naturalistic images in high-level visual regions, but the factors driving this predictivity remain unclear. To investigate this, we systematically studied how images are described and which language models are used to embed those descriptions.
For a common set of images, we considered six caption types -- including human-annotated and multiple machine-generated captions -- differing along several dimensions. Each caption was represented with five LMs, spanning autoregressive LMs trained to predict upcoming words and text embedders, i.e., LMs fine-tuned on semantic tasks requiring sentence/document-level representations.
Machine-generated captions yielded significant brain predictivity and alignment, often surpassing human-annotated captions used in previous work. Across caption types, text embedders consistently outperformed autoregressive LMs, a pattern replicated when measuring behavioral alignment with image-similarity judgments.
Analyses of caption representations from different model layers further revealed that both brain predictivity and behavioral alignment peak at intermediate network depth, shortly after a point thought to mark the emergence of syntactic and semantic structure. Altogether, our results demonstrate that both the content of image captions and the LM used to represent them influence brain- and behavior-modelling performance, establishing caption embeddings as a useful tool for studying high-level visual perception.
Blogger's Review: This study provides a deep analysis of the potential of machine-generated descriptions in predicting human brain responses, highlighting the significance of semantic structure. It offers new perspectives on understanding human visual perception, especially in the intersection of deep learning and cognitive science applications.