Human social perception unfolds incrementally: we infer intent before an utterance finishes, revise our interpretation as new cues appear, and attend to facial, gestural, or prosodic signals that shape meaning. Existing multimodal LLM benchmarks mainly test final‑answer understanding on complete episodes, rewarding post‑hoc recognition rather than the real‑time perception needed for interaction.
Inspired by interactional Gestalt principles, we introduce GesBench, a human‑grounded benchmark for cue‑level social perception in multimodal LLMs. GesBench uses cumulative sub‑utterance clips to measure when models form stable interpretations of speaker intention and social affordance, and employs human‑gaze‑guided corruption to test whether regions attended by humans are functionally important for model judgments.
Built from over 6000 utterance‑level clips drawn from scripted and unscripted conversations, GesBench provides human annotations of unfolding judgments and gaze for a representative subset. Across open‑ and closed‑source multimodal models we observe a clear human‑model gap: humans make early predictions and revise frequently, whereas models often hedge or repeat across clips. GesBench shifts evaluation from whether models eventually recognize meaning to when and where that meaning emerges.
The code is publicly available at https://anonymous.4open.science/status/GesBench-share-212C.
Review