Vision-language models excel in video captioning but often fail to capture individual viewer attention effectively.
We propose VEGAS (Video Caption Evaluation via Gaze Score), a training-free metric that leverages test-time gaze to sample personalized, attention-aligned text. This is a cross-modal, information-theoretic metric that quantifies how well a candidate caption aligns with a viewer's focus.
To evaluate VEGAS, we curated a dataset of egocentric activities and instructional slides paired with synchronized gaze and reference annotations. We then selected captions based on VEGAS using rejection sampling without the need for model retraining.
Experiments show that VEGAS-selected captions significantly better align with human focus and improve downstream caption-to-video retrieval, demonstrating the practical utility of incorporating viewer attention during inference.
Blogger's Review: The introduction of VEGAS offers a fresh perspective on video caption generation, highlighting the importance of viewer attention in enhancing caption quality. Its strong potential in personalized recommendations and retrieval tasks suggests that future research could further explore how to better understand and leverage gaze information from viewers.