As vision-language models (VLMs) increasingly demonstrate the ability to interpret visual scenes, it remains unclear whether they can leverage this information for strategically effective decision-making. We investigate this question in the context of soccer, where models observe the seconds leading up to an on-ball decision and must choose whether to shoot or pass to a specific teammate. Unlike conventional visual understanding tasks, soccer allows decisions to be quantitatively evaluated by estimating the value of each available action.
We introduce SportD, a benchmark comprising 478 on-ball decisions from the 2022 FIFA World Cup. Each model choice is evaluated against a possession-value model that estimates the action that most increases the attacking team's probability of scoring. This enables us to measure both optimal-action accuracy and the value forfeited by suboptimal decisions. Among three frontier VLMs, the best model selects the highest-valued action on 31.4% of events, compared to 38.9% for professional players, with all models incurring significantly greater regret.
Further analysis reveals a systematic preference for lower-variance and lower-reward actions: VLMs shoot less often and select substantially less progressive passes than either the optimal policy or real players. Moreover, these models replicate specific player actions above chance, even when those actions are suboptimal, suggesting partial imitation of familiar play patterns rather than consistent evaluation of counterfactual alternatives. SportD provides a value-grounded testbed for measuring physical strategic reasoning in VLMs.
Blogger's Review: The introduction of the SportD benchmark effectively explores the capabilities of vision-language models in complex decision-making environments. While current models still fall short in strategic decision-making, this research provides a clear direction for future improvements, emphasizing the importance of higher-level strategic evaluation and simulation.