We discovered that even the simplest visual symbols, such as a red heart, can markedly steer vision-language models (VLMs) on tasks that appear unrelated. For instance, inserting a heart cue into a basic coding problem nudges the model toward recursive solutions. Crucially, this steering is largely implicit: the model's explicit interpretation of the visual symbol diverges sharply from its observed behavior.
A "generation‑interpretation gap" emerges in visual communication between models. While models tend to emit shape primitives that feel intuitive to both themselves and humans, these cues often fail to guide the receiving model effectively.
In extended communication games involving abstract concepts and narratives, models creatively employ visuals to convey meaning to one another, although some of these exchanges remain non‑intuitive to humans.
These findings suggest that VLMs are capable of rich, creative, and visually grounded interaction, yet their production and interpretation of visual meaning differ from human conventions in surprising ways. The work highlights both the promise and the challenges of visual communication between humans and VLMs.
Review