This study explores a striking feature of the human visual system: it ingests visual information through a series of local, foveated glimpses rather than a single global computation. This characteristic makes human vision distinctly different from most popular computer vision models today, which typically input images globally in a single shot. Thus, a natural question arises: do local, sequential vision models provide any fundamental computational benefits beyond being more biologically plausible than global models?
We investigate this question from the perspective of visual state tracking and length generalization. Inspired by recent studies on length generalization in language models, we examine the behavior of vision models trained on simple visual tasks requiring the aggregation of local information across an image. Our experiments reveal that, similar to language models, vision models can exploit global shortcuts, leading to failures in generalization over task length or complexity.
Moreover, we show that recurrent vision policies based on strictly local perception can mitigate these failures, thereby enabling models to generalize on these tasks. Our results indicate that local attention may be an essential overlooked requirement for robust compositional generalization.
Blogger's Review: This article reveals the potential of local visual processing in computer vision, particularly regarding generalization capabilities in complex tasks. The research emphasizes biologically-inspired model design, which could lead to the development of more efficient visual reasoning systems. By introducing local attention mechanisms, it may be possible to overcome current models' limitations in complexity, warranting further exploration.