Abstract
Do vision models see colors the way humans do? Existing evaluations of color representations usually compare them with geometric spaces like CIELAB or discrete color labels. These references capture perceptual distance or category membership but do not reflect the nuanced way people organize colors.
We evaluate color grounding against a fuzzy perceptual model with 86 graded categories fitted to human survey data. The framework can be applied to any image encoder and measures three complementary properties: category boundaries, category compactness, and graded alignment beyond what color geometry alone can explain.
Across eleven Vision Transformer encoders, category-level results are broadly similar, whereas graded alignment differs substantially. Masked Autoencoders (MAE) achieve the strongest beyond-geometry alignment, with confidence intervals that do not overlap those of the other encoders. A layer-wise analysis further shows that masked reconstruction preserves this structure toward the output.
On natural images, MAE represents surface color globally, while language-supervised models encode color more strongly in relation to the foreground object. These results show that human-like color grounding has several distinct aspects that should not be reduced to a single score.
Blogger's Review: This study delves into the differences between vision models and human color perception, emphasizing the importance of multi-dimensional evaluations. By introducing a fuzzy perceptual model, the authors provide a new perspective for future visual system designs, pushing the boundaries of progress in the field of computer vision.