Food similarity underlies retrieval, recommendation, substitution and other food‑computing tasks, yet most visual representations are evaluated only by identity preservation.
This work asks what happens beyond identity: how various visual embeddings arrange foods into relational neighborhoods, what culinary relationships these neighborhoods reveal or hide, and how consistently such structures appear across embeddings.
We employ an expanded Khana dataset and seven pretrained visual models that span convolutional, transformer, self‑supervised and image‑text aligned approaches.
Experiments cover image‑to‑image retrieval and image‑to‑dish neighborhood analysis, focusing on culinary‑family structure, contrast cases and cross‑model agreement. We also aggregate image embeddings to the dish level and compare visual similarity with recipe‑derived similarity for the same dishes.
Findings show that image‑level neighborhoods differ markedly across models, often crossing culinary‑family boundaries. At the dish level, aggregation reveals a common visual structure, but this structure aligns weakly with recipe‑based similarity.
The takeaway is that recognition performance alone cannot characterize relational structure; food similarity depends on the representation, modality and level of abstraction, and understanding these structures is a prerequisite for fusing multiple food information modalities into a coherent similarity notion.
Review