Scalable Vector Graphics (SVG) generation has attracted growing interest as generative models become more expressive and controllable. Progress is hampered by the lack of domain‑specific evaluation protocols; the common practice relies on metrics such as CLIPScore, which were trained on natural images and only partially align with human judgment. We first conduct controlled caption and image perturbation experiments and show that CLIP‑based scores barely react to typical SVG generator errors—wrong colors, counts, and spatial relations—while off‑the‑shelf Vision‑Language Model (VLM) judges are more sensitive but respond unevenly across error types and SVG styles. We then introduce a human‑annotated dataset for Semantic Alignment that measures how faithfully a generated SVG reflects its caption. Building on this dataset, we develop two complementary evaluators: (1) CLIP scorers adapted to vector graphics and aligned to human preferences for fast large‑scale evaluation, and (2) a VLM judge trained via supervised fine‑tuning and reward‑shaped reinforcement learning for more expressive and interpretable assessment. Using both evaluators, we benchmark major open‑source, commercial, and optimization‑based SVG generators on an independent caption set.
Review