Multimodal understanding models that can jointly judge text‑to‑image (T2I), text‑to‑video (T2V) and text‑to‑speech (TTS) generation are increasingly employed as “OmniJudges” for evaluation and automatic annotation. Existing benchmarks and training data, however, tend to over‑represent positive examples and conflate distinct failure modes, so a model may score highly without truly recognizing errors, leaving capability gaps hidden.
To address this, we introduce D3‑Omni, a balanced and decoupled benchmark for fine‑grained multimodal understanding. It covers 53 orthogonal binary dimensions (17 for T2I, 22 for T2V, 14 for TTS) and contains 10,671 samples (3,526 T2I, 1,998 T2V, 5,147 TTS). Instead of re‑generating outputs—which could leak information across dimensions—we fix verified fully‑positive seeds and derive negatives via controlled prompt rewriting and atomic, dimension‑isolating perturbations.
D3’s design has three key properties:
- Dual‑balanced: alleviates negative‑sample scarcity and per‑dimension label imbalance;
- Decoupled: each error is attributable to a single capability;
- Dynamic: as generative models improve, construction steers toward under‑represented regions of the label distribution.
The suite achieves near 1:1 per‑dimension parity and a uniform distribution over all total‑score levels. Under this balanced view, even strong OmniJudges struggle on modality‑related dimensions; they confirm satisfied requirements far more reliably than they detect violated ones, and they tend to treat nominally distinct attributes as a single decision. Hence, aggregate accuracy can hide systematic blind spots that a balanced, decoupled lens can expose and help address.
Blogger's Review: D3‑Omni’s rigorous positive‑negative construction and dimension‑wise decoupling provide a more trustworthy diagnostic for multimodal judges. It reminds us that overall accuracy alone is insufficient to gauge true model capability—fine‑grained, balanced benchmarks are essential for uncovering and remedying hidden weaknesses.