In the field of instance segmentation, large-vocabulary segmentation is constrained by long-tailed category distributions and fine-grained inter-class ambiguity. Data synthesis offers a promising alternative, but current paradigms have complementary limitations: text-to-image (T2I) methods inherit noisy pseudo-labels and struggle with rare classes, while copy-paste methods compromise contextual realism.
To address these issues, we propose a hybrid pipeline that couples T2I generation with context-aware image-to-image (I2I) editing. The T2I branch provides broad category and scene diversity, while a teacher-student scheme ensures label reliability by selectively retaining only prompt-specified categories.
To strengthen supervision for rare classes, we introduce VRAIN (Verified Rare-class Augmentation via INstructed editing), a novel I2I editor. VRAIN inserts high-confidence instances at semantically appropriate locations within in-the-wild scenes, yielding semantically coherent and visually natural edits that reduce domain gaps and enable targeted augmentation.
On the LVIS benchmark, our method surpasses existing baselines, improving overall AP by up to +4.0 points and rare-class AP by up to +9.5 points, while scaling effectively with backbone capacity. Our project page is available at TMI Project Page.
Blogger's Review: This paper innovatively combines T2I and I2I methods to tackle challenges in instance segmentation, showcasing the immense potential of deep learning in image processing. The introduction of VRAIN significantly enhances performance on rare classes, warranting further exploration in real-world applications.