Guideline‑consistent semantic segmentation demands more than mere category recognition; real‑world labeling policies require fine‑grained, task‑specific decisions. Recent multi‑agent refinement systems improve compliance with textual guidelines by detecting and correcting errors, but they operate statelessly: the feedback from the critiquing agent is discarded after each iteration, causing the same guideline‑specific mistakes to be rediscovered and corrected across the dataset, incurring extra refinement cost.
We introduce InsightSeg, an episodic‑memory mechanism that converts successful correction episodes into reusable, visually grounded insights. A meta‑analyzer distills each qualifying episode into directive natural‑language insights and anchors them to the local image regions that caused the error using patch‑level visual concept vectors.
For subsequent images, these concepts are matched against dense patch embeddings to retrieve relevant insights, which condition the segmenting agent before it makes its first prediction. This shifts the system from correcting recurring errors to preventing them, improving segmentation quality before any refinement occurs.
Experiments on Waymo and Cityscapes show that InsightSeg improves both first‑pass and final guideline‑consistent segmentation performance while requiring fewer refinement steps, demonstrating that multi‑agent refinement can become more accurate and efficient by drawing on past correction experience.
Review