Accurate segmentation of polyps and skin lesions is essential for clinical diagnosis, yet current approaches often suffer from low contrast, vague boundaries, and cross‑domain distribution gaps. Most discriminative networks and diffusion‑based segmenters output standalone binary masks, leaving the visual priors of large‑scale pretrained generative models untapped. InstEditSeg reformulates medical segmentation as an instruction‑driven image editing problem. Instead of producing a mask, the model renders a color‑coded overlay on the original image conditioned on a textual instruction, aligning the edited output with the natural image distribution learned by latent diffusion models and reducing the domain gap between natural and medical imagery. To recover fine anatomical details, DINOv3 is employed as an auxiliary visual encoder and a DINO Feature Guidance Block builds a multi‑scale feature pyramid. The pyramid is fused into the diffusion U‑Net via channel concatenation and zero‑initialized convolution, injecting hierarchical discriminative priors without perturbing pretrained weights. A dual‑branch classifier‑free guidance strategy requires only two forward passes per denoising step, cutting inference cost. On polyp and skin lesion benchmarks the framework attains accuracy comparable to strong discriminative baselines while showing concrete generative benefits: better cross‑domain generalization on unseen data, more complete multi‑lesion segmentation, instruction‑conditioned task control, and sampling flexibility. Analysis reveals sensitivity to color and lack of attribute‑conditioned selection. Code is released.
Review