Abstract
Text-guided medical image segmentation leverages clinical semantics to improve lesion delineation. However, many existing models tightly couple cross-modal fusion, supervision, and decoder design into a task-specific architecture. This tight coupling complicates the reuse of language guidance modules across heterogeneous vision and text backbones, often necessitating network redesign when the encoder pair changes.
This paper presents BTHA, a backbone-transferable hierarchical adapter framework for text-guided medical image segmentation. BTHA is built around a stable feature-level interface: given multi-scale visual features and a text representation, it injects semantic guidance through shape-preserving adapters while maintaining the decoder-side tensor contract.
To make this interface effective, we introduce a Hierarchical Coarse-to-Fine Supervision Strategy that decomposes learning into global image-text alignment, multi-scale auxiliary localization, and boundary-aware final mask refinement. Additionally, we design a Scale-Adaptive Gated Semantic Guidance (SAGSG) adapter, where resolution-specific gates adaptively control textual injection, and channel recalibration suppresses redundant cross-modal responses.
Evaluations across diverse vision and text backbones show that the same adapter and supervision design remains effective across convolutional and transformer-based visual encoders as well as different language encoders. Experiments on four public datasets further demonstrate that BTHA improves strong text-guided baselines with modest computational overhead.
Blogger's Review: The BTHA framework presented in this paper significantly enhances the flexibility and efficacy of text-guided medical segmentation by decoupling language guidance from backbone models. This approach not only boosts performance but also minimizes the need for redesign, paving the way for future research in medical image segmentation.