Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. However, most systems offer limited control over how a garment should be worn — its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions.
First, we define and solve Visual-Instance-Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level segmentation.
Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds segmentation masks as pixel-level control over garment layout, including style, size, and spatial placement on the body.
VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity.
Blogger's Review: The innovation of CtrlVTON lies in merging traditional virtual try-on methods with image editing technology, greatly enhancing user control over garment presentation. This advancement not only improves user experience but also opens new avenues for future applications in fashion technology. The VIP-SAM approach offers fresh insights for instance-level segmentation, warranting further exploration.