Abstract
Fine-tuning large diffusion models for new domains or styles involves a trade-off: improving target-specific generation often degrades the pretrained model's broad generative capability. Existing full and parameter-efficient fine-tuning methods typically handle this trade-off only implicitly.
In this work, we propose a novel source-prior-driven selective adaptation method to efficiently fine-tune diffusion models, achieving a favorable trade-off. Our method relies on two key observations:
- The loss of general generative capability is highly inconsistent across pretrained parameters;
- Parameters that have a relatively small impact on the model's general generative capability remain structurally inconsistent across layers and parameter types.
Motivated by these observations, we first learn a static mask to explicitly identify parameters better suited for downstream adaptation, and then construct structured update strategies for the selected subset. Experiments show that our method achieves a better adaptation-retention trade-off than existing strong baselines.
Blogger's Review: The source-prior-driven selective adaptation method proposed in this paper provides a novel approach to the challenges of fine-tuning diffusion models. By explicitly identifying adaptable parameters and combining it with structured update strategies, it showcases how to optimize target-specific generation while preserving broad generative capabilities. This method may drive further research and applications in related fields.