We propose D2PO (Dynamic Direct Preference Optimization), a principled framework for optimizing diffusion sampling policies concerning timestep schedules and classifier-free guidance (CFG) weights. The motivation behind our work is a fundamental limitation of existing student-teacher regression frameworks. Low-NFE student samplers are trained to mimic high-NFE teachers, often sacrificing high-frequency texture fidelity while preserving coarse global structures, thereby misaligning the sampler with perceptual quality. D2PO addresses this challenge by reformulating sampler optimization as a preference-based alignment problem, leveraging the Direct Preference Optimization (DPO) framework.
To make DPO applicable to diffusion samplers, we model the sampling policy as an energy-based model (EBM), transforming preference comparisons into tractable energy differences. Furthermore, we introduce a novel energy formulation derived directly from the pretrained score network, enabling preference evaluation in perturbed spaces that jointly capture structural consistency and fine-grained details. We also introduce dynamic preferences, where the preferred samples used for alignment progressively improve as the sampling policies are learned. This self-improving mechanism replaces rigid static teacher supervision with an iterative, preference-guided refinement process, providing progressively stronger alignment signals.
Extensive experiments demonstrate that D2PO aligns diffusion samplers with perceptual quality more faithfully, unlocking the full potential of high-quality teachers and consistently outperforming conventional regression-based schedulers under low-NFE constraints.
Blogger's Review: The D2PO framework significantly enhances the performance of diffusion samplers through its dynamic preference mechanism, addressing the shortcomings of traditional methods in capturing high-frequency details. Its energy-based model design is not only innovative but also provides a fresh perspective for future sampler optimization, warranting further exploration and application.