Reinforcement learning from human feedback (RLHF) faces significant feedback inefficiency when applied to diffusion models. Traditional methods often demand extensive human or reward model evaluations, limiting practicality. This paper introduces two complementary strategies that substantially enhance the feedback efficiency of diffusion RLHF while maintaining generalization to unseen prompts.
Firstly, we note that reward information in diffusion trajectories is unevenly distributed: not all denoising timesteps or trajectories contribute equally to learning. To address this, we emphasize informative timesteps and trajectories during optimization, leading to more effective gradient updates. We introduce a per-timestep weighting scheme that reweights denoising steps during policy optimization. Theoretically, we connect this weighting to the optimal convergence properties of proximal policy optimization (PPO) and empirically approximate the resulting weighting trend.
Secondly, we propose a replay mechanism that prioritizes informative trajectories, enabling the model to reuse past samples instead of repeatedly querying new rewards. Together, these strategies significantly enhance the feedback efficiency of diffusion RLHF. Under identical hyperparameter settings, our approach achieves up to a 6× improvement in sample efficiency compared to widely used diffusion RLHF baselines.
Blogger's Review: This paper addresses the pressing issue of feedback inefficiency in diffusion RLHF with innovative weighting and replay mechanisms, showcasing the potential for improving sample utilization in practical applications and providing valuable insights for future research directions.