NeFut Logo NeFut
中 Admin Login

[CS.AI] Learning Noise Optimization for Reward Alignment in Few-Step Generative Models

Published at: 2026-11-07 22:00 Last updated: 2026-11-08 12:04
#AI #Machine Learning #optimization

Reward alignment for diffusion‑ and flow‑based generative models is usually defined over stochastic trajectories, which is ill‑suited for few‑step deterministic generators. Initial noise optimization offers a trajectory‑free alternative, yet existing gradient‑free approaches require costly per‑sample search at inference time. We introduce LeNO (Learning Noise Optimization), which amortizes reward‑guided initial noise optimization into a learned residual transport while keeping the pretrained generator frozen. LeNO learns an anchor‑conditioned distribution over noise corrections using Local GRPO and scalar rewards, thereby eliminating iterative test‑time optimization. For Gaussian initial noise with norm‑preserving projection we derive a tractable upper bound on the KL divergence between the corrected noise distribution and the original prior: $$ KL\big(p{\text{corr}}|p{\text{prior}}\big) \le \frac{1}{2}|\Delta|^2, $$ where $\Delta$ denotes the noise correction vector. This bound serves as a principled regularization objective. We evaluate LeNO on few‑step text‑to‑image and protein structure generation, the latter involving expensive black‑box designability rewards. Results show that a single noise correction at inference improves target rewards while largely preserving generation quality and diversity.\n\nReview: LeNO achieves reward‑driven few‑step generation through learned noise correction, balancing computational efficiency with output fidelity, and offers a practical route for rapid fine‑tuning in real‑world scenarios.

Original Source: https://arxiv.org/abs/2611.06255

[h] Back to Home