Image diffusion models achieve impressive generation quality, yet they are far harder to control than classical physically based rendering pipelines. We observe an intrinsic link between light transport phenomena and the distribution of latent vectors produced by such models. Accordingly, we embed physically based rendering into the feature space learned by variational autoencoders (VAEs) of generative models, enabling light transport simulation directly in the latent space.
We modify the rendering equation as follows: $$\tilde{L}_o = \int_{\Omega} f_r\,\Phi(L_i)\,\cos\theta\,d\omega$$ where $\Phi$ maps illumination to latent features. Paired with a differentiable renderer, this yields an optimal set of scene parameters that require only minimal refinement to render accurately into the pretrained latent space.
Our method is trained on a single rendered image and demonstrates strong generalization to changes in geometry, lighting, and camera viewpoint.
Review