Transferring policies across domains poses a vital challenge in reinforcement learning due to the dynamics mismatch between the source and target domains. This paper considers the setting of online dynamics adaptation, where policies are trained in the source domain with sufficient data while only limited interactions with the target domain are allowed. Existing works address the dynamics mismatch using domain classifiers, value-guided data filtering, or representation learning.
Instead, we study the domain adaptation problem from a generative modeling perspective. We introduce DADiff, a diffusion-based framework that leverages the discrepancy between source and target domain generative trajectories to estimate the dynamics mismatch in the next state generation process. Both reward modification and data selection variants are developed to adapt the policy to the target domain.
We provide a theoretical analysis showing that the performance difference of a given policy between the two domains is bounded by the generative trajectory deviation. Extensive discussions on the applicability of the variants and connections between our theoretical analysis and prior work are provided.
We conduct extensive experiments in various environments to validate the effectiveness of our method, which demonstrates superior performance compared to existing approaches in addressing the dynamics mismatch. The code is available at GitHub.
Blogger's Review: DADiff offers an innovative perspective on addressing the dynamics mismatch challenge in reinforcement learning through generative modeling. Its theoretical analysis and experimental results highlight the effectiveness of this approach, making it a significant contribution worth further exploration and application in the field.