Learning representations from multimodal observations is challenging in reinforcement learning due to limited interaction data. The existing Multimodal Joint Predictive Representation combines latent prediction with contrastive learning across modalities, but its recursive rollout depends on earlier predictions and its pairwise alignment only captures correspondence between modalities. We introduce Improved Predictive Representation (IPR), which merges action‑prefix prediction with contrastive multimodal alignment anchored at the observed state. At each horizon $h$, the future representation is predicted directly from the current anchor $z_t$ and the action sequence $a_{t:t+h-1}$:
$$\hat{z}_{t+h}=f(z_t, a_{t:t+h-1})$$
Two cropping configurations are evaluated: IPR‑S retains a minimum crop area of 10%, while IPR‑H retains 50%. Across five DeepMind Control Suite tasks, IPR‑H achieves an average return 10.0% higher than the baseline when using visual and depth images, and 3.4% higher when using visual images and position vectors. The relative performance of the two configurations varies by task. In Webots, both IPR settings yield higher mean final returns than the baseline without LiDAR, but not when LiDAR is available.
Review