Reinforcement learning (RL) is essential for adaptive decision making such as robust meta‑RL, domain randomization, and post‑training of large language models, where tasks are usually expressed as Markov decision processes or prompts.
A central bottleneck is the interaction cost required for task selection and policy optimization. Model predictive task sampling (MPTS) improves sample efficiency by querying informative tasks through a risk predictive model (RPM) trained on optimization history.
Existing RPMs suffer from two failure modes: (i) sparse optimization histories limit generalization across the task space; (ii) a one‑step lag in risk approximation introduces bias as the policy evolves.
We formalize these issues and derive a lagged‑risk decomposition $$R_t = \hat{R}_{t-1} + \Delta_t$$, exposing the temporal bias, and recast risk prediction as an in‑context sequence modeling problem.
The proposed RPM incorporates a one‑step look‑ahead mechanism and a temporally weighted sliding window to mitigate data non‑stationarity and scarcity, all without extra rollouts.
Empirical results on robust meta‑RL, domain randomization, and prompt‑curriculum RL for LLM post‑training show more reliable difficulty estimates and consistent performance gains.
Review