Online reinforcement learning (RL) has achieved remarkable success in decision‑making tasks, yet it remains highly sensitive to the quality of training experience. Data‑centric RL therefore requires a way to quantify the contribution of each training sample. Existing attribution methods focus only on the local influence within a single training round and ignore non‑local effects across multiple rounds.
We formalize data attribution as a trajectory‑level leave‑one‑out quantity (RL‑LOO), which measures how a particular training sample influences a downstream target after subsequent online updates. From this definition we derive RL‑IF, a first‑order estimator that propagates influence through both optimization effects and policy‑induced sampling effects, and we provide theoretical guarantees on its approximation error under smoothness assumptions. Our analysis further shows that, although both effects are modeled, the estimator can often be well approximated by considering the optimization effect alone.
Leveraging this insight, we propose a practical data‑filtering framework called RINSE. In standard control benchmarks and an RLHF‑style toxicity‑mitigation setting, RINSE improves training efficiency and final performance compared with PPO and local attribution baselines.
These results demonstrate that RL‑IF offers a principled and practical framework for understanding and improving online RL.
Review