Reinforcement learning is often used to train language agents in interactive environments, yet it cannot be applied directly when rewards are missing. Recent approaches treat environmental feedback as privileged context for hindsight self‑distillation, but conditioning the teacher on feedback alone proves insufficient. We introduce the SELF framework, which jointly optimizes environmental feedback modeling and hindsight self‑distillation. SELF learns to predict how the environment responds to actions while distilling guidance from a feedback‑conditioned self‑teacher into the policy. Our analysis reveals a mutually reinforcing mechanism: feedback modeling strengthens hindsight supervision and policy learning, and self‑distillation improves the model's ability to predict feedback. Experiments with Qwen3‑8B show that SELF raises the $$\tau$$‑bench success rate by 6.4% and 4.1% over SDPO and GRPO, and improves AppWorld task goal completion by 10.71% and 3.57% respectively. These results demonstrate that SELF leverages environmental feedback more effectively, enhancing agent capabilities.
Review