Generative recommender systems are increasingly leveraging Reinforcement Learning (RL) to optimize outcomes that may only manifest after hundreds or thousands of interactions. Conventional RL propagates reward based on temporal proximity rather than explicit relevance, which attenuates the signal for distant yet crucial actions and leads to sample inefficiency.
We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to pinpoint earlier events that plausibly contributed to a realized outcome. EWM instantiates a frontier language model to generate explanations and employs a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return.
To investigate what constitutes a useful explanation, we operationalize three complementary variants:
- Actual causation inspired by Halpern;
- Explanatory simplicity from Lombrozo;
- Hard‑to‑vary explanations as described by Deutsch.
We evaluate EWM on two controlled long‑horizon RL benchmarks (Crafter and WordCraft), a public sequential‑recommendation benchmark (KuaiRand), and an industry‑scale generative recommender. Results show that:
- On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with the largest gains when rewards are most delayed;
- On KuaiRand, EWM beats SFT and LaRe offline and A2C online;
- In an offline evaluation of an industrial generative recommender, EWM markedly improves Recall and raises behavioral proxies.
These findings suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.
Review