In distributional reinforcement learning, agents learn complete return distributions that are increasingly interpreted for interpretability, risk-sensitive control, and safety monitoring. This paper poses a theoretically anticipated but directly unmeasured question: Are the risk claims of a trained distributional agent true? Our audit combines a decision-relevant screening metric—the excess Wasserstein gap between the top two actions (representing the mass by which first-order stochastic dominance is violated), ground truth from snapshot-restart Monte Carlo, and a statistical harness (permutation nulls, bootstrap refutation, FDR control) without which the audit could yield false conclusions.
Across 33 runs of QR-DQN, C51, and IQN on MinAtar, 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence. The statistical placement of the strongest claims is indistinguishable from truth-blind scenarios, and essentially no claim is confirmable: for these agents, the learned 'risk' reflects a training artifact rather than environmental stochasticity.
The artifact is structural (fully formed early in training, uncorrelated with final score, idiosyncratic to each seed) and appears unchanged at full-Atari scale, with every top Breakout claim of a pretrained near-state-of-the-art QR-DQN refuted. Positive controls of known magnitude confirm 96-100% of real claims (correlation 0.89-0.92): the reading measures the agents, not the audit.
Acting on the heads' CVaR advice at their most-flagged states ranges from beneficial to significantly worse than chance. Neither training for risk nor ensembling removes the artifact, and recalibration passes the audit only by nullifying the claims: the head is uninformative, not merely miscalibrated. We release the toolkit and document two silent pitfalls that produced convincing but incorrect audits of our own.
Blogger's Review: This paper rigorously uncovers the fallacies behind risk claims in distributional reinforcement learning, highlighting the issue of artifacts in training. The findings are crucial for enhancing the reliability of reinforcement learning models and provide valuable insights for future risk-sensitive control approaches.