In this paper, we study finite-horizon Markov Decision Process (MDP) planning under root-based (resolute) risk objectives. These objectives apply a rank-dependent functional to the distribution of total returns, introducing non-linearity that generally breaks Bellman optimality, making direct optimization through scenario-tree enumeration intractable.
To address this, we propose a novel method called ERQDP, which is enumeration-free and sampling-free, solving a rank-quantile surrogate via exact Dynamic Programming (DP). This method evaluates candidate policies exactly by DP over return Probability Mass Functions (PMFs) on a discretized return grid, with an explicit rounding bound, and refines the surrogate in an anytime loop that reports an explicit upper-lower gap (certificate) for the target objective up to discretization budgets.
Across tested benchmarks, ERQDP returns certified solutions or explicit residual gaps, enables fast risk-parameter sweeps with substantial runtime gains, and supports both risk-averse and risk-seeking behaviors.
Blogger's Review: This research introduces the ERQDP method, providing an effective approach to tackle complex risk decision-making problems, overcoming the limitations of traditional methods, and holds significant theoretical and practical implications.