Quantile‑based reinforcement learning optimizes a chosen quantile of the cumulative‑return distribution, offering an interpretable way to handle risk‑sensitive decisions. Optimizing a single point quantile is difficult because quantiles can shift abruptly under tiny changes in the return distribution, and exact quantile‑sensitive planning demands costly distributional optimization. Lower‑buffered quantiles smooth this issue by averaging neighboring quantiles just below the target level, providing a smoother surrogate while preserving the original point‑quantile intent. Existing approaches that use this idea remain model‑based, requiring explicit return‑law planning and thus scaling poorly to large or continuous state spaces. We introduce Deep‑BQRL, a model‑free distributional RL framework that brings buffered‑quantile learning to neural function approximation. The algorithm learns conditional return quantiles directly from sampled transitions, builds buffered action scores from the region of the learned quantile function around the target level, and employs ensemble disagreement to guide exploration. An augmented input representation lets the policy exploit trajectory information without explicitly implementing the quantile‑state recursion needed for exact planning. Experiments on an asset‑selling optimal‑stopping problem and the slippery FrozenLake environment compare Deep‑BQRL with the model‑based UCB‑BQRL and tabular PPO and TRPO baselines. Results show that on the asset‑selling task Deep‑BQRL achieves smaller mean cumulative point‑quantile policy gaps than PPO and TRPO at the reported target quantiles, while UCB‑BQRL retains the smallest gaps. The learned stopping decisions vary with the target quantile, providing an intuitive illustration of risk‑sensitive behavior.
Review