Soft Q-learning has emerged as a versatile model-free method for entropy-regularised reinforcement learning, optimizing for returns augmented with a penalty on the divergence from a reference policy. Despite its success, multi-step extensions of soft Q-learning remain relatively unexplored, limited to on-policy action sampling under the Boltzmann policy.
This note first presents a formal $n$-step formulation for soft Q-learning, then extends this framework to the fully off-policy case by introducing a novel Soft Tree Backup operator. Finally, we unify these developments into Soft $Q(\lambda)$, an elegant online, off-policy, eligibility trace framework that allows for efficient credit assignment under arbitrary behaviour policies.
Our derivations propose a model-free method for learning entropy-regularised value functions that can be utilized in future empirical experiments.
Blogger's Review: This paper significantly expands the application of traditional soft Q-learning by introducing the Soft Tree Backup operator, providing new insights for off-policy learning, especially in credit assignment problems in complex environments.