Q-learning is a fundamental algorithm in reinforcement learning (RL) for solving discounted Markov decision processes (MDPs) when the transition kernel is unknown. The deep Q-network (DQN) extends Q-learning by using a deep neural network for Q-function approximation, making Q-learning applicable to more practical high-dimensional problems.
Dueling Q-learning decomposes the Q-function into a value function and an advantage function, learning both components jointly, which enhances learning efficiency. However, the theoretical understanding of dueling Q-learning is still limited.
Recent work has initiated an analysis of tabular dueling Q-learning, but existing guarantees focus on a regularized formulation, leaving the pure tabular update less understood. This paper strengthens that line of analysis by providing a direct interpretation of the centered tabular decomposition and establishing convergence guarantees for the unregularized, unprojected constant step-size recursion.
In particular, we derive an exact switching linear system representation for deterministic dueling Q-learning and a finite-time error bound in expectation for the sampled stochastic version. The analysis clarifies how the value and advantage updates act as different gains on the action-common (value function) and action-differential (advantage function) components of the Q-function.
Blogger's Review: This paper significantly contributes to the theoretical foundation of dueling Q-learning, especially concerning convergence analysis without regularization. By clarifying the roles of value and advantage updates, the findings enhance the understanding and efficiency of Q-learning algorithms in practical applications.