This paper investigates Reinforcement Learning in Parametrized Action Markov Decision Processes (PAMDP), where each decision comprises a symbolic action and numerical parameters. Traditional Reinforcement Learning algorithms typically utilize one-shot estimators for parameter determination, leading to sample inefficiency. Although explicit but incomplete knowledge (e.g., rules, safety constraints, or expert heuristics) is often available in PAMDP environments, it is rarely leveraged to enhance the sample efficiency of training Reinforcement Learning agents. To address this gap, we propose a novel Neuro-Symbolic Knowledge and Gradient-Guided Reinforcement Learning (KGRL) algorithm.
KGRL employs domain knowledge from a Datalog knowledge base to derive the applicable actions and feasible parameters for a given state. This enables the pruning of non-applicable actions from the decision space and constrains the parameter spaces of the remaining actions. We then implement a gradient-based parameter refinement loop to estimate optimal parameters during the agent's training and deployment. By recording activated rules along the trajectory, KGRL additionally offers local procedural explanations regarding action pruning and parameter constraining. Overall, KGRL guides the agent's exploration and deployment towards feasible and constraint-aware decisions while enhancing sample efficiency during training. KGRL outperforms state-of-the-art RL baselines for PAMDPs in both sample efficiency and episodic return.
Blogger's Review: The KGRL algorithm significantly improves sample efficiency in complex decision-making processes by integrating domain knowledge with gradient optimization, showcasing the potential of knowledge-driven approaches in modern reinforcement learning. This could pave the way for the development of smarter decision-making systems in practical applications.