In this paper, we explore how artificial agents can learn to switch between fast reactive decision-making and slower deliberative planning, known as meta-reasoning. We model reactive decision-making as a policy that directly maps state observations to actions. Such policies can be trained using reinforcement learning (RL) or imitation learning, but they may generalize poorly outside their training distribution.
In contrast, model-based decision-time planning is more likely to yield good actions across a broader set of states but requires additional computation time, leading to delays in action. Therefore, we introduce an RL method for training a meta-reasoning policy that allocates computation based on a reactive-policy uncertainty score. This score enables the agent to predict when the reactive policy is likely to perform poorly and when planning is necessary.
We conduct an empirical study in motion planning and navigation environments, demonstrating that this design allows the meta-reasoning policy to learn when the reactive policy provides a good enough action versus when decision-time planning is required. Furthermore, we show that our design enables the meta-agent to shift towards fully reactive control as the reactive policy improves.
Blogger's Review: This study provides significant insights into the decision-making processes of AI, particularly in balancing rapid responses with thoughtful planning in dynamic environments. By incorporating an uncertainty score, the agent can allocate computational resources more intelligently, which is crucial for enhancing decision efficiency. This approach holds wide potential for real-world applications, especially in complex navigation tasks.