Exploration is essential to reinforcement learning (RL) as a policy cannot improve by repeatedly sampling behaviors it already prefers. Standard methods inject stochasticity into the action space, but such jitter yields rollouts close to the original policy.
Escaping a weak policy often requires global perturbations that action noise cannot produce. Large language models (LLMs) and vision-language-action (VLA) models provide a pathway: they condition the policy on a natural language prompt, and since the rollout follows from it, modifying the prompt induces global changes.
The challenge is to find prompts that induce useful global changes. With a weak policy that rarely succeeds, rewards are too sparse to select from. Our idea is to refine prompts from the rollouts themselves: a vision-language model (VLM) reasons over the rollout video, diagnoses how the policy responded, and rewrites the prompt to elicit better behavior next time.
This procedure realizes posterior sampling, a classical RL exploration framework, at the prompt level: the VLM maintains an implicit distribution over useful prompts and updates it from observed rollouts. We call this strategy Prompt-Driven Exploration (PDE). Across manipulation and reasoning tasks, PDE enables RL to learn successful policies even from zero-reward starts and improves sample efficiency more broadly.
Our website is available at Prompt-RL.
Blogger's Review: This study leverages the flexibility of large language models to achieve effective exploration through prompt refinement, showcasing how to overcome traditional methods' limitations in RL. The PDE strategy not only enhances sample efficiency but also points to new directions for future research, making it worthy of further attention.