Large language model (LLM) agents are increasingly expected to interact with environments and improve their decisions during deployment. Existing studies have paid little attention to how future outcome prediction and accumulated interaction experience can jointly support decision making.
We introduce a test‑time framework inspired by Model Predictive Path Integral (MPPI) control that merges multi‑branch predictive analysis with experience‑based adaptation. At each step the agent proposes several candidate actions and imagines their subsequent observations with a world model, allowing comparison without actual execution.
Because language environments lack explicit dynamics and cost functions, we evaluate the imagined outcomes for task progress and risk, obtaining a cost $S_i$ for each candidate. MPPI‑style weights are then computed as $$ w_i = \frac{\exp\big(-\frac{S_i}{\lambda}\big)}{\sum_j \exp\big(-\frac{S_j}{\lambda}\big)} $$ which guide the final action selection.
A dual‑branch experience memory enables self‑improvement: the planner branch refines candidate generation and action selection, while the world‑model branch improves next‑observation prediction. The two branches continuously update each other during interaction, enhancing both candidate quality and predictive accuracy.
Experiments on ALFWorld and ScienceWorld with three different backbone models demonstrate that our method achieves strong overall performance and validates the effectiveness of experience‑guided predictive control.
The code will be released publicly.
Review: This work creatively transfers MPPI control ideas to language‑based environments, showing that combining prediction with experience can yield self‑improving LLM agents and offering a promising direction for deployable AI systems.