Abstract
LLM agents operate in external environments where each action alters the state that subsequent decisions depend on, and a single misstep can waste interaction budget or trigger irreversible side effects long before final failure is observed. Reliable deployment thus requires step-level confidence estimation: a calibrated probability that each proposed action is productive, available before the action is executed. Existing LLM confidence estimators are designed to score a response from a given prompt, but agent confidence also hinges on execution consequences: whether similar actions in similar situations actually advanced the task after the environment responded.
We introduce the Critic Experience Bank (abbreviated as CEB), a self-evolving critic framework where an LLM critic accumulates evidence from its own past judgments and their observed consequences. After each trajectory, a hindsight LLM that sees the full execution feedback votes on whether each step was productive. The resulting pseudo-labels populate a memory bank from which related productive and unproductive experiences are retrieved into the critic's prompt whenever a similar step recurs. CEB requires no training and utilizes no ground truth step labels. Across three agent benchmarks and three critic backbones, CEB achieves the best calibration (ECE and Brier) and ranking (AUC) in every dataset-critic combination, reducing ECE by up to $54\%$ relative to the strongest training-free baseline.
Blogger's Review: This research presents an innovative solution for confidence estimation in LLM agents through a self-evolving critique mechanism, enhancing decision reliability. The approach reduces dependency on training data while significantly improving model performance, showcasing the potential and adaptability of LLMs in dynamic environments.