Learning from experience is essential for LLM agents to adapt to unfamiliar and dynamic environments. Evaluating this ability helps gauge how well agents acquire and apply new knowledge. Existing benchmarks mostly focus on tasks whose rules are supplied in the instructions or already familiar to pretrained models, making it hard to separate interactive learning from reasoning with prior knowledge. To address this, we introduce Learn2Play Bench, a suite of newly designed text‑based games whose rules are novel or counter‑intuitive, forcing agents to obtain knowledge through interaction rather than relying solely on pretraining. The games provide reproducible feedback and automatic scoring, enabling controlled evaluation of learning over repeated attempts. We also vary game instances to test whether agents can transfer what they have learned to new situations. Using this setup we examine the impact of different backbone models, self‑evolving methods, and agent harnesses on learning ability, revealing three findings: (1) Experience retention: keeping complete records of actions and feedback supports more effective learning than summarizing them into rules or strategies. (2) Human‑agent gap: top‑performing human players achieve higher peak scores than all evaluated agents, explore a wider range of strategies, and repeat actions less often. (3) Harness matters: with the backbone fixed, switching the harness can improve performance while reducing estimated inference cost. These insights shed light on how LLM agents learn from experience and suggest directions for future work to enhance their learning capability. Project website: https://liushiliushi.github.io/learn2play-bench-website/
Review