Reinforcement‑learning environments are now a primary lever for improving large language model (LLM) capabilities after pre‑training, yet most agent benchmarks remain static: the world only advances when the agent acts, rewards are terminal verdicts, and pass thresholds are set arbitrarily. StoreBench addresses this gap by providing a live‑commerce setting where an agent runs a mid‑size online apparel store on a production‑grade backend, testing long‑horizon planning and economic judgment under uncertainty. Customers place orders around the clock, suppliers reprice or fail, and market shocks can arrive with or without warning. The agent operates through the same 29 merchant tools a human operator would use, constrained by a windowed operation budget that makes simulated time a function of actions taken, so model latency cannot affect simulated time. Pass thresholds are calibrated against scripted anchor policies, the reward is hardened against a catalogue of reward hacks, and each episode replays identically given a sequence of actions. We evaluate seven frontier LLMs on eleven scenarios lasting 30‑45 days and a full simulated year, across three world seeds with matched reasoning effort. No model matches the scripted smart‑triage policy on average: the best, DeepSeek‑V4‑Pro, passes 49% of task‑seed cells versus the heuristic’s 97%. Human experts using the same tools and budgets outscore every model (mean composite 0.708 vs. 0.700). Over a full simulated year under the Claude Code harness, most models show dramatic performance improvement. In a GRPO post‑training run, Qwen3.5‑27B trained on only five disjoint tasks raises its mean composite on held‑out evaluation tasks from 0.136 to 0.373. We release five example training‑split tasks, ten sample trajectories, and the scoring and verification tooling; the full environment and evaluation suite are withheld to keep the benchmark uncontaminated.
Review