SlowLab is a slow‑moving benchmark for autonomous AI agents, simulating a one‑year experimental campaign. The environment consists of four greenhouse compartments whose crop types, facility configurations and market prices are hidden. An agent must decide when to plant, which climate‑control policy to apply, whether to terminate a crop early, and which sensor readings to collect while crops grow. At the end of the season the agent submits a policy recommendation that is scored by a private evaluator using weather data the agent never observed.
In a preregistered study, 48 unseen sites were used for evaluation. Three low‑cost language‑model agents performed near a standard grower baseline but fell clearly short of a Bayesian‑optimization baseline that started from the same point. The agents operated the facility in two synchronous batches, so in‑season measurements never altered planting decisions, and those measurements did not improve any method’s final score. Allowing other methods to re‑read the collected evidence also failed to produce significantly better recommendations, indicating that the agents’ initial priors and experiment‑selection strategies are the main bottlenecks.
The code is released under the MIT license.
Review: This benchmark exposes the limitations of current language models in long‑term, irreversible experimental settings and highlights the need for better initial strategies and experiment design.