We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It merges resolved prediction‑market questions with dated news articles, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools enable repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve.
Forecast-Dojo contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8 M dated news articles. In an evaluation of 12 models, research tools lower the Brier score for every model, and forecasts improve as events unfold, with the largest gains at steps where newly dated evidence is added. Nevertheless, all models still trail historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates reduces research cost but does not consistently improve forecast quality.
Beyond evaluation, Forecast-Dojo provides interaction trajectories and outcome feedback for agent learning, and we demonstrate supervised fine‑tuning as a proof of concept.
Review