Recent AI progress makes recursive self‑improvement (RSI) increasingly plausible. Large language models (LLMs) can already propose, select, and run machine‑learning experiments, yet the design space is huge and an LLM must judge whether an idea is worth the long training time. This work measures that intuition by asking frontier LLMs to predict the impact of their own experiment ideas before any training is run. We assembled 288 LLM‑generated ideas covering eight MLE‑bench tasks and prompted five models—GPT‑6 Astra, Claude Opus 5.5, Claude Fable 5.1, Kimi K3, and GLM‑5.3—to forecast outcomes, then compared predictions with actual results. Four key findings emerged: (1) predictions beat random chance but remain far from perfect; the best model, Claude Opus 5.5, explains only half the variance (R² = 0.49). (2) All models systematically underestimate effect magnitudes. (3) Prediction quality varies widely across tasks. (4) The coding harness used to implement ideas has little influence on results, and no self‑preference is observed. Higher reasoning settings and richer context yield modest gains. Overall, current LLMs possess useful yet limited intuition about which ML ideas will work. We release the dataset to enable further study.
Review