LLM agents have recently shown promise in automating machine learning engineering by editing model and training code with execution feedback, yet data largely remains outside this agentic optimization loop. We cast pre‑training data selection as heuristic engineering over per‑document features such as lexical statistics, categorical labels, and perplexity. AutoData is an agent that searches directly over executable selection algorithms. Unlike prior data‑mixture methods that only optimise weights over a fixed set of domains, AutoData explores a richer program space comprising scoring, stratification, and stochastic selection rules, automatically discovering feature interactions by iteratively refining algorithms with validation feedback from a proxy model. Within an overnight search, AutoData discovers a selection algorithm that outperforms existing human‑designed curation pipelines on the downstream CORE metric. Although the search is performed on a small proxy, the discovered recipe transfers to larger scales and improves performance. These results suggest that data engineering can be treated as an agentic machine‑learning problem, extending autonomous research from model and training‑code optimization to the data layer.
Review: AutoData demonstrates the feasibility of automatically uncovering data‑selection strategies via program search, offering a fresh automation avenue for data engineering that merits further exploration on larger models and multi‑task settings.