Data curation is a primary design choice in machine learning, yet tailoring training data to a downstream model and task often demands substantial labor, expertise, and compute. Consequently, datasets are usually processed through fixed pipelines and reused across applications. We demonstrate that this assumption is fragile: across 55 dataset‑model tasks spanning biological domains, the same curation recipe can benefit one model while harming another. Effective curation must adapt to the downstream setting, motivating automated search over data recipes. We first build an agentic system that searches for curation policies at deployment, enabling \textit{test‑time curation (TTC)} and improving data utility. Although it yields performance gains, the search is driven largely by recent feedback and remains local. Adding persistent memory lets the agent retain evidence across iterations but reveals a deeper issue: it forms claims that extend beyond the observed conditions, sometimes discarding an entire strategy after a single failure. To address this, we introduce CaliMem, a calibrated‑memory agentic loop that requires the agent to predict the outcome of each proposed policy before execution. Comparing this prediction with the observed result provides a \textit{reality check} on the underlying claim. The prediction error is used to retain supported claims, revise mistaken ones, and specify the conditions under which a conclusion holds. This makes the agent's beliefs testable, reduces over‑generalization, and supports more effective long‑horizon exploration. Across biological and LLM post‑training tasks, CaliMem consistently outperforms fixed curation baselines and published methods, achieving up to $19.4\%$ improvement over full‑data training. Code is open‑sourced.
Review