Leadership computing facilities manage large-scale scientific datasets that often require substantial transformations before they can serve as AI training data. However, no existing framework fully unifies automated transformation, readiness assessment, provenance tracking, and agent-native deployment.
We present REDI, an open-source framework that bridges this gap through a unified five-stage pipeline: ingest, preprocess, transform, structure, and output, with instrumentation for reproducibility and deployment as an agent-callable skill. The companion tool, SetGo, automates FAIR compliance and catalog publication.
Evaluated across climate, proteomics, materials science, and nuclear fusion, REDI transforms all datasets from raw to AI-ready, with outputs validated against domain-expert references. Preliminary results show near-ideal parallel scaling up to 100 nodes on Frontier for the climate case.
Provenance-instrumented profiling reveals file I/O as the dominant pipeline cost, with format selection being a first-order optimization lever. These results establish REDI as a cross-domain platform providing automated data readiness for scientific AI, turning data preparation bottlenecks into reproducible, reusable community assets.
Blogger's Review: The introduction of the REDI framework marks a significant advancement in automated data preparation for scientific AI, particularly in its effectiveness across multiple domains. By streamlining the data processing workflow, REDI not only enhances efficiency but also provides researchers with higher reproducibility and reliability in the data preparation phase. With the proliferation of such tools, the barriers to scientific research are expected to lower significantly.