Abstract
Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services. Locally deployable open-weight models offer an alternative since sensitive data never leave the local environment.
We introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs on one of the most persistent bottlenecks in research on longitudinal population studies: data preparation. The framework comprises:
- A curated ground-truth dataset (cleaning scripts preparing six sweeps of data from a British cohort study)
- Task definitions encompassing tasks such as category harmonization and multi-wave merging
- Automated routines for evaluating the LLM-produced R code and outputted data.
We benchmark LLMs across the (consumer grade) deployment spectrum to assess their efficacy in 20 data preparation tasks (creation of 102 variables). Current state-of-the-art, 31-35B parameter models almost saturated our benchmark ("average task completion" up to 87.9%). The performance of open-weight LLMs running on consumer-grade hardware shows promise of a viable path toward AI-assisted data preparation in governance-restricted research settings. Our framework is publicly available at: GitHub - RRBench.
Blogger's Review: This open-source framework effectively addresses the challenges of data preparation under governance restrictions, showcasing the potential of open-weight LLMs in personal data processing. For researchers in related fields, this tool will significantly enhance the efficiency and security of data handling.