Scientists increasingly rely on open-source tools to support their research workflows, yet discovering relevant software among over 600 million GitHub repositories remains challenging. Existing code search benchmarks focus on general software engineering tasks and fail to capture the domain-specific vocabulary and needs of scientific computing.
To address this, we present a curated corpus of 5,264 high-quality, domain-classified scientific repositories spanning five NASA Science Mission Directorate divisions -- Earth Science, Astrophysics, Planetary Science, Heliophysics, and Biological & Physical Sciences. This corpus is enriched with cleaned READMEs, extracted topics, and additional context from crawled links.
Building on this corpus, we introduce two novel information retrieval benchmarks:
- A repository search benchmark with 219 expert-curated queries designed by domain scientists.
- A large-scale code snippet retrieval benchmark containing 117,950 code snippets and 119,720 queries across seven programming languages.
Baseline evaluations on repository search reveal significant performance variation across scientific domains. Code snippet retrieval proves equally challenging, with substantial variation driven by differing documentation practices, coding standards, and programming language conventions across scientific communities. All datasets and benchmarks are publicly released on HuggingFace to support research on scientific tool discovery.
Blogger's Review: This article offers a fresh perspective on code search in the scientific domain, significantly enhancing the discoverability of relevant software through multi-domain datasets and benchmarks. This will help scientists more efficiently leverage open-source tools, advancing scientific research.