Abstract
The trade-off between reproducibility, quantity, and diversity in vulnerability datasets has long been acknowledged, where enhancing one aspect often compromises the others. In practice, reproducibility has frequently been neglected, limiting the automatic extraction capabilities from historical bug datasets and reducing their utility in downstream security research.
In this work, we propose a method to ensure reproducibility for diverse vulnerabilities at scale by identifying key obstacles to large-scale bug reproduction and addressing them with general solutions. Using this method, we introduce full reproducibility to the largest open-source software vulnerability dataset (OSS-Fuzz) and construct the ARVO dataset (Atlas of Reproducible Vulnerabilities).
ARVO is a large-scale dataset comprising over 6,100 real-world vulnerabilities across 311 projects. By focusing on reproducibility, ARVO differs from existing datasets by providing each vulnerability in a form that can be consistently rebuilt, triggered, and analyzed across versions. Reproducibility also enables automatic identification of the corresponding patch for each vulnerability and supports direct interaction with vulnerabilities after code changes, capabilities that existing large-scale datasets do not offer.
In our evaluation, ARVO successfully reproduces 81% of vulnerabilities and achieves 89.4% accuracy on the located patches. We also discuss ARVO's influence on both upstream practices and downstream security research.
Blogger's Review: The launch of the ARVO dataset marks a significant advancement in open-source software security research. By ensuring reproducibility, it provides researchers with more efficient tools. This method not only enhances the reliability of vulnerability analysis but also lays a solid data foundation for subsequent security measures, making it a model for other fields to follow.