We introduce JavaVulBench, a benchmark dataset and evaluation harness for Java vulnerability detection. This dataset contains approximately 30,600 Java methods covering 1,740 CVEs and over 700 projects, labeled at both method and line granularity, with publication dates for each CVE.
We have designed five realistic splitting strategies: random, project-disjoint, temporal, deduplicated, and unseen CWE-family.
The harness provides a unified LlmPrediction schema across three backend types (encoder classifiers, local generative models served by Ollama, and API-served LLMs routed through OpenRouter), allowing twelve reference detectors (CodeBERT, GraphCodeBERT, UniXcoder, DeepSeek-Coder-1.3B, and eight API/open-weight LLMs: GPT-4o, GPT-4.1-mini, Claude Sonnet 4, DeepSeek-v3, DeepSeek-Coder-v2, Qwen-2.5-Coder-14B/7B, CodeLlama-13B) to be evaluated under identical conditions.
Each model comes with a pre-training contamination audit, enabling users to distinguish genuinely unseen test CVEs from potentially memorized ones. Data, code, and fine-tuned checkpoints are archived on Zenodo, and a short demonstration video is available on YouTube Watch here.
Blogger's Review: JavaVulBench establishes a standardized platform for Java vulnerability detection, significantly enhancing the ability of researchers and developers to compare and evaluate different detection models, particularly in terms of dataset diversity and evaluation rigor.