The rapid progress of large language models (LLMs) has intensified the search for scalable methods to evaluate the security of generated and security‑sensitive software. Static analysis is popular because it is scalable, reproducible, and inexpensive, but it only inspects source code and cannot directly observe runtime exploit behavior. Vulnerabilities that depend on adversarial inputs, execution context, or exploit chaining often evade static checks while remaining exploitable in practice, yet a clean static report is frequently taken as evidence of safety.
This work introduces the Static‑Pass Dynamic‑Fail (SPDF) phenomenon and a three‑stage agentic pipeline. First, a composite Bandit‑Semgrep gate performs static scanning. Next, an LLM conducts Common Weakness Enumeration (CWE) reasoning to flag potential vulnerabilities. Finally, autonomous exploit verification runs in isolated Docker containers.
We evaluated 1,355 Python samples from the SecurityEval, RedCode, and CyberNative datasets. After the Bandit‑Semgrep gate, 654 samples produced no findings. The LLM stage identified 394 candidate vulnerabilities across 235 files. Dynamic verification confirmed or partially confirmed exploitability in 95 files, yielding an overall pipeline rate of 14.53% (roughly 1 in 7 statically clean samples). By dataset, confirmed exploitability was 33.7% for RedCode, 28.6% for CyberNative, and 5.4% for SecurityEval. Frequently confirmed classes such as CWE‑338 and CWE‑916 were missed by both Bandit and Semgrep.
These findings demonstrate that static‑analysis success and runtime security constitute hierarchical layers of assurance rather than interchangeable metrics. Evaluating AI‑generated and security‑sensitive code should incorporate both static and dynamic perspectives to avoid false confidence.
Review