We evaluated eight code language models ranging from 6.7B to 15B parameters on three C‑code datasets (DELTASECOMMITS, SVEN, PRECISE‑BUGS). Two diagnostic tasks were defined on the same representations: (1) identifying the pattern instantiated by the code, and (2) determining whether the code is vulnerable. The models encode the first task with high fidelity, accurately capturing the underlying code patterns.
Under the paired C‑code datasets and a linear‑probing setup, vulnerability information emerges more readily as a relational signal—whether code A is riskier than code B—rather than as a globally separable categorical label. Consequently, the models can reliably rank which snippet is riskier, but standard linear classifiers applied to isolated samples achieve limited accuracy because vulnerable and secure samples occupy overlapping regions in the representation space.
This structural distinction suggests that pairwise comparison or ranking approaches may be more effective than direct binary classification for vulnerability detection.
Review