Abstract
Machine learning-based vulnerability detection has produced impressive benchmark numbers, yet practical tools remain unreliable. This thesis investigates why, and what can be done about it. We develop VulnBench, a unified benchmarking framework that evaluates seven models across twelve vulnerability detection datasets spanning synthetic, curated, and real-world code. Using this framework, we address three research questions covering evaluation methodology, model architecture, and dataset quality. The finding with the most impact is methodological: decision threshold optimization—selecting a classification threshold on a validation set rather than defaulting to 0.5—improves F1 in 96.7% of experiments, with an average absolute gain of +0.148 for standard BCE-trained models. This single change, requiring no modification to model architecture or training, outperforms the gains typically attributed to architectural improvements in the literature. Class-balanced focal loss with F1-based checkpoint selection provides further benefit on severely imbalanced datasets, with gains of up to +0.235 F1, though the effect is negligible on balanced data. Examining architecture, pre-trained transformer models (e.g., CodeBERT, CodeT5, GraphCodeBERT, NatGen, LineVul) outperform graph neural networks on all twelve datasets, with an average F1 advantage of +0.070. Pre-trained transformers appear to capture sufficient structural information through self-attention that explicit graph construction provides no systematic benefit.