Abstract
CVSS base metrics are assigned from short, free-text CVE descriptions that are often ambiguous, yet transformer-based models are increasingly used to automate this process. This thesis evaluates how reliably such models can predict CVSS v3.1 metrics and whether their decisions are grounded in CVSS-relevant evidence rather than dataset artefacts.
Under controlled and reproducible training conditions, encoder-only trans- formers achieve strong predictive performance, but results depend critically on correct model initialisation and dataset consistency. DistilBERT and BERT provide the best balance between accuracy, stability, and training cost, while heavier architectures are impractical for large-scale or repeated use. Across models, Balanced Accuracy reveals systematic weaknesses on minority classes that are hidden by headline accuracy scores.
To examine model behaviour, multiple interpretability methods are applied, including Integrated Gradients, SHAP, LIME, and Layer-wise Relevance Propagation. These analyses show that correct predictions frequently rely on indirect textual cues rather than explicit CVSS justification. A comple- mentary user study reveals similar uncertainty in human reasoning, indicat- ing that disagreement often stems from underspecified descriptions rather than model error alone. Together, the findings argue that predictive per- formance alone is insufficient to assess the reliability of automated CVSS scoring systems and support the use of transformer models as decision- support tools that expose ambiguity rather than replace expert judgement.