Results
Test set: 500 CVEs published in 2025–2026, drawn at random from 3,000 held-out ones; nothing
published after 2024 was trained on, and test descriptions that repeat an earlier one were
removed. The label is NVD's vector or, where NVD has none, the CNA's vector on a CVE that NVD
analysed. Every model answers the same questions with the same prompts; basal 2.0 4.5B goes
through its own server.
Table with columns: exact vector, severity, base score error (MAE), CWE | exact vector | severity | base score error (MAE) | CWE |
|---|
| Most frequent value per metric | 7.8% | 11.6% | 2.83 | 44.1% |
| basal 2.0 4.5B, Q8_0 GGUF via Ollama | 6.2% | 46.0% | 1.62 | – |
| Qwen3.5-4B, untuned, same prompts | 10.8% | 45.4% | 1.65 | 50.5% |
| Shrike 4B | 44.0% | 65.2% | 0.97 | 80.6% |
Accuracy per base metric, %:
Table with columns: AV, AC, PR, UI, S, C, I, A | AV | AC | PR | UI | S | C | I | A |
|---|
| Most frequent value per metric | 67.4 | 90.6 | 46.2 | 75.8 | 84.0 | 56.6 | 45.4 | 54.6 |
| basal 2.0 4.5B, Q8_0 GGUF via Ollama | 84.2 | 87.0 | 61.6 | 81.8 | 74.4 | 44.8 | 62.8 | 57.2 |
| Qwen3.5-4B, untuned, same prompts | 77.8 | 92.0 | 61.8 | 77.2 | 79.8 | 65.0 | 60.2 | 59.8 |
| Shrike 4B | 91.8 | 93.8 | 78.4 | 92.4 | 94.4 | 81.2 | 83.4 | 84.4 |
- Against the untuned base: exact vector +33.2 (95% CI +28.8 to +37.6), mean question accuracy +17.4 (95% CI +15.8 to +18.8) points.
- Against basal 2.0 4.5B (CVSS questions only; through Ollama it reads at most 20 options, the CWE question has 26): exact vector +37.8 (95% CI +33.4 to +42.2).
For scale: where both NVD and the reporting CNA scored a 2024 CVE (16,020 CVEs), they gave the
same full vector 28.2% of the time and the same severity 57.8% of the time. Scoring a CVE from
its description is a judgement call, and no model or analyst will match NVD every time.
How to use
shrike_infer.py in this repo is self-contained (torch, transformers, peft,
bitsandbytes, accelerate; a CUDA GPU):
from shrike_infer import Shrike shrike = Shrike()print(shrike.triage("SQL injection in the login form of Acme CMS 2.1 allows remote " "unauthenticated attackers to execute arbitrary SQL commands via the username parameter."))
Shrike.ask(description, question, options) returns the probability of every option for one
question, if you need the full distributions. The prompt is a chat message with the
description as the state, the question and the lettered options with their definitions; the
assistant turn is prefilled with {"answer": " and the decision is a softmax over the logits
of the option letters, divided by the fitted temperature (in CALIBRATION.json).
- Confidence. On the 2024 calibration split the probabilities are well calibrated (expected
calibration error 2.4%). On the 2025–2026 test, answers with a probability of 0.8 or more are
82% of all answers and 93.3% of them are right; between 0.8 and 0.95 the model claims 0.91
and is right 91.5% of the time. Review the metrics it is unsure of.
- Speed. Nine forward passes per CVE; on a laptop GPU in WSL about 4 questions per second, so about 2 s per CVE.
Training
- QLoRA: base frozen in 4-bit NF4, LoRA r 32, alpha 64, dropout 0.05 on the attention,
Gated DeltaNet and MLP projections.
- 12,000 examples (80% the eight CVSS questions, 20% the CWE question), one pass, batch 16,
learning rate 1e-4 with cosine decay. Rows were drawn with two weights: CVEs published in 2024
four times as likely (2024 is closest to the CVEs the model will see, and Linux kernel CVEs,
10% of the 2025–2026 test, only appear from 2024), and rare CVSS answers slightly more likely
(weight frequency^−0.2).
- The adapter with the lowest NLL on a 392-row development panel was kept (step 500 of 750,
early stop at 700). About 3 h 40 min on a laptop GPU, evaluations included.
- Every example is rendered with a random option order and frame language, so neither a
letter's position nor the frame carries the answer. Each question has three phrasings.
- Three runs on a laptop GPU, no paid compute. The first (9,000 rows, uniform draw) reached 39.0%
exact vector, the second (30,000 rows, rare answers oversampled) 36.2%: it lost PR on Linux
kernel CVEs. This third run fixed that.
Data
- Records: every CVE in the NVD API 2.0 (CVE records with NVD's analysis), up to October 2026.
- Labels: NVD's CVSS v3.x base vector (v3.0 and v3.1 share the base metrics); where NVD
has none, the CNA's or CISA-ADP's vector on a CVE that NVD marked as analysed. NVD's
CWE where it names one of the 25 most common weaknesses, else "other".
- Splits by publication date: training up to 2024; development and calibration panels are
two disjoint 8% slices of 2024; test from 2025. Descriptions are deduplicated across splits
(numbers masked), so a reworded copy of a test CVE cannot sit in training.
- No outputs of commercial models or decision APIs were used.
Limitations
- It reads only the description. A vague or wrong description gives a vague or wrong vector,
and it does not see code, exploits, advisories or the deployment.
- Rare values are under-predicted. High attack complexity is the right answer for about 1 CVE
in 10 in the test set but is predicted less often than that, so watch for missed
"hard to exploit" cases (e.g. a bug "very difficult to perform" that still gets AC:L).
- CVSS v3.1 base metrics only: no temporal, environmental or CVSS 4.0 scores.
- The test labels are NVD's judgement, which is itself not the only defensible one.
- Use it to prioritise reading, not as an official score. A score that will be published or
drive a decision about people should be checked by a person.
Licence and attribution
- The adapter and
shrike_infer.py are released under the Apache License 2.0. The base model
is Qwen3.5-4B (Apache 2.0).
- This product uses data from the NVD API but is not endorsed or certified by the NVD.
- CVE records are published by the CVE Program under the CVE Terms of Use.
CVE® is a registered trademark of The MITRE Corporation.
- CWE™ names are used under the CWE Terms of Use,
Copyright © 2006–2026, The MITRE Corporation. CWE is a trademark of The MITRE Corporation.
- CVSS is owned by FIRST and used by permission; the base score follows the
CVSS v3.1 specification.