Pankei

soc-narrative-sft-qwen3-14b

README

License: apache-2.0

Description

SOC Narrative is a framework for insider threat detection using small open-weight LLMs. A model receives a user/day window of events from the CERT Insider Threat Dataset R4.2 and must produce a structured response with:

Risk label: normal, suspicious, or malicious
Evidence: cited event IDs supporting the decision
Reasoning: brief explanation of the investigation logic

This project explores whether small LLMs (3B–14B) can match or exceed traditional ML baselines for UEBA (User and Entity Behavior Analytics).

Metrics (dev_balanced_50)

Table with columns: Metric, Value
Metric	Value
Accuracy	0.74
Macro F1	0.735
Recall Malicious	0.88
Valid Format Rate	0.68
Actionability Rate	0.68

Quick Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-14B"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "Pankei/soc-narrative-sft-qwen3-14b")

tokenizer = AutoTokenizer.from_pretrained(base)
inputs = tokenizer("<your prompt>", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(output[0]))

Note: This is a LoRA adapter (~30–160 MB). You need the full base model (Qwen/Qwen3-14B) to load it.

Training Details

Base model: Qwen/Qwen3-14B
Method: SFT LoRA
Train data: 512 balanced user/day windows (CERT R4.2)
Checkpoint: step 32
LoRA rank: 32, alpha: 64, target modules: q_proj, k_proj, v_proj, o_proj
Format: Structured SOC Narrative (risk + evidence + reasoning)
Hardware: NVIDIA H100 (80 GB)

Limitations

Evaluated on a small balanced sample (n=50) — results may not generalize to production distributions
Highest malicious recall (0.88) but low format compliance. Many responses mix normal/malicious tokens.
Dataset is based on synthetic insider threat scenarios from CERT R4.2 (2016) — real-world performance may differ

Citation

bibtex
@misc{soc-narrative-2026,
  author = {Research project},
  title = {SOC Narrative: Small LLMs for UEBA / Insider Threat Detection},
  year = {2026},
  howpublished = {\url{https://github.com/Pancake2021/research_work_by_a_student}}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Pankei

Model Tree

Base

Qwen/Qwen3-14B

Adapter

this model

Input Modalities

Text

Output Modalities