Description

SOC Narrative is a framework for insider threat detection using small open-weight LLMs. A model receives a user/day window of events from the CERT Insider Threat Dataset R4.2 and must produce a structured response with:

• Risk label: normal, suspicious, or malicious
• Evidence: cited event IDs supporting the decision
• Reasoning: brief explanation of the investigation logic

This project explores whether small LLMs (3B–14B) can match or exceed traditional ML baselines for UEBA (User and Entity Behavior Analytics).

Metrics (dev_balanced_50)

Quick Usage

python
Copy code
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3.5-9B"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "Pankei/soc-narrative-sft-qwen3.5-9b")

tokenizer = AutoTokenizer.from_pretrained(base)
inputs = tokenizer("<your prompt>", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(output[0]))

Note: This is a LoRA adapter (~30–160 MB). You need the full base model (Qwen/Qwen3.5-9B) to load it.

Training Details

• Base model: Qwen/Qwen3.5-9B
• Method: SFT LoRA
• Train data: 512 balanced user/day windows (CERT R4.2)
• Checkpoint: step 32
• LoRA rank: 32, alpha: 64, target modules: q_proj, k_proj, v_proj, o_proj
• Format: Structured SOC Narrative (risk + evidence + reasoning)
• Hardware: NVIDIA H100 (80 GB)

Limitations

• Evaluated on a small balanced sample (n=50) — results may not generalize to production distributions
• Excellent format compliance (0.96) and actionability (0.96) compared to Qwen3-14B SFT, but lower malicious recall.
• Dataset is based on synthetic insider threat scenarios from CERT R4.2 (2016) — real-world performance may differ

Citation

bibtex
Copy code
@misc{soc-narrative-2026,
  author = {Research project},
  title = {SOC Narrative: Small LLMs for UEBA / Insider Threat Detection},
  year = {2026},
  howpublished = {\url{https://github.com/Pancake2021/research_work_by_a_student}}
}