Category: <endpoint|cloud_saas|identity_access|network|application_data|cloud_native>
Disposition: <malicious|suspicious|benign|expected_admin|authorised_testing|misconfiguration|unknown>
Severity: <high|medium|low|info>
Action: <escalate_incident_response|investigate_with_owner|collect_missing_evidence|fix_configuration|verify_change_record|verify_assessment_scope|document_and_close>
Table with columns: Field, Source| Field | Source |
|---|
| Category | dataset domain (research metadata) |
| Disposition | dataset target |
| Severity | deterministic mapping of disposition (not dataset ground truth) |
| Action | short code for the dataset's next_review_step, 1:1 with disposition |
Table with columns: Disposition, Severity, Action| Disposition | Severity | Action |
|---|
malicious | high | escalate_incident_response |
suspicious | medium | investigate_with_owner |
unknown | medium | collect_missing_evidence |
misconfiguration | |
Prompt
System prompt:
You are a security operations triage assistant. Analyze the provided security alert and its evidence and produce a concise triage decision. Base the decision only on the evidence in the alert. Do not invent facts that are not present in the alert and do not add explanations.
User message template (Qwen2.5 chat template):
Alert:
{alert}
Respond exactly in this format:
Category: <endpoint|cloud_saas|identity_access|network|application_data|cloud_native>
Disposition: <malicious|suspicious|benign|expected_admin|authorised_testing|misconfiguration|unknown>
Severity: <high|medium|low|info>
Action: <escalate_incident_response|investigate_with_owner|collect_missing_evidence|fix_configuration|verify_change_record|verify_assessment_scope|document_and_close>
Training data
Table | |
|---|
| Dataset | tegridydev/opensec-triage, config research |
| Revision | b9a5d6fb80b329b19efc5c25087b76a804c0a324 |
| Train / validation / test | 40000 / 5000 / 5000 |
| Split policy | official splits; family profiles and counterfactual groups disjoint (verified); exact-duplicate removal across splits |
| OOD probe | model_trainer/test (420 rows; held-out behaviour families and wording; category not scored) |
Training procedure
Table with columns: Hyperparameter, Value| Hyperparameter | Value |
|---|
| Method | LoRA (assistant-only loss) |
| LoRA r / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj |
| Trainable params | 8,798,208 of 502,830,976 |
| Epochs / steps | 2.0 / 2500 |
| Learning rate / scheduler | 0.0002 / cosine |
| Warmup ratio / weight decay | 0.05 / 0.01 |
| Batch (per device x accum) | 8 x 4 = 32 |
| Max sequence length |
Evaluation
Greedy decoding (do_sample=False, max_new_tokens=48). Outputs are parsed per field (case/whitespace/punctuation normalised); a response is valid only if all four fields appear exactly once. Invalid fields count as errors.
Table with columns: metric, base_zero_shot / ood_test, base_zero_shot / test, finetuned / ood_test, finetuned / test, finetuned / validation| metric | base_zero_shot / ood_test | base_zero_shot / test | finetuned / ood_test | finetuned / test | finetuned / validation |
|---|
| valid_format_rate | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| joint_triage_accuracy | 0.0619 | 0.0252 | 0.9167 | 1.0000 | 1.0000 |
| exact_match_accuracy |
Reference (non-LLM) baselines on the same test split: majority class disposition accuracy 0.1424; TF-IDF + logistic regression disposition accuracy 1.0 (OOD model_trainer/test: 0.8143), category accuracy 1.0.
joint_triage_accuracy: all scored fields correct.high_severity_undertriage_rate: share of gold high examples predicted below high or invalid.internal_consistency_rate: predicted severity and action agree with the predicted disposition's mapping.- Macro F1 is reported per field; per-class scores are in
evaluation_results.json.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "thealper2/Qwen2.5-0.5B-Instruct-opensec-triage-lora").merge_and_unload()
tokenizer = AutoTokenizer.from_pretrained("thealper2/Qwen2.5-0.5B-Instruct-opensec-triage-lora")
import json
from huggingface_hub import hf_hub_download
schema = json.load(open(hf_hub_download("thealper2/Qwen2.5-0.5B-Instruct-opensec-triage-lora", "triage_schema.json")))
alert = "Identity audit on ASSET-... recorded that ... <evidence sentences>"
messages = [
{"role": "system", "content": schema["system_prompt"]},
{"role": "user", "content": f"Alert:\n{alert}\n\n{schema['response_format']}"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, do_sample=False, max_new_tokens=48)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
triage_schema.json (in this repository) contains the exact system prompt, response format, label vocabularies and mappings used in training. The prompt must match it exactly.
Limitations
- Training and test data are synthetic and strongly templated; a TF-IDF baseline reaches near-perfect accuracy on the in-distribution test split, so in-distribution scores mainly measure template fit. The OOD column is the more informative generalisation estimate.
- Severity is a fixed projection of disposition; the dataset itself does not assign severity.
- Decisions depend on the evidence sentences in the input (approvals, baselines, correlation). Bare log lines without such context are outside the training distribution.
- English only. Not evaluated on real SOC telemetry. Intended as a triage assistant, not an autonomous response system.
