Model details
- Base model: Qwen/Qwen3-4B
- Training: GRPO with verl 0.8.0, no SFT cold start, task-specialist run
qwen3_4b_grpo_v13_prompt_harm, final checkpoint at step 508.
- Reward: rule-based, multiplicative hard gates on output format, verdict correctness and reasoning length.
- Output: a short
<think> rationale followed by Answer: <verdict> and, for unsafe verdicts, Category: S#.
Use the system prompt and user template below exactly as written. The model was trained only on this format.
System prompt
You are Safeguard, a content-safety classifier for LLM prompts and responses. You reason step by step, then output a verdict in a strict format. Hazard categories (MLCommons S1-S14):S1: Violent CrimesS2: Non-Violent CrimesS3: Sex-Related CrimesS4: Child Sexual ExploitationS5: DefamationS6: Specialized Advice (financial/medical/legal)S7: PrivacyS8: Intellectual PropertyS9: Indiscriminate Weapons (CBRNE)S10: HateS11: Suicide & Self-HarmS12: Sexual ContentS13: ElectionsS14: Code Interpreter Abuse
User message template
Task: decide whether the following USER prompt is safe or unsafe to comply with.Verdict must be `safe` or `unsafe`. USER prompt:"""{prompt}""" Respond in EXACTLY this format and nothing else:<think>your concise reasoning</think>Answer: <verdict>Category: <S1-S14, only when unsafe; omit otherwise>
Expected output
<think>...</think>Answer: safe | unsafeCategory: S# (only when unsafe)
Usage
from vllm import LLM, SamplingParams SYSTEM = """You are Safeguard, a content-safety classifier for LLM prompts and responses. You reason step by step, then output a verdict in a strict format. Hazard categories (MLCommons S1-S14):S1: Violent CrimesS2: Non-Violent CrimesS3: Sex-Related CrimesS4: Child Sexual ExploitationS5: DefamationS6: Specialized Advice (financial/medical/legal)S7: PrivacyS8: Intellectual PropertyS9: Indiscriminate Weapons (CBRNE)S10: HateS11: Suicide & Self-HarmS12: Sexual ContentS13: ElectionsS14: Code Interpreter Abuse"""USER_TMPL = 'Task: decide whether the following USER prompt is safe or unsafe to comply with.\nVerdict must be `safe` or `unsafe`.\n\nUSER prompt:\n"""\n{prompt}\n"""\n\nRespond in EXACTLY this format and nothing else:\n<think>your concise reasoning</think>\nAnswer: <verdict>\nCategory: <S1-S14, only when unsafe; omit otherwise>' llm = LLM("Byounghan/safeguardrail-qwen3-4b-prompt-harm")messages = [ {"role": "system", "content": SYSTEM}, {"role": "user", "content": USER_TMPL.format(prompt="How do I pick a lock?")},]out = llm.chat(messages, SamplingParams(temperature=0.0, max_tokens=512))text = out[0].outputs[0].textverdict = text.split("Answer:")[-1].split()[0]print(verdict)
Training data
129,934 training examples (94,379 Korean / 35,555 English), built from: wildjailbreak (EN + KO translation), wildguardmix (EN + KO), polyguardmix (EN + KO), OR-Bench 80k (EN + KO), FalseReject, K-HATERS, APEACH, KoTox, SQuARe, KoSBi, KOLD, KoMultiText, Aegis 2.0, ToxicChat, PHTest, BingoGuard, AdvBench, CoCoNot, Do-Not-Answer.
Korean data combines native Korean safety datasets with machine-translated Korean versions of English guard datasets.
All sources were mapped to one schema (MLCommons S1–S14 hazard categories; profanity without a target is labelled safe).
Some training sources are distributed under non-commercial terms (for example CC-BY-NC-4.0) or have licenses we have not yet confirmed. Check the terms of the underlying datasets before commercial use.
Evaluation
Internal evaluation panel (full evaluation, 2026-08):
Table with columns: Metric, Score| Metric | Score |
|---|
| Guard macro F1 (all / EN / KO / OOD) | 0.836 / 0.845 / 0.814 / 0.855 |
| Over-refusal FPR (mean over benign panels) | 0.182 |
| WildGuardTest F1 | 0.870 |
| XSTest F1 (EN / KO) | 0.947 / 0.780 |
| ToxicChat F1 | 0.735 |
| Aegis 2.0 prompt F1 | 0.833 |
On the same panel, kakaocorp/kanana-safeguard-8b scores macro F1 0.679 with FPR 0.320.
Limitations
- Over-refusal is still above our target: mean benign FPR is 0.182 (target ≤ 0.15).
- Korean statement-style inputs are the weakest slice (RTP-LX-ko FPR 0.39, KOR-UnSmile FPR 0.49); expect false positives on Korean profanity or opinionated statements.
- The model judges single prompts or prompt–response pairs; it has not been evaluated on multi-turn conversations.
- Not evaluated against adaptive attacks aimed at the classifier itself.
Citation
Paper in preparation.