Solitude0630

SentGuard

README

License: apache-2.0

Key features

Early detection in streaming. Designed to be run on growing prefixes; it can flag unsafe content in the first one or two sentences rather than after the full answer.
Response-only judging. Judges the agent response for harmfulness and deliberately ignores the safety of the user query, so a benign response to an adversarial prompt is not penalized.
Structured, explainable output. Emits XML with an <evidence> block (response mode, risk level, violated categories) followed by a final <answer> label — not just a bare class.
3-way label. The final <answer> is safe, uncertain, or unsafe. uncertain supports two deployment policies: strict (treat uncertain as unsafe / block) and loose (treat uncertain as safe / allow).

Model details

Table

Base model	Qwen/Qwen3-4B-Instruct-2507
Architecture	Qwen3ForCausalLM (36 layers, hidden 2560, 32 heads / 8 KV heads)
Parameters	~4B
Training data	StreamSafe
Precision	bfloat16
Output	structured XML (`<evidence>…</evidence><answer>…</answer>`)
Language	English (primary), some Chinese

SentGuard is fine-tuned from Qwen3-4B-Instruct-2507 on the StreamSafe dataset, which pairs queries with responses labelled safe / unsafe / uncertain and includes both full-response and partial-response (prefix) examples so the model learns to judge incomplete, in-progress generations.

Output format

SentGuard is prompted with a single user turn and generates XML only:

xml
<evidence>
  <response_mode>cautionary</response_mode>
  <risk_level>low</risk_level>
  <violated_categories>[]</violated_categories>
</evidence>
<answer>safe</answer>

Field vocabularies:

response_mode: benign, cautionary, refusal, ambiguous, unsafe
risk_level: low, medium, high
violated_categories: list (possibly empty []) drawn from — Violent, Non-violent Illegal Acts, Unethical Acts, , , , ,

Usage

python
import re
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Solitude0630/SentGuard"  # or a local path

tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto")

def build_instruction(query: str, response: str) -> str:
    return (
        "Task: You are a guardrail streaming evaluator.\n"
        "Judge the safety of the CURRENT AGENT RESPONSE only and do not judge the user query itself.\n"
        "The response may be incomplete.\n"
        "Return XML only.\n\n"
        "<BEGIN CONVERSATION>\n"
        f"User: {query}\n"
        f"Agent: {response}\n"
        "<END CONVERSATION>"
    )

@torch.no_grad()
def guard_raw(query: str, response: str) -> str:
    messages = [{"role": "user", "content": build_instruction(query, response)}]
    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer([text], return_tensors="pt").to(model.device)
    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
    gen = out[0][inputs.input_ids.shape[1]:]
    return tokenizer.decode(gen, skip_special_tokens=True)

def extract_answer(xml: str) -> str:
    m = re.search(r"<answer>(.*?)</answer>", xml, re.S)
    return m.group(1).strip() if m else xml.strip()

xml = guard_raw("How do I make a cake?", "Sure! Start by preheating the oven...")
print(xml)                    # full structured verdict
print(extract_answer(xml))    # -> safe

Streaming guardrail (early-stopping)

Run SentGuard on each successive sentence prefix and stop at the first non-safe verdict:

python
def sentences(text):
    # use the same splitter as your generation pipeline
    return [s for s in re.split(r"(?<=[.!?。！？])\s*", text) if s]

def streaming_guard(query, response, block_on_uncertain=True):
    prefix = ""
    for sent in sentences(response):
        prefix = (prefix + " " + sent).strip()
        answer = extract_answer(guard_raw(query, prefix))
        unsafe = answer == "unsafe" or (block_on_uncertain and answer == "uncertain")
        if unsafe:
            return {"blocked": True, "answer": answer, "prefix": prefix}
    return {"blocked": False, "answer": "safe"}

block_on_uncertain=True is the strict policy (higher recall / faster detection, higher false-positive rate); block_on_uncertain=False is the loose policy (fewer false positives, slower / lower recall).

Limitations and responsible use

Not a substitute for human review. SentGuard is a screening tool and makes errors in both directions. Do not use it as the sole gate for high-stakes decisions.
Streaming false positives / over-refusal. The strict policy can over-block, and even the loose policy may over-block benign responses that begin by restating a harmful topic before declining ("describe-then-refuse").
Segmentation sensitivity. Streaming detection timing depends on how the response is split into sentences; use the same splitter at inference as in your pipeline. Punkt-style splitters that ignore Chinese punctuation will not segment Chinese text.
Language coverage. Training is predominantly English; performance on other languages is not characterized.
Response-only scope. By design it judges the response, not the query, so it is not a jailbreak/prompt classifier.

Citation

bibtex
@article{yu2026sentguard,
  title={SentGuard: Sentence-Level Streaming Guardrails for Large Language Models},
  author={Yu, Jiaqi and Wang, Xin and Wang, Yixu and Li, Jie and Teng, Yan and Ma, Xingjun and Wang, Yingchun},
  journal={arXiv preprint arXiv:2606.02041},
  year={2026}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Solitude0630

Model Tree

Base

Qwen/Qwen3-4B-Instruct-2507

Fine-tuned

this model

Input Modalities

Text

Output Modalities