Usage
import sysfrom pathlib import Pathfrom huggingface_hub import snapshot_downloadfrom transformers import AutoModelForCausalLM, AutoTokenizer local = Path(snapshot_download("VelaOrg/vela-agent-guard-qwen3-0.6b"))sys.path.insert(0, str(local))from lm_guard.format import build_messages, parse_output tok = AutoTokenizer.from_pretrained(local)model = AutoModelForCausalLM.from_pretrained(local, dtype="auto", device_map="auto") row = {"task": "toolcall", "state": { "request": "send 10 usdc to 0x1111111111111111111111111111111111111111", "tool_call": '{"name": "transfer", "arguments": {"token": "USDC", "to": "0x2222222222222222222222222222222222222222", "amount": "10"}}'}}prompt = tok.apply_chat_template(list(build_messages(row)), tokenize=False, add_generation_prompt=True, enable_thinking=False)ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)out = tok.decode(model.generate(**ids, max_new_tokens=256, do_sample=False)[0, ids["input_ids"].shape[1]:], skip_special_tokens=True)print(out, parse_output(row, out))
For secrets, pass {"task": "secrets", "state": {"request": text}}; parse_output maps the returned strings back to character offsets. For injection, pass {"request": ..., "tools": ...}. Malformed output falls back to block / unsure.
Results (eval v2, 300 rows per task, templates unseen in training)
Same eval set and metrics as the Vela release. "Hard" rows are the deliberately tricky ones (e.g. tx hashes that look like private keys, pages that quote attacks without making them).
Table with columns: System, Secrets char-F1 (hard), Injection acc / F1, Tool call acc (hard)| System | Secrets char-F1 (hard) | Injection acc / F1 | Tool call acc (hard) |
|---|
| This model (Qwen3-0.6B, model only) | 0.830 (0.632) | 0.883 / 0.897 | 0.960 (0.919) |
| Vela Agent Guard 4B (heads + hybrid) | 0.946 (0.884) | 0.970 / – | 1.000 (1.000) |
Notes, plainly:
- No tool call that should be blocked was ever allowed (
block_passed_as_allow = 0). The errors are block → confirm.
- Injection is over-eager: 29 of 96 hard benign pages (ones that merely quote an attack) were flagged.
- Secret detection is the weakest part. Pair it with regex for known key formats (as the Vela release does) rather than relying on the model alone.
- These numbers are on synthetic data. Real traffic will be different.
How it was trained
- Data: 1,000 synthetic rows per task (3,000 total) sampled from the Vela Agent Guard training set (
vela_guard/gen_*.py in the sibling repo). Fake keys and addresses are random but format-correct; eval templates are disjoint from training.
- Method: full fine-tune, 2 epochs, AdamW lr 3e-5, batch 16, loss on the answer tokens only, thinking disabled (
lm_guard/train.py). Weights saved in bf16.
- Compute: one A40 on RunPod, about 40 minutes including evaluation.
- Eval: greedy decoding,
lm_guard/evaluate.py; full metrics in eval/metrics.json.
Limitations
A guard model reduces risk; it does not remove it. It can miss secrets in formats it has not seen, be fooled by new injection styles, and its tool-call verdict only knows what is in the request and the call. Keep deterministic checks (allow-lists, amount and recipient checks, human confirmation for fund movements) in front of anything that moves money. Use it as one layer of defence, not the only one.
License
Apache-2.0, same as Qwen3-0.6B.
surplus-launcher: 0x32f8544de85B4e9d874c7AEAda9493359D43502b
surplus-launcher: 0xc2F0cf5F080339930974833B97d2ecd4919720f9