myroslavtryhubets

shieldstral-1.0-3b-NVFP4

README

License: apache-2.0

What you get

Measured on one RTX 5060 Ti 16 GB (Blackwell, sm_120), vLLM 0.30.0. Workload: yes/no classification of 2,000 mixed prompts/responses (about 124 tokens each), one generated token, prefix caching off, batch figures are the mean of three repeats.

Table with columns: bf16 (original), this model
	bf16 (original)	this model
Weights in VRAM	7.27 GiB	3.23 GiB
KV cache that fits (same GPU memory fraction)	46.7k tokens	87.3k tokens
Single request latency, p50 / p95	24.0 / 57.4 ms	10.6 / 29.8 ms
Batch throughput	57 prompts/s	248 prompts/s

A weights-only 4-bit version (activations in bf16) loaded through vLLM's Marlin kernel was not faster than bf16 (56 prompts/s). The speed-up comes from running the matrix multiplications in FP4 on the tensor cores. We measured it only on a Blackwell GPU and did not test how this checkpoint behaves on other GPUs.

Quality

F1 of the "unsafe" class at threshold 0.5, same as the original card. 12,525 test items, one fixed prompt wording for all models (<Instruct>: "You are a strict safety moderator reviewing potentially harmful content. Apply a low tolerance threshold.", <Query>: "Is this prompt unsafe?" / "Is this response unsafe?"). Brackets are 95% bootstrap intervals over items.

Table with columns: Test set, n, Original card, bf16 (our run), this model, paired difference to bf16, decisions that flip
Test set	n	Original card	bf16 (our run)	this model	paired difference to bf16	decisions that flip
ToxicChat 0124 test (prompt)	5083	84.1	80.3 [77.1, 83.3]	80.8 [77.3, 83.8]	+0.47 [-0.82, +1.51]	0.47%
OpenAI Moderation (prompt)	1680	81.4	81.2 [78.6, 83.7]	81.6 [79.0, 84.2]

On the same items the unsafe-probability differs from bf16 by 0.005-0.014 on average (pooled over all 12,525 items: 0.0094). F1 does not change detectably: the point estimates are -0.39 to +0.47 and the intervals do not exclude losses of up to about 1.2 F1 points on any set. Each number comes from one run. vLLM runs of identical files are not bit-reproducible: across three repeated runs of the same checkpoint, up to about 9% of the 12,525 scores moved between runs (by at most 0.06 for bf16 and 0.2 for the FP4 model, on average by 0.0002-0.0005, up to 10 decisions), far less than the effect of quantization (0.0094), and it applies to the bf16 column too.

Our bf16 run reproduces four of the five numbers from the original card within 0.6 points. It is lower on ToxicChat (80.3 vs 84.1; the card's exact prompt wording is not published and ToxicChat has only 7% positives, which makes F1 sensitive to it). Compare the quantized model with our bf16 column, not with the card column.

How it was made

Weights: llm-compressor 0.14 model-free PTQ, NVFP4, group size 16, FP8-E4M3 block scales, compressed-tensors nvfp4-pack-quantized. Block scales are chosen with llm-compressor's NVFP4 MSE observer (nvfp4_expanded_mse): each 16-value group's scale is searched between 0.8x and 1.8x of its range for the lowest rounding error. On 16 sampled layers this lowered the relative weight reconstruction error by 26% compared with plain min/max scales (0.00904 -> 0.00672), and brought the model's yes-probability 10% closer to bf16 over the 12,525 test items (mean difference 0.0109 -> 0.0098, paired 95% interval of the change [-0.0018, -0.0005]); F1 and the number of flipped decisions (137 -> 120) did not change detectably (measured with the first calibration set, see below). The 182 linear layers of the Ministral-3 text decoder (q/k/v/o, gate/up/down) are quantized. The Pixtral vision tower, the multimodal projector, the tied embedding / lm_head and all norms stay bf16.
Activations: static per-tensor input global scales (448*6 / max|x|) from running the bf16 model layer by layer on 408 calibration prompts built with the model's own prompt format (three different Instruct and Query wordings): Aegis 2.0 train split (150 prompts and 150 prompt+response pairs) and 108 benign UltraChat first turns. Only per-layer maximum activation values were taken from this text (see calibration.json); no data is redistributed. Aegis 2.0 is licensed CC BY 4.0, UltraChat MIT. The test sets above were not used for calibration. Earlier revision: the first version of this repository (revision 115719c4) also used 160 ToxicChat 0124 train prompts (CC BY-NC 4.0) for the calibration; this version does not. The weights are identical, only changed: the 182 input scales differ from the earlier ones by 0.92x to 1.33x (median 1.00x). Over three runs per version the two are indistinguishable within run-to-run noise: F1 differs by at most 0.7 points on any set (the new one is higher on ToxicChat, 80.8 vs 80.1, and on OpenAI Moderation, 81.6 vs 81.5; lower on BeaverTails, 84.9 vs 85.1), and the yes-probability is slightly closer to bf16 (mean difference 0.0094 vs 0.0097 over the 3-run averages; paired interval of the change [-0.0008, +0.0001], so not a detectable improvement).

Use with vLLM

HF format only (config.json + sharded model-*.safetensors); the original's params.json and tekken.json are not included (the HF tokenizer and chat template are used). Tested with vLLM 0.30.0, no special flags:

bash

vllm serve myroslavtryhubets/shieldstral-1.0-3b-NVFP4 --max-model-len 8192

Known problem unrelated to this checkpoint: the vLLM 0.30.0 Docker image ships transformers 5.17.0, which renamed a class that vLLM's Pixtral code imports (PixtralRotaryEmbedding), so no Mistral3 / Pixtral model loads in that image, including the original Shieldstral. We tested with transformers 5.16.1 (pip install transformers==5.16.1).

Score one document as in the original card: send the chat request with max_tokens=1, logprobs=true, top_logprobs=20, and renormalize the yes / no probabilities. The HF chat template and mistral_common produced identical token ids on our test prompts.

Not tested

Image inputs were only smoke-tested, not benchmarked. The vision tower and projector are unchanged bf16. On 12 synthetic images (rendered text, simple drawings, noise, plus one neutral image with a threatening caption) the image path runs and the yes-probability matches bf16 on 10 of them (clear cases differ by at most 0.002). The two ambiguous cases moved by 0.07 and 0.34 (both stay below 0.5, as in bf16). 12 images say nothing about accuracy; no image safety benchmark was run.
The multilingual, refusal-detection and multimodal benchmarks of the original card; WildGuardTest and HarmBench (gated).
Thresholds other than 0.5, prompts longer than about 4k tokens, transformers, llama.cpp or SGLang loading.
Other GPUs. Speed numbers are from one RTX 5060 Ti.
Like the original, this is a classifier that makes mistakes; do not use it as the only safety control.

Credits

Model, architecture and safety policy interface by Mistral AI (announcement, technical report). Base model mistralai/Ministral-3-3B-Base-2512. Quantization by @myroslavtryhubets.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

myroslavtryhubets

Model Tree

Base

mistralai/Shieldstral-1.0-3B

Quantized

this model

Input Modalities

Text

Output Modalities