Model description
This is a W4A16 quantized version of gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-unresolvable-pattern-finetune-v1, which is a fine-tuned Mistral-Small-3.2-24B-Instruct-2506 designed to improve the evaluation of SBVR (Semantics of Business Vocabulary and Business Rules) rules.
The quantization was performed using AutoRound 0.15.0 with SignRoundV2 optimization, reducing the model size from 45 GB (BF16) to 15 GB while maintaining high fidelity to the original model. The model supports both text-only and image+text (vision) inference.
The multimodal projector tensors are stored under their Mistral-native names (multi_modal_projector.w_in, multi_modal_projector.w_out, pre_mm_projector_norm, patch_merger.merging_layer) so that vLLM's Mistral-native loading path matches them correctly; these tensors are BF16 and are not quantized.
Quantization configuration
Table with columns: Parameter, Value| Parameter | Value |
|---|
| Quantization method | AutoRound (SignRoundV2) |
| AutoRound version | 0.15.0 |
| Weight bits | 4 (W4A16) |
| Activation bits | 16 (BF16) |
| Group size | 128 |
| Symmetric | True |
| Iterations | 1000 |
| Calibration samples | 512 |
| Sequence length | 4096 |
| Packing format |
Calibration dataset
The calibration dataset was constructed from multiple domain-specific sources to ensure the quantized model retains accuracy across all relevant domains:
Table with columns: Domain, Samples, Description| Domain | Samples | Description |
|---|
| SBVR rules | ~50% | SBVR compliance rules in JSON format, rule status assignment (PASSED/FAILED/UNRESOLVABLE), insurance terminology |
| Slovak text | ~25% | Slovak Wikipedia, FineWeb2, EN↔SK parallel corpus (opus100) — ensures diacritics preservation |
| English general | ~15% | UltraChat conversations, general English text |
| Reasoning | ~10% | MMLU-Pro (14 categories), BoolQ, TruthfulQA |
All 512 calibration samples were concatenated to a minimum of 4,096 tokens each (20,480 characters) to maximize the calibration signal per sample. The dataset is published alongside this model as calibration_dataset.jsonl.
Quantization script
The quantization script is published as quantize.py in this repository.
Evaluation results
WikiText-2 Perplexity
Standard WikiText-2-raw-v1 benchmark with sliding window (stride=512, max_length=2048, 590 windows, 301,616 tokens):
Table with columns: Model, WikiText-2 PPL, Degradation| Model | WikiText-2 PPL | Degradation |
|---|
| BF16 (original) | 4.7596 | — |
| W4A16 (this model) | 4.9160 | +3.28% |
Comprehensive evaluation
30 domain-specific prompts (SBVR, Slovak, English, code, reasoning) comparing token-level agreement, KL divergence, and generation fidelity against the BF16 baseline:
Table with columns: Metric, Value| Metric | Value |
|---|
| Average PPL degradation | +1.39% |
| KL divergence | 0.0231 |
| Cosine similarity | 0.9945 |
| Spearman correlation | 0.9984 |
| Top-1 token agreement | 70.00% |
| Top-5 token agreement | 90.69% |
| Top-10 token agreement | 91.05% |
| Greedy match | 90.46% |
Per-category PPL
Table with columns: Category, BF16 PPL, W4A16 PPL, Degradation| Category | BF16 PPL | W4A16 PPL | Degradation |
|---|
| SBVR | 30.43 | 30.89 | +1.51% |
| Slovak | 11.09 | 11.16 | +0.63% |
| English | 5.89 | 6.16 | +4.58% |
| Code | 3.41 | 3.38 | -0.88% |
| Reasoning |
Usage
vLLM (recommended for both text and vision)
This model requires vLLM's Mistral-native loading mode. Deploy with config_format: mistral and tokenizer_mode: mistral (the Tekken tokenizer is shipped as tekken.json; using the default HF tokenizer mode produces garbled output):
docker run -d --name vllm-mistral-sbvr \
--runtime=nvidia --gpus '"device=0"' \
-p 8000:8000 \
-v /path/to/model:/workspace/model \
--ipc=host --shm-size=16g \
vllm/vllm-openai:cu129-nightly \
--model /workspace/model \
--served-model-name mistral-24b-sbvr-w4a16 \
--config-format mistral --tokenizer-mode mistral \
--quantization auto_round \
--dtype bfloat16 \
--trust-remote-code
Both text-only and image+text (vision) requests are supported:
import base64, requests
img_b64 = base64.b64encode(open("image.jpg", "rb").read()).decode()
requests.post("http://localhost:8000/v1/chat/completions", json={
"model": "mistral-24b-sbvr-w4a16",
"messages": [{"role": "user", "content": "Evaluate this SBVR rule: ..."}],
"max_tokens": 512,
})
requests.post("http://localhost:8000/v1/chat/completions", json={
"model": "mistral-24b-sbvr-w4a16",
"messages": [{"role": "user", "content": [
{"type": "text", "text": "Describe this image."},
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
]}],
"max_tokens": 512,
})
The projector tensors use Mistral-native names for vLLM compatibility. transformers 5.x loads Mistral3 checkpoints without a name-conversion layer, so the projector tensors need manual renaming (w_in→linear_1, w_out→linear_2, pre_mm_projector_norm→norm, patch_merger→multi_modal_projector.patch_merger.merging_layer) when loading this model with transformers. For production use, the vLLM path above is the recommended and tested route.
Model details
Table with columns: Property, Value| Property | Value |
|---|
| Architecture | Mistral3ForConditionalGeneration |
| Parameters | 1.78B (quantized) / 24B (original) |
| Model size | 15 GB |
| Precision | W4A16 (4-bit weights, 16-bit activations) |
| Context length | 128K (inherited from base model) |
| Vision | Pixtral tower, 1024×1024+ images, 4 images per prompt |
| License | Apache 2.0 |
HELM comparison: fine-tuned BF16 vs W4A16
The BF16 fine-tuned source model and this W4A16 quantized version were evaluated side by side on Stanford HELM benchmarks (crfm-helm >= 0.5.0, 1,000 instances per scenario, served via an OpenAI-compatible endpoint).
General benchmarks
Table with columns: Suite, FT BF16, FT W4A16, Δ| Suite | FT BF16 | FT W4A16 | Δ |
|---|
| bbq | 90.60% | 91.90% | +1.30 pp |
| boolq | 85.10% | 89.40% | +4.30 pp |
| commonsense | 82.80% | 82.90% | +0.10 pp |
| damaged_items_mc | 100.00% | 100.00% | 0.00 pp |
MMLU Pro
Table with columns: Subject, FT BF16, FT W4A16, Δ| Subject | FT BF16 | FT W4A16 | Δ |
|---|
| biology | 76.43% | 76.01% | −0.42 pp |
| business | 34.98% | 37.77% | +2.79 pp |
| chemistry | 31.50% | 32.20% | +0.70 pp |
| computer_science | 53.90% | 52.68% | −1.22 pp |
Summary averages
Table with columns: Metric, FT BF16, FT W4A16, Δ| Metric | FT BF16 | FT W4A16 | Δ |
|---|
| HELM overall avg (23 suites) | 61.77% | 61.19% | −0.58 pp |
| MMLU Pro avg (14 subjects) | 50.34% | 50.02% | −0.32 pp |
| Non-MMLU avg (9 scenarios) | 79.54% | 78.56% | −0.98 pp |
| SBVR unresolvable overall (1,432 items) | 90.92% | 91.97% | +1.05 pp |
BBQ bias
Table with columns: Metric, FT BF16, FT W4A16| Metric | FT BF16 | FT W4A16 |
|---|
| Ambiguous accuracy | 88.67% | 88.67% |
| Non-ambiguous accuracy | 95.29% | 95.29% |
| Bias gap ( | amb − non-amb | ) |
Original model
This model is a quantized version of gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-unresolvable-pattern-finetune-v1. Please refer to the original model card for detailed information about the fine-tuning methodology, training configuration, and HELM evaluation results.