SinclairSchneider

Mistral-Medium-3.5-128B-W4A16-KV8

README

License: modified-mit

Files and size

Table

Weights	70 GiB (from ~130 GiB FP8)
Scheme	`compressed-tensors` / `pack-quantized`, INT4 symmetric, group size 128
Activations	16-bit, unquantised
KV cache	static fp8 E4M3, per-tensor, calibrated
Quantised modules	616 Linears (88 decoder layers × 7)
Left in original precision	`lm_head`, `embed_tokens`, the Pixtral vision tower (339 Linears), multimodal projector

recipe.yaml in this repository records the exact recipe.

Usage

bash
vllm serve <path-or-repo> \
  --tokenizer mistralai/Mistral-Medium-3.5-128B \
  --tokenizer-mode mistral \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --enable-auto-tool-choice \
  --dtype auto \
  --limit-mm-per-prompt '{"image":0}' \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92

--tokenizer and --tokenizer-mode mistral are required. The architecture forces MistralTokenizer, and this checkpoint carries HF-format tokenizer files rather than the native tekken.json, so vLLM must be pointed at the source repo for it. Only the weights were quantised; the tokenizer is unchanged.

At 70 GiB this is ~35 GiB per GPU at TP=2.

KV cache precision

--kv-cache-dtype auto resolves to fp8 because this checkpoint carries KV scales. On this model that is close to free:

Table with columns: German, English
	German	English
fp8 KV cost vs bf16 KV	−0.01%	+0.37%
Top-1 agreement between the two modes	0.977	0.974

Zero cost on German — noise-level, marginally negative — for roughly double the KV budget. Pass --kv-cache-dtype bfloat16 to decline it.

That asymmetry is not a coincidence: the KV scales were computed from German activations, and German is what they fit. It is the cleanest evidence in this family that the calibration language reaches the KV path, because the scales are the only thing differing between those two measurements.

Evaluation

Perplexity and top-1 agreement on 64 held-out samples × 2048 tokens per language from wikimedia/wikipedia (20231101.de / 20231101.en), 131,008 scored positions each. Generations greedy (temperature=0, top_p=1, seed=0). Measured on NVIDIA H200 (SM90) under vLLM. Eval samples use a different shuffle seed from the calibration set (1234 vs 42).

Perplexity

Table with columns: config, German, Δ, English, Δ
config	German	Δ	English	Δ
FP8 reference	4.8333	—	3.1010	—
W4A16, bf16 KV	4.9826	+3.09%	3.3703	+8.68%
W4A16, fp8 KV	4.9819	+3.07%	3.3826	+9.08%

Top-1 agreement with the FP8 reference

Fraction of positions where the argmax token is unchanged. More sensitive than perplexity to damage that shows up as malformed morphology.

Table with columns: config, German, English
config	German	English
W4A16, bf16 KV	0.9439	0.9174
W4A16, fp8 KV	0.9402	0.9145

German damage is 37% of English damage

Table with columns: German, English, German as % of English
	German	English	German as % of English
Mean NLL increase (nats/token)	+0.0304	+0.0833	37%

Reported in absolute nats rather than as a ratio, because the two languages start from different baselines and a sharper distribution shows a larger relative rise from the same absolute damage.

Generation quality

770,000 characters generated across four sampling configurations — greedy, temperature=0.7, temperature=0.8, and temperature=0.8 with min_p=0.05 — on two long German prompts, one of them deliberately loaded with Chinese institution names in Hanzi.

Zero non-Latin token bleed in every configuration, including the CJK-carrying prompt at temperature 0.8. Chinese terminology appearing in the output was correct usage — organisations named and glossed in German analytic prose — not corruption.

No temperature recommendation follows from this, because no artefacts were found to mitigate. Use the sampling settings the base model recommends.

Compared with an AutoRound build

An AutoRound build from the same base and the same German calibration corpus:

Table with columns: German, English, Top-1 de, KV scales
	German	English	Top-1 de	KV scales
GPTQ (this)	+3.09%	+8.68%	0.9439	yes
AutoRound, 200 iters	+3.34%	+7.44%	0.9434	no

GPTQ is marginally ahead on German and level on top-1 agreement; AutoRound is better on English. The deciding factor was the KV scales: AutoRound accepts --static_kv_dtype and then silently ignores it under --format llm_compressor, producing a checkpoint whose kv_cache_scheme is null with no warning in the log.

Note the two builds also differ in calibration construction — AutoRound used 225 packed blocks, GPTQ 512 single documents — so this is not a clean algorithm-versus-algorithm comparison.

Limitations

The reference is FP8. Every figure above measures INT4 on top of Mistral's own 8-bit quantisation. The compounded loss from the original training weights is larger and cannot be measured without weights that were never released.

No standard benchmark suite. No MMLU, HellaSwag or similar. Perplexity, top-1 agreement, and targeted German generation only.

Calibration corpus is general German, not domain text: 512 samples × 8192 tokens from German Wikipedia (wikimedia/wikipedia, 20231101.de), filtered to samples that fill the full window.

Evaluated at 2048-token sequences despite being calibrated at 8192. Not validated at the model's full 256k window.

Quantisation does not reduce hallucination, and may increase it. Factual recall is concentrated in relatively few weights while fluency is distributed, so a quantised model can retain its confident register while losing specifics. The measurements here — perplexity, argmax agreement — are largely blind to that. For grounded generation, verify that names, dates, figures and URLs in the output trace back to the input.

Vision is untested. The Pixtral tower is left unquantised, but no image-input evaluation was performed. Serve with --limit-mm-per-prompt '{"image":0}' unless you intend to test it yourself.

Reproducing this checkpoint

The quantisation script is in this repository as quantize_model.py.

1. Environment

bash
python3 -m venv venv && source venv/bin/activate
export PYTHONNOUSERSITE=1

pip install transformers llmcompressor datasets accelerate

# Mistral's FP8 weights run through a fused kernel that needs this exact range
pip install "kernels==0.15.2"

Verify the pair imports — llmcompressor and compressed-tensors must match, and vLLM pins compressed-tensors exactly, so do not install vLLM into the same venv:

python
from compressed_tensors.quantization import is_cached_attention_module
from llmcompressor.modifiers.quantization import GPTQModifier

2. Smoke test

Runs the entire path — model load, fp8 dequantisation, module walk, GPTQ, serialisation, the output assertions — against a token-sized calibration set.

bash
MODEL=mistral-medium-3.5 QUANT_TESTING=1 QUANT_KV_FP8=1 \
  python quantize_model.py

3. Build

bash
MODEL=mistral-medium-3.5 QUANT_SCHEME=w4a16 QUANT_KV_FP8=1 \
SEQLEN=8192 NSAMP=512 \
  CUDA_VISIBLE_DEVICES=0 python quantize_model.py 2>&1 | tee quant.log

Roughly 14 hours on one H200, plus ~10 minutes of fp8 dequantisation and however long the calibration filter takes to find 512 German Wikipedia articles above 8192 tokens. Needs ~250 GiB of CPU RAM for the dequantised model.

Watch the first block for:

markdown
targets: ['FP8Linear'] (not plain Linear -- the source checkpoint already
                        carries a quantisation format)
target: model.language_model (Ministral3Model), 616 of 956 Linears
starting GPTQModifier on Ministral3Model: 616 Linears

A different module or a different count means the walk picked the wrong submodule and the vision tower would be quantised — stop there.

Configuration

Table with columns: Variable, Default, What it does
Variable	Default	What it does
`MODEL`	`apertus-70b`	`mistral-medium-3.5` here. Also `apertus-8b`, `apertus-70b`; anything else via `MODEL_REPO` + `EXPECTED_LINEARS`.
`QUANT_SCHEME`	`w4a16`	`w4a16` — INT4 group-128 via GPTQ, what produced this checkpoint. — , weights from min/max with no calibration. — GPTQ with per-tensor weight scales and static activations; measured far worse on a sister model.

Notes for this architecture

The source weights are fp8, and llm-compressor cannot observe them. Its min/max observer calls torch.amin, which has no CUDA kernel for Float8_e4m3fn; the run dies at the first sequential epoch end with NotImplementedError: "min_values_cuda" not implemented. The script therefore dequantises every fp8 Linear to bf16 first, handling the block-wise scales (weight_scale_inv over 128×128 tiles). Nothing is recovered by this — the fp8 rounding is already in the values — it only makes them observable.
targets="Linear" matches nothing in the language model, because its modules are FP8Linear before dequantisation. llm-compressor resolves targets by class name. The script derives the target classes from the model rather than assuming.
The module walk must not count only nn.Linear. Before dequantisation the language model holds 616 FP8Linear and the vision tower 340 plain Linear, so a naive count selects the vision tower — and would quantise it while leaving the language model untouched.
dispatches to vision and text branches; pass () to , then save the so the wrapper config and are written too.

python
recipe = GPTQModifier(
    targets=["Linear"],          # after dequantisation
    scheme="W4A16",
    ignore=["lm_head", "re:.*vision_tower.*", "re:.*multi_modal_projector.*"],
    kv_cache_scheme={"num_bits": 8, "type": "float", "strategy": "tensor",
                     "symmetric": True, "dynamic": False},
)

License and provenance

Inherits the licence of mistralai/Mistral-Medium-3.5-128B — a modified MIT licence under which enterprises above an unpublished revenue threshold are directed to Mistral's paid API rather than self-hosting. Consult the upstream repository before deploying commercially.

Quantisation only — no fine-tuning, no additional training data, no modification of model behaviour beyond numerical precision.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.