quantization_config declares:
Table with columns: Field, Value| Field | Value |
|---|
quant_method | compressed-tensors |
format | mxfp4-pack-quantized |
quantization_status | compressed |
config_groups.MXFP4A16.weights | num_bits 4, type float, strategy group, group_size 32, symmetric true, dynamic false, scale_dtype torch.uint8 |
config_groups.MXFP4A16.targets | ["Linear"] |
Each quantized Linear is stored as two tensors:
.weight_packed — uint8, shape [out, in // 2]. Two E2M1 codes per byte,
low nibble first.
.weight_scale — uint8, shape [out, in // 32]. E8M0 exponents
(127 + floor(log2(scale))), one per group of 32 weights.
Kept in BF16: lm_head, the whole vision tower, the MTP head (mtp.*), the
embedding table, the GDN A_log / dt_bias vectors and the GDN convolution.
They are listed in quantization_config.ignore, which the loader matches by
exact name and by re: regex.
This is the same layout and the same config fields as
slybase/Swift-1.5-Qwen3.8-27B-heretic-MXFP4-GPTQ,
the checkpoint the Radiance image benchmarks, except that this one is RTN
(calibration-free) and the vision tower, embedding and GDN tensors are kept in
BF16 rather than quantized.
Serve it with vllm-sly-radiance
docker pull ghcr.io/slybase/vllm-sly-radiance:1.8.1-rocm10.1
The image's serve-mxfp4.sh launcher points at a checkpoint snapshot. To use
this one, point the same launcher at it:
vllm serve /models/qwopus-ct-mxfp4 \ --served-model-name qwopus-coder-flash \ --host 0.0.0.0 --port 8080 \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.93 \ --max-model-len 262144 \ --max-num-seqs 8 \ --max-num-batched-tokens 4096 \ --attention-backend R4D \ --speculative-config '{"method":"mtp","num_speculative_tokens":4,"disable_padded_drafter_batch":true}' \ --no-async-scheduling \ --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \ --override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}'
Image environment variables that select the native MXFP4 path:
RADIANCE_MXFP4=1RADIANCE_MXFP4_W4A8=1RADIANCE_MXFP4_W4A8_MIN_M=0VLLM_ROCM_USE_AITER=1
Because the checkpoint declares re:^mtp.* in its ignore list, the MTP head
stays in BF16 and needs no post-processing; the DFlash2 drafter
(tcclaviger/Qwen3.8-27B-DFlash2-FP8 for the MXFP4 launcher,
syvai/Qwen3.8-27B-DFlash2-W4A16 for the general one) is the speculative path.
The MTP tensors live in model-00018-of-00018.safetensors of the main
checkpoint, not in a separate model_mtp.safetensors as in the reference
checkpoint. Point the loader at the whole directory and it will find them.
How it was made
pip install llmcompressor python - <<'PY'from llmcompressor import model_free_ptq model_free_ptq( model_stub="./src", # the BF16 safetensors checkpoint save_directory="./out-ct-mxfp4", scheme="MXFP4A16", ignore=[ "lm_head", "re:.*visual.*", "re:^mtp.*", "re:.*embed_tokens", "re:.*A_log", "re:.*dt_bias", "re:.*conv1d", ], device="cpu", max_workers=1,)PY
model_free_ptq walks the safetensors shards one at a time, so the 56 GB
source is never materialised in memory. The run took 2m50s on an M4 Pro.
The extra ignore entries (embed_tokens, A_log, dt_bias, conv1d) are
needed because the model-free matcher is name-based and offers every 2-D
tensor to the scheme, while a module-class matcher would skip them.
Verification
Dequantizing a sample of 12 quantized tensors with compressed-tensors' own
MXFP4 decompressor and comparing against the BF16 source gives a relative
mean absolute error of 0.109 to 0.126, which is the E2M1 code resolution at
group size 32. The non-quantized tensors are byte-identical to the source.
License and limitations
Apache-2.0, inherited from the upstream checkpoint. The upstream model card's
stated limitations apply: the reported evaluations are small local runs, and
generated code and tool actions can be incorrect. See the
upstream model card
for the full description and the intended use.