GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
markian-rybchuk
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
markian-rybchuk
Model Tree
deepseek-ai/DeepSeek-V4-Flash
Input Modalities
Output Modalities
Supported Functionality
This checkpoint was tested with a local vLLM 0.23.0 build containing DeepSeek
V4 support for this exact mixed compressed-tensors configuration. The loader
validates the routed-expert group as packed, symmetric INT4/G128 with no
activation quantization, keeps the official non-experts on the specialized
DeepSeek V4 FP8 path, and sends the routed experts to Marlin WNA16.
A stock vLLM build that does not recognize this mixed configuration may reject the checkpoint or interpret the expert packing incorrectly. A compatible backend must implement the serialized INT4 contract described below. Do not silently treat the packed nibbles as unsigned affine INT4, do not invent a zero point, and do not group along the output/N dimension.
The routed-expert BF16 source was:
RedHatAI/DeepSeek-V4-Flash-BF168c9b4490a493fba692221ed92162badcbc4dde0cThe authoritative official checkpoint was:
deepseek-ai/DeepSeek-V4-Flash60d8d70770c6776ff598c94bb586a859a38244f1The official checkpoint supplied the non-expert tensors and was also the source of truth for checking the BF16 routed experts.
Every routed expert tensor in the BF16 checkpoint was compared against an independent BF16 dequantization of its official MXFP4 counterpart. The MXFP4 decoder used the official low-nibble-first packing, the E2M1 codebook
text
[0, 0.5, 1, 1.5, 2, 3, 4, 6, -0, -0.5, -1, -1.5, -2, -3, -4, -6]
and the official E8M0FNU scale for each group of 32 weights. Dequantized values were cast to BF16 before comparison.
The initial comparison found 15 localized tensors in the RedHatAI checkpoint that were not bit-identical to the official dequantization. Those 15 tensors were replaced with direct official MXFP4-to-BF16 dequantizations in a small repair overlay. The overlay SHA-256 was:
text
6196731a1406323e263bbb3bef2dc14ce628ec99b2d45124eb1f44dba490f004
The complete repaired source then passed a bitwise BF16 comparison, including signed zero:
The complete audit is stored in
source-provenance-report.json, and the 15
replacements are recorded in
source-repair-manifest.json.
This step matters because the quantizer did not quantize directly from packed MXFP4. The intended route was:
text
official MXFP4 -> exact BF16 dequantization -> calibrated INT4
Going through BF16 ensures GPTQ and iMatrix operate on the actual dequantized expert values rather than on MXFP4 codes or scales interpreted as ordinary floating-point tensors.
The repaired source was loaded with Transformers as BF16 using CPU offload:
python
model = AutoModelForCausalLM.from_pretrained(bf16_model_path,dtype=torch.bfloat16,device_map="auto_offload",max_memory={"cpu": 140 * 2**30},offload_folder=offload_dir,)
The DeepSeek V4 _keep_in_fp32_modules_strict set was cleared, matching the
llm-compressor DeepSeek V4 example, so that Transformers did not unexpectedly
leave target-module norms or staging values in FP32 because of the upstream
quantization metadata. The 15 repaired tensors were applied after model load
and rechecked by SHA-256 before calibration began.
Calibration used:
HuggingFaceH4/ultrachat_200ktrain_sftmoe_calibrate_all_experts=TrueThe first 64 examples from train_sft were shuffled with seed 42. Messages
were rendered with DeepSeek V4's repository-specific BOS, user, assistant, and
EOS tokens, then tokenized without adding another set of special tokens.
There were two calibration traversals in total:
The model was processed sequentially by DeepseekV4DecoderLayer. All experts
were calibrated even when ordinary token routing would not have selected them.
Error was not propagated between decoder layers.
The exact target expression was:
text
^model.layers.<layer>.mlp.experts.<expert>.(gate_proj|up_proj|down_proj)$
No router, shared-expert, attention, embedding, normalization, or output-head weight was targeted. The projection-name mapping back to the official checkpoint is:
text
gate_proj = w1down_proj = w2up_proj = w3
The quantization recipe used llm-compressor
0.1.dev1+ge5eb138 and compressed-tensors
0.17.2a20260724, with these parameters:
text
IMatrixGatherer:targets: routed expert gate/up/down projectionsweight observer: imatrix_msestrict: trueGPTQ:weight type: integerbits: 4symmetric: truestrategy: groupgroup size: 128group axis: logical input/K dimensionscale dtype: BF16zero point: noneblock size: 128dampening fraction: 0.01activation ordering: static/weightserialized g_idx: noneoffload Hessians: falseinput activation quantization: noneoutput activation quantization: none
The full 43-layer quantization took about 9.4 hours on the single B200. Each completed decoder layer was immediately converted to compressed-tensors' packed representation and streamed to disk. This avoided retaining both the complete BF16 expert model and a second complete GPTQ-updated copy.
The resulting expert weight contract is:
text
logical weight type: signed INT4logical code range: -8 through +7quantization: symmetricgroup size: 128 weightsgroup axis: logical input/K dimensionscale dtype: BF16scale meaning: multiplicativedequantization: weight = signed_code * scalezero point: noneexpert inputs: BF16 (W4A16)
Compressed-tensors stores eight 4-bit codes in each INT32 weight_packed
element. The physical nibble is an offset-binary storage code in [0, 15]:
text
signed_code = packed_nibble - 8weight = signed_code * bf16_scale
That packing offset is not an affine quantization zero point. The semantic quantization remains symmetric and has no serialized zero-point tensor.
Each original expert projection is represented by:
weight_packed: INT32 packed nibbles;weight_scale: BF16 multiplicative G128 scales;weight_shape: the original logical matrix shape.The packed-code range was independently checked as [0, 15], corresponding to
the full logical signed range [-8, 7].
After expert quantization, every non-expert tensor was copied from the pinned official checkpoint with the same name, dtype, shape, and bits. Each copied tensor was reopened from the output serialization and checked for bitwise equality. This preserved 1,564 official non-expert tensors totaling 8,845,959,388 bytes.
The mixed compressed-tensors configuration contains two explicit groups:
A fresh tensor index was generated. The calibrated main-model checkpoint has:
model-xxxxx-of-00046.safetensors shards.The MTP module is not included. DeepseekV4ForCausalLM did not expose MTP to
the calibration pipeline, so including uncalibrated MTP experts would have made
the GPTQ+iMatrix provenance claim inaccurate. Consequently,
num_nextn_predict_layers is set to 0.
The initial streamed layout was subsequently repacked into 46 balanced standard
Hugging Face safetensors shards. Repacking did not requantize or otherwise
transform any tensor. The current layout's validation report is in
repack-validation.json. The complete recipe and
assembly records are in quantization-contract.json
and quantization-manifest.json.
The benchmark server ran on one NVIDIA B200 with approximately 191.5 GiB of GPU memory. The tested software environment was:
text
vLLM: 0.23.0, with the mixed DeepSeek V4 W4A16 loader changePyTorch: 2.11.0+cu130CUDA reported by PyTorch: 13.0compressed-tensors: 0.17.0 in the vLLM environment
FlashInfer JIT compilation also required the ninja executable. It was added
to the vLLM virtual environment with:
bash
uv pip install --python /home/mark/.venvs/vllm/bin/python ninjaexport PATH="/home/mark/.venvs/vllm/bin:$PATH"
The local DeepSeek V4 loader change was deliberately narrow. It accepted the
outer quant_method=compressed-tensors only when the routed group was
pack-quantized signed INT4, symmetric, G128, activation-unquantized, and had no
zero point. It then selected:
DeepGemmFp8BlockScaledMMKernel for the unchanged official FP8 linears;MARLIN WNA16 for the routed INT4 expert GEMMs.The server was launched as follows:
bash
export PATH="/home/mark/.venvs/vllm/bin:$PATH"MODEL=/home/mark/models/DeepSeek-V4-Flash-W4A16-G128-GPTQ-iMatrix/home/mark/.venvs/vllm/bin/vllm serve "$MODEL" \--host 0.0.0.0 \--port 8001 \--served-model-name deepseek-ai/DeepSeek-V4-Flash \--max-model-len 32768 \--max-num-seqs 64 \--max-num-batched-tokens 32768 \--gpu-memory-utilization 0.96 \--kv-cache-dtype fp8 \--block-size 256 \--enable-prefix-caching \--enable-chunked-prefill \--enforce-eager \--moe-backend marlin \--reasoning-parser deepseek_v4 \--tool-call-parser deepseek_v4
Again, --kv-cache-dtype fp8 applies only to the attention KV cache. The
routed-expert GEMM inputs remain BF16.
On the successful launch, vLLM reported:
--max-num-seqs 64 is the scheduler's sequence-count ceiling; it does not
override the physical KV-token capacity. The GPQA requests were much shorter
than 32K, so all 30 evaluation requests could run concurrently.
The model was evaluated through the local OpenAI-compatible chat-completions endpoint with lm-eval. The exact command was:
bash
HF_TOKEN="$HF_TOKEN" lm-eval run \--model local-chat-completions \--model_args "model=deepseek-ai/DeepSeek-V4-Flash,base_url=http://localhost:8001/v1/chat/completions,num_concurrent=32,max_retries=3,timeout=1200" \--tasks gpqa_diamond_cot_zeroshot \--apply_chat_template \--gen_kwargs "max_gen_toks=3000,temperature=0.0" \--limit 30 \--log_samples \--output_path runs/gpqa-diamond-w4a16-g128-gptq-imatrix-c32-n30
The task was gpqa_diamond_cot_zeroshot version 2.2. This was a 30-question
limited run, not the full 198-question GPQA Diamond set.
| Metric | Result |
|---|---|
| Flexible-extract exact match | 0.7333 |
| Correct questions | 22 / 30 |
| Standard error | 0.0821 |
| Strict-match exact match | 0.0000 |
All 30 API requests finished with normal stop reasons. There were no request errors, aborts, or length truncations. The concurrent generation phase took about 104 seconds.
The reported score is the task's flexible-extract result. Strict match was zero because the model emitted chain-of-thought explanations and final answers such as boxed option letters rather than returning only the task's exact bare target string. The flexible extractor recovered 22 correct choices from those responses.
Because only 30 questions were used, the 73.33% result has substantial sampling uncertainty and should be treated as a small-run quality check, not a definitive full-benchmark estimate.