GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
bbin2022
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
bbin2022
Model Tree
deepseek-ai/DeepSeek-V4-Flash
Input Modalities
Output Modalities
Supported Functionality
| Original tensors | Format | Converted to |
|---|---|---|
| attention, shared experts, output projections | FP8 E4M3 with E8M0 block-128 scales | BF16 |
| 256 routed experts per layer | int8-packed FP4 (E2M1) with E8M0 block-32 scales | BF16 |
| norms, router weights, embeddings, head | BF16 | unchanged |
| hyper-connection scalars, attention sinks | FP32 | unchanged |
tid2eid hash routing table | int64 | unchanged |
The dequantization is exact: FP4 values times power-of-two scales and FP8 E4M3 values times
power-of-two scales are all representable in BF16, so this checkpoint reproduces the original
weights bit for bit. Sampled tensors were verified against DeepSeek's own inference/convert.py
helpers with a maximum absolute difference of 0.0. Tensor names are unchanged; the .scale
tensors and the quantization_config / expert_dtype entries of config.json are removed.
The MTP head is included.
Size: 542 GiB in 46 safetensors shards, 35,020 tensors.
DeepseekV4ForCausalLM with --tokenizer-mode deepseek_v4,
--kv-cache-dtype fp8 --block-size 256, parsers deepseek_v4.vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
(return 0 rotated rows when not on gfx942/gfx950) restores parity with the pure-torch reference
(generation logprobs within 0.03 nats). A second patch lets compress-ratio-4 layers use the Triton
indexer path on gfx90a for prompts longer than one prefill chunk.transformers 5.17 DeepseekV4ForCausalLM on CPU with this checkpoint produces correct greedy
output (bf16 autocast is required because the hyper-connection streams are FP32). vLLM CUDA on an
H100 and vLLM ROCm on MI250X (with the fix) match that reference on prompt and generation logprobs.
Same as the original model (MIT, see LICENSE). Conversion scripts and the vLLM fixes are
described in the accompanying write-up.