GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Rarri
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Rarri
Model Tree
deepseek-ai/DeepSeek-V4-Flash-0731
Input Modalities
Output Modalities
Supported Functionality
The initial upload left the MTP layers' expert weights in the source MXFP4
format, which does not load correctly through the NVFP4 fused-MoE path — MTP
speculative-decoding acceptance collapsed to ~15% and spec decode was a net
loss. This update recasts the routed-expert weights of all 3 MTP layers to
NVFP4 with the same closed-form lossless cast: 2,304 tensors,
603,979,776 / 603,979,776 blocks lossless (100.0%), max dequant error 0.0
(see mtp_nvfp4_build_report.json). Changed files: shards 46–48, the index,
config.json, and hf_quant_config.json (recipe version
dsv4-nvfp4-experts-mtp-fallback).
With this checkpoint MTP speculative decoding (DSpark) works, roughly doubling single-stream decode throughput.
vLLM, TP=2 on 2× RTX PRO 6000 Blackwell (96 GB), PCIe (no P2P), 300 W/GPU
power cap, 131k context. 512-token greedy completions, temp 0, unique prompt
nonces; acceptance measured from vLLM /metrics counter deltas.
| Metric | MTP off | MTP spec decode (DSpark, 5 draft tokens) |
|---|---|---|
| Single-stream decode, median | 112.3 tok/s | 199.9 tok/s (1.78×) |
| Single-stream decode, mean | 112.3 tok/s | 220.2 tok/s (1.96×) |
| Single-stream range | 112.2–112.9 | 167–302 (content-dependent) |
| 2-stream aggregate | — | 353 tok/s |
| 4-stream aggregate | — | 411 tok/s |
| TTFT, ~1.8k-token prompt, streaming | — | 253–258 ms |
A second suite on the identical config measured median 242.5 / mean 232.7 tok/s — greedy tie-breaks change the generated text between sessions, so single-stream medians move ±20% run to run. Typical: ~200–240 tok/s.
Acceptance: mean acceptance length 3.85 tokens/step (of 6 max), cumulative draft acceptance 49.4% — in line with DeepSeek's published DSpark figure (50.8%). Throughput tracks acceptance roughly linearly (accept 2.8 → ~168 tok/s; accept 5.6 → ~302 tok/s); the spread is content predictability, not jitter.
The source release already ships routed experts as MXFP4 (E2M1 nibbles +
E8M0 power-of-two scale per 32-element block). Instead of the usual lossy
dequantize→requantize round trip, this conversion uses a closed-form cast:
weight_scale_2 is pinned to 2^(k_max − 8) and each 16-element block's E4M3
scale to 2^(k_j − m), so every NVFP4 nibble equals the source MXFP4 nibble
verbatim (w1/w3 share one scale_2 for the fused GEMM1).
Result: 8,657,043,456 / 8,657,043,456 main-layer blocks and 603,979,776 / 603,979,776 MTP-layer blocks lossless (100.0000%) — the expert weights are byte-faithful to the original release. Quantization only introduces the calibrated activation scales.
input_scale (per-tensor activation scale for W4A4) comes from post-training
calibration with Model-Optimizer:
abisee/cnn_dailymail and
nvidia/Nemotron-Post-Training-Dataset-v2 (stem/chat/math/code splits)Original HF 48-shard layout. Per routed-expert weight (main and MTP layers),
the MXFP4 .scale sibling is dropped and three keys are added:
| key | contents |
|---|---|
…weight | NVFP4-packed uint8, shape (out, in/2) |
…weight_scale | per-16-block E4M3 scale, shape (out, in/16) |
…weight_scale_2 | per-tensor FP32 scale |
…input_scale | per-tensor FP32 activation scale |
config.json keeps the source FP8 quantization_config and adds
moe_quant_algo: "NVFP4" (group size 16); hf_quant_config.json carries the
full quantized-layer manifest (43 MoE layers + mtp.{0,1,2}.ffn.experts).
MTP shared-expert weights remain in the source format.
deepseek-ai/DeepSeek-V4-Flash-0731 @ 7872f01b (304B params, 43
layers, 256 routed experts, 3 MTP layers)LICENSE)