endnai

DeepSeek-V4-Flash-W4A8-FP8

What this is

Table

Base architecture	DeepSeek-V4-Flash (284B total / ~13B active MoE, 43 layers, 256 routed experts top-6 + 1 shared, MLA, hybrid sparse attention + Lightning indexer)
Derived from	`canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP` (identical INT4 expert weights)
MoE experts	INT4 group-quantized weights + FP8 e4m3 dynamic per-token activations (W4A8)
Attention / dense	FP8 block-quantized weights (unchanged from base)
`format`	`mixed-precision` (compressed-tensors)
Footprint	~159 GB materialized, fits TP2 on 2×H200 (identical to the W4A16 base). Weights not stored here — see Getting the weights.
Target hardware	NVIDIA Hopper (H100/H200, SM90)

How it was made

DeepSeek-V4-Flash's MoE experts are stored as INT4. A W4A16 checkpoint runs those INT4 weights through a Marlin dequant→BF16 GEMM; a W4A8 checkpoint instead pairs the same INT4 weights with FP8 activations, so vLLM dispatches them to the native CutlassExpertsW4A8Fp8 kernel on SM90 (_is_fp8_w4a8_sm90).

Because the weights are unchanged, the conversion is a pure config.json edit — no re-quantization, no calibration:

jsonc
// experts config group, input_activations: null  ->
"input_activations": {
  "num_bits": 8, "type": "float", "strategy": "token",
  "dynamic": true, "symmetric": true
}

The _w4a8_conversion key in config.json records this provenance.

Getting the weights

The INT4 weight shards are identical to the base. Materialize a full checkpoint by downloading the base weights and overwriting config.json with this repo's W4A8 config:

bash
# 1. base weights (INT4 shards, tokenizer) — the actual ~159 GB
hf download canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP --local-dir dsv4-w4a8

# 2. this repo's W4A8 config + card (the only real diff)
hf download endnai/DeepSeek-V4-Flash-W4A8-FP8 config.json README.md --local-dir dsv4-w4a8

# dsv4-w4a8/ is now a complete W4A8 checkpoint (INT4 weights + FP8-activation config)

The .safetensors bytes are unchanged; only config.json's expert input_activations differ (see below).

Serving (vLLM)

Requires a recent vLLM nightly and, at the time of writing, four small patches to load the DeepSeek-V4-Flash compressed-tensors checkpoint (these are model-loading fixes, not W4A8-specific — the same patches are needed for the W4A16 base on nightly):

packed_modules_mapping for the model and MTP module (fused_wqa_wkv, fused_wkv_wgate, gate_up_proj).
hash_moe added to the transformers ALLOWED_LAYER_TYPES global allowlist.
o_proj weight-scale name alias (weight_scale_inv → weight_scale).

Launch (2×H200, TP2) from the materialized directory (see Getting the weights):

bash
vllm serve ./dsv4-w4a8 \
  --tensor-parallel-size 2 \
  --disable-custom-all-reduce \
  --trust-remote-code

--disable-custom-all-reduce avoids a TP2 init hang under confidential-compute (custom all-reduce needs CUDA-IPC/symmetric memory, which is unavailable inside TDX CVMs).

Correctness: verified matching the W4A16 base on a temp=0 quality probe (GSM8K 3/3 identical).

Investigation & findings

This checkpoint was built to test a hypothesis: the DeepSeek-V4-Flash prefill bottleneck is the INT4→BF16 Marlin MoE GEMM, so a W4A8 path (native FP8 activation GEMM) should be ~1.5–2× faster. The hypothesis was refuted. Full sweep on 2–8×H200 (TP2 unless noted), single-request prefill ladder (c=1), long-context (ISL up to 24k):

Headline: W4A8 leads the TP2 matrix, but ties W4A16

Table with columns: Config, Engine, TP, Prefill TTFT @24k, Prefill tok/s/GPU @24k
Config	Engine	TP	Prefill TTFT @24k	Prefill tok/s/GPU @24k
W4A8 (this model)	vLLM	2	1658 ms ⭐	7410 ⭐
W4A16 (base)	vLLM	2	1691 ms	7267
FP4 (marlin)	vLLM	2	1824 ms	7090

W4A8 is the fastest TP2 config and the highest per-GPU throughput of every cell measured. Two things to read carefully:

vs W4A16 (its parent): a tie — 1658 vs 1691 ms is ~2%, within run-to-run noise. The specific hypothesis this checkpoint was built to test — "FP8-activation MoE GEMM should be ~1.5–2× faster than W4A16" — was refuted. At prefill batch-M the MoE is weight-bandwidth-bound, so activation precision doesn't move it and Marlin-W4A16 already matches Cutlass-W4A8.
vs FP4-marlin: a real ~9–13% win — int4→Marlin beats nvfp4→Marlin, so W4A8 (and W4A16) beat the FP4 base. FP4-marlin is what production currently runs, so W4A8/W4A16 are meaningfully faster than the deployed config on the same 2-GPU footprint.
The FP8-TP4 cell's low absolute TTFT (892 ms) is tensor-parallel scaling (2× the GPUs); per-GPU, W4A8-TP2 still wins (7410 > 6888).

Per-GPU throughput spans a narrow ~6.8–7.4k tok/s/GPU band across all cells — the architecture sets a ceiling — but within that band W4A8 sits at the top.

TP4 for this checkpoint is not yet benched — see To-do. Given W4A8-TP2 already leads on both TTFT and per-GPU, W4A8-TP4 is the most likely config to beat the FP8-TP4 892 ms absolute latency.

Why the activation-precision lever doesn't help

At prefill batch sizes, the DeepSeek-V4-Flash MoE (top-6 of 256 small experts) is weight-bandwidth-bound, not compute-bound on the expert GEMM. INT4 weights are already the bandwidth-optimal format, and Marlin's INT4→BF16 path already matches the Cutlass W4A8 kernel in practice. Switching activations from BF16/FP8-implicit to FP8 changes the activation precision but not the dominant cost. The compute-bound portion of prefill is dominated by format-shared work — FP8-block MLA attention and the sparse / Lightning-indexer passes over long context — which is identical across all three checkpoints.

The prefill ceiling is architectural on Hopper

Prefill scales linearly above ~~8k tokens (~~+547 ms per +8k) with GPUs at ~100% util and ~690 W (near TDP) → tensor-core-bound, not launch- or attention-quadratic-bound.
The two kernel improvements that would help — native NVFP4 MoE GEMM and the FP4 Lightning-indexer cache — are Blackwell-only (SM100). On Hopper, sglang/vLLM fall back to Marlin.
A W4A8 SM90 grouped-GEMM tuned for the DeepSeek-V4 MoE path is unimplemented upstream (relevant issues closed inactive). Even so, the wash above suggests it would offer little at prefill batch-M.

What does move the needle (deployment)

Prefix caching is the dominant lever: in production, DeepSeek-V4-Flash realizes ~55% radix prefix-cache hit on real agent/RAG traffic (measured over 24h), i.e. more than half of all prefill is skipped. This is already captured by sglang RadixAttention in production.
Larger chunked-prefill (8192 → 16384) gives ~7% faster long-context prefill TTFT on sglang, at the cost of KV-concurrency — a free win when the server isn't KV-bound.

Bottom line

W4A8 is the best-measured DeepSeek-V4-Flash config on Hopper at TP2 — top prefill TTFT and top per-GPU throughput. It ties its W4A16 sibling (so the ~2× hypothesis failed), but it beats the FP4-marlin config that ships in production by ~9–13% on the same footprint. The practical catch is that this checkpoint format loads on vLLM only, so capturing that win over an sglang FP4 deployment means an engine switch, not a config swap. The dominant serving lever remains prefix caching (~55% radix hit in prod); larger absolute-latency wins beyond this need Blackwell (native NVFP4 + FP4 indexer).

To-do

Bench TP4 for this checkpoint. W4A8-TP2 already leads the matrix on TTFT and per-GPU; W4A8-TP4 is the strongest candidate to beat the FP8-TP4 892 ms absolute TTFT while keeping INT4 weight footprint. (Not yet run.)

Reproducibility

Weights: byte-identical to canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP.
Transformation: the single config.json input_activations edit shown above (see the _w4a8_conversion provenance key).
To rebuild: take the W4A16 base, apply the config edit, serve with the vLLM nightly + patches above.

Acknowledgements

Built and benchmarked by Evrard Nil with Claude (2026-06). Base quantization by canada-quant; original model by DeepSeek-AI.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider