TL;DR
Table | |
|---|
| Recommended hardware | RTX PRO 6000 Blackwell at TP=2 (2 GPUs/replica) or TP=4 (4 GPUs/replica) — both validated · or 8× H200 TP=2 |
| Quality | GSM8K 93.71% (8-shot strict); HumanEval 84.76% pass@1; MMLU 86.88% |
| Throughput | RTX PRO 6000 98.83 @ TP=2 / 107.32 @ TP=4 at bs=1; 88.35 on H200 TP=2 |
| MTP acceptance | 89% calibrated workload / 70% on random prompts at bs=1 k=1 |
| Spec-decode speedup | 1.49× at bs=1, k=1 (TPOT 6.02 ms vs 8.93 ms, same artifact) |
| Differentiator | First V4-Flash W4A16 quant where MTP survives the calibration load; transformers 5.8.1 silently strips MTP keys by default |
Why this exists
The W4A16-FP8 predecessor and RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8 both drop the MTP block because transformers 5.8.1's DeepseekV4PreTrainedModel declares:
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
which silently filters every mtp.* tensor at from_pretrained time — without warning, without error. Calibration pipelines that go through from_pretrained produce quantized main weights paired with an absent MTP block; serving falls back to plain decode, losing the ~1.5–2× spec-decode speedup that V4-Flash's architecture provides.
This artifact bypasses the silent drop, runs the full 8-rank GPTQ calibration on a 768-sample corpus against the main routed experts, preserves the MTP block unquantized in BF16, and produces a serving artifact where speculative decoding actually fires.
Architecture & precision
Base model
Table with columns: Property, Value| Property | Value |
|---|
| Total parameters | ~284 B (~13 B active per token) |
| Decoder layers | 43 |
| Routed experts / layer | 256 (top-K = 6) |
| Hidden size | 4096 |
| Base BF16 size | ~543 GB |
| Quantized size | 159 GB (+3 GB vs predecessor for the BF16 MTP block) |
Component precisions
Table with columns: Component, Format, Method| Component | Format | Method |
|---|
| Routed experts (256 × 43 layers × 3 projections) | W4A16 INT4, group_size=128, symmetric | GPTQ via llm-compressor, 768 calibration samples |
Attention path (wq_a, wq_b, wkv, wo_a, wo_b, indexer, compressor) | FP8_BLOCK 128×128 | Dynamic scales, scale_fmt=ue8m0 |
MTP block (mtp.0.*) | |
Hardware validated
Table with columns: Platform, SM, HBM/GPU, Interconnect, TP, Role| Platform | SM | HBM/GPU | Interconnect | TP | Role |
|---|
| 8× NVIDIA H200 SXM5 | 9.0a | 141 GB HBM3e | NVLink | 2 (4× replicas) | Calibration + initial benchmarks (p5en.48xlarge) |
| 4× NVIDIA RTX PRO 6000 Blackwell Server Edition | 12.0, sm_120 | 96 GB HBM | PCIe | TP=2 (2 GPUs, 2 replicas on a 4-GPU box) or TP=4 (4 GPUs, 1 replica) | Workstation Blackwell deployment + $/token sweet spot |
Same artifact, no weight changes between SKUs. Both validated cuda graphs ON.
Benchmarks
All numbers from the same artifact, vLLM HEAD 50d9dd902 + 4 patches cherry-picked (PRs #43248 / #43288 / #43290 / #43319).
Quality
Sampling: greedy, temperature 0. Methodology disclosed per row.
Table with columns: Benchmark, Setting, This artifact, Predecessor (W4A16-FP8, no MTP), RedHat (NVFP4-FP8, no MTP), Delta| Benchmark | Setting | This artifact | Predecessor (W4A16-FP8, no MTP) | RedHat (NVFP4-FP8, no MTP) | Delta |
|---|
| GSM8K | 8-shot, strict-match | 93.71% ± 0.67 | 95.07% (RTX PRO 6000) / 95.45% (Spark) | 91.0% (self-reported) | -1.28 pts vs predecessor (within 1 SE) |
| GSM8K | 8-shot, flexible-extract | 93.63% ± 0.67 | 95.37% (Spark) | — | within SE |
| MMLU |
¹ The prior 30.0% AIME number was an lm-eval-harness aime24 task artifact — completions-mode prompt (no chat template), exact_match scorer on a thinking-mode model whose answers are wrapped in <think>…</think> + \boxed{N}. The scorer matched the literal answer string and missed virtually every correct response. A 1-shot smoke under proper chat-templated thinking=high methodology returned 2024-II-4: pred=33, exp=33, correct in 2072 completion tokens — model behavior is correct. Full 30-problem re-bench attempted 2026-05-24 on this RTX PRO 6000 box hit a reproducible CUDA illegal memory access (Worker_TP2: torch.AcceleratorError) under any concurrent thinking-mode load (cuda graphs and --enforce-eager both crash; concurrency=4 dies at ~11 min, concurrency=8 dies at ~90 s). Single-shot inference works. Re-bench deferred to H200 with jasl/vllm@ds4-sm120-experimental@abad5dc71 (the build the original Card D H200 numbers used).
² IFEval re-bench attempted 2026-05-24 hit the same RTX PRO 6000 stability issue. Deferred alongside AIME.
Throughput
vllm bench serve random 256-in / 256-out, MTP-spec num_speculative_tokens=1 (k=1 cap on this build — see Honest limitations), cuda graphs ON.
Table with columns: Hardware, TP, bs=1 output tok/s, bs=1 TPOT median, bs=4 output tok/s, bs=16 output tok/s, MTP acceptance @ bs=1| Hardware | TP | bs=1 output tok/s | bs=1 TPOT median | bs=4 output tok/s | bs=16 output tok/s | MTP acceptance @ bs=1 |
|---|
| 8× H200 | 2 (per replica) | 88.35 | 6.02 ms | 138.80 | 367.13 | 89% calibrated / 70% random |
| 4× RTX PRO 6000 box | TP=2 (per replica, 2 replicas fit) | 98.83 | 8.55 ms | 219.53 |
Per-replica, RTX PRO 6000 wins output throughput at every batch size; H200 still wins per-token TPOT median.
MTP draft-token acceptance per workload
Same artifact, bs=1, k=1.
Table with columns: Workload, Prompts, Accepted / emitted, Acceptance| Workload | Prompts | Accepted / emitted | Acceptance |
|---|
| Random 256-token prompts (200 samples) | random | 21024 / 30058 | 69.94% |
| Code, raw completion (15 short signature+docstring prompts) | code-raw | 1847 / 1988 | 92.91% |
| Chat-templated prose (15 prompts) | chat-prose | 1946 / 2376 | 81.90% |
| Raw natural language (15 continuation prompts) | nl-raw | 1745 / 2086 |
Spec-decode wins at low concurrency (single-user interactive). At bs≥4 the verifier is already filling its batch lane, so extra verifier passes add overhead without saving wall-clock — matches the sibling artifact's framing of bs=1 as the headline operating point.
Cost per output token (node-level)
Boxes priced for cloud-rented hardware. Single-replica numbers measured; multi-replica totals are linear extrapolation.
Table with columns: Box, Replicas, bs=1 total tok/s, bs=16 total tok/s, $/h, $/(1000 tok/h) at bs=1| Box | Replicas | bs=1 total tok/s | bs=16 total tok/s | $/h | $/(1000 tok/h) at bs=1 |
|---|
p5en.48xlarge (8× H200) | 4× TP=2 | ~353 | ~1468 | $98 | $278 |
g7e.24xlarge (4× RTX PRO 6000) | 2× TP=2 | ~198 | ~965 | $19.92 | |
At bs=1 (interactive), RTX PRO 6000 2×TP=2 is ~2.7× cheaper than H200 4×TP=2. At bs=16 the gap narrows because H200's per-replica throughput scales better with batch — H200 wins absolute throughput when you can fill it; RTX wins on $/token unless you genuinely need >1500 tok/s aggregate output.
Cross-validation: 2026-05-29 fresh Docker on RTX PRO 6000 ✓
Hardware coverage: all numbers below are from RTX PRO 6000 Blackwell Server Edition (SM 12.0a) on a Brev g7e.24xlarge. The image is expected to work on Workstation Edition (same SM 12.0a, same Marlin native cubins, same model + serve path) but we have not directly verified it ourselves. Reference TP=2 Workstation numbers from jasl's bench harness (baselines/20260512_sm120_deployment_1c20f1a6d) confirm the underlying stack runs on Workstation. Expect a 5-15% throughput delta from clock/memory-bandwidth differences. If anything misbehaves on Workstation Edition, open an issue at the repo.
Full bench matrix on canada-quant/dsv4-w4a16-rtxpro6000:v1 (the HF-published Docker image, built from jasl/vllm@27fd665b + canada-quant BF16-MTP cherry-pick + Marlin MoE c_tmp/workspace patches). All AIME runs at max_tokens = max_model_len - 500 = 65036 so reasoning runs to natural stop:
Table with columns: AIME-2024 thinking-mode sweep (c=4, n=30), TP=2 (max_num_seqs=4), TP=4 (max_num_seqs=16)| AIME-2024 thinking-mode sweep (c=4, n=30) | TP=2 (max_num_seqs=4) | TP=4 (max_num_seqs=16) |
|---|
| chat (no think) | 18/30 · MTP 95.78% · 53m | 19/30 · MTP 93.06% · 5m |
| thinking-high | 27/30 · MTP 91.97% · 152m | 29/30 · MTP 91.01% · 13m |
| thinking-max | 24/30 · MTP 92.52% · 177m | 27/30 · MTP 91.68% · 26m |
Table with columns: AIME-2024 single-shot reference (c=1, thinking-high, n=30), TP=2, TP=4| AIME-2024 single-shot reference (c=1, thinking-high, n=30) | TP=2 | TP=4 |
|---|
| c=1 high | 27/30 · MTP 91.68% · 48m | 28/30 · MTP 90.76% · 41m |
Table with columns: GSM8K (n=50, 8-shot), TP=2, TP=4| GSM8K (n=50, 8-shot) | TP=2 | TP=4 |
|---|
| flexible-extract | 45/50 (90.0%) | 43/50 (86.0%) |
| strict-match | 42/50 (84.0%) | 40/50 (80.0%) |
Table with columns: Throughput random 256/256 (single replica, MTP on), TP=2 tok/s @ TPOT p50, TP=4 tok/s @ TPOT p50| Throughput random 256/256 (single replica, MTP on) | TP=2 tok/s @ TPOT p50 | TP=4 tok/s @ TPOT p50 |
|---|
| bs=1 | 95.2 @ 8.05 ms | 108.1 @ 7.32 ms |
| bs=4 | 40.6 @ 83.18 ms | 104.3 @ 11.31 ms |
| bs=8 | 45.7 @ 79.02 ms | 433.2 @ 16.44 ms (sweet spot) |
| bs=16 | 34.9 (capped by max_num_seqs=4) | 164.3 (scheduler thrash) |
Table with columns: Throughput random 1024/1024, TP=2, TP=4| Throughput random 1024/1024 | TP=2 | TP=4 |
|---|
| bs=1 | 30.7 tok/s | 138.1 tok/s |
| bs=4 | 45.1 tok/s | 363.7 tok/s |
Headlines from this run:
- Zero CUDA illegal-memory-access in 240 AIME thinking-mode problems across c=4 chat/high/max + c=1 high on both TP=2 and TP=4 = the Marlin MoE concurrent-decode race is fixed by the
c_tmp clamp removal in PR vllm#43730 (which is baked into the v3 image via jasl/vllm@27fd665b).
- TP=4 is 7-12× faster than TP=2 at AIME (chat 53m→5m, high 152m→13m, max 177m→26m). MoE expert sharding across 4 GPUs decisively wins.
- Thinking-max regresses correctness AND triples wall (TP=4: high 29/30 in 13m vs max 27/30 in 26m). The artifact's sweet spot is
reasoning_effort=high.
- MTP holds 91-93% across all thinking modes and TP configs — the BF16-retained draft head is doing its job everywhere.
Raw JSON + per-bench logs in the reproduction repo.
Tuning attempts that DID NOT win on TP=4 Server (documenting so you don't repeat them)
We A/B-tested adopting jasl's TP=2 Workstation env tunings at TP=4 Server — none of them transferred. Stick with the v3 image defaults:
Table with columns: Change from defaults, Result, Why| Change from defaults | Result | Why |
|---|
num_speculative_tokens=2 (jasl's deepseek_mtp k=2 default) | −86% bs=8 (433 → 60 tok/s) | k=2 doubles main-model forward cost; at TP=4 the all-reduce overhead exceeds the ~1.5 tokens-per-draft acceptance gain that's net-positive at TP=2 |
--enable-expert-parallel (jasl recommends) | similarly bad combined with k=2 | TP=4 all-to-all expert-gather is expensive |
VLLM_TRITON_MLA_SPARSE_QUERY_CHUNK_SIZE=512 + ..._TOPK_CHUNK_SIZE=512 (jasl's chunk-size tunings) | CUDA illegal memory access at cudagraph capture | Tuned for SM 12.0a c128a Workstation single-request prefill; exceed safe limits at TP=4 Server |
The image's defaults are the optimal config for TP=4 RTX PRO 6000 Server Edition as of 2026-05-29. If you're deploying on TP=2 Workstation Edition, jasl's reference config (sm120_tp2_serve.env.example) is the right starting point — it was tuned on that exact hardware.
Quick start
RTX PRO 6000 Blackwell — Docker (recommended)
The pre-built canada-quant/dsv4-w4a16-rtxpro6000:v1
image bakes the full 13-layer recipe (jasl/vllm@27fd665b + canada-quant BF16
MTP cherry-pick + Marlin MoE c_tmp/workspace patches + cute.arch.fmin shim).
~3-5 min from docker load to a working endpoint on a g7e.24xlarge.
# 1. Pull the image tarball (~14 GB compressed)
hf download canada-quant/dsv4-flash-w4a16-rtxpro6000-image \
--include "*.tar.gz" --local-dir .
docker load < dsv4-w4a16-rtxpro6000-v1.tar.gz
# 2. Cache the W4A16 model onto NVMe (~159 GB, ~1-2 min via xet on Brev)
HF_HOME=/opt/dlami/nvme/hf-cache hf download \
canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP
# 3. Pull the serve helper
git clone https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp.git
cd dsv4-flash-w4a16-fp8-mtp
# 4. Serve TP=2 (or TP=4 with --gpus all -e TP=4 -e MAX_NUM_SEQS=16)
docker run -d --gpus '"device=0,1"' --name dsv4-w4a16-serve \
--shm-size=16g --ipc=host -p 8000:8000 \
-v /opt/dlami/nvme/hf-cache:/root/.cache/huggingface \
-v $(pwd)/scripts:/workspace/scripts:ro \
-e TP=2 -e MAX_NUM_SEQS=4 -e MAX_MODEL_LEN=65536 -e GPU_MEM_UTIL=0.95 \
canada-quant/dsv4-w4a16-rtxpro6000:v1 \
bash /workspace/scripts/serve_rtx6000pro_w4a16.sh
# 5. Wait for /v1/models (~3-5 min model load + cudagraph capture)
until curl -sf http://127.0.0.1:8000/v1/models >/dev/null; do sleep 5; done
# 6. Run the full bench matrix (AIME chat/high/max + GSM8K + throughput)
docker exec dsv4-w4a16-serve bash -c \
"TAG=tp2_64k MAX_MODEL_LEN=65536 bash /workspace/scripts/bench_matrix.sh"
RTX PRO 6000 Blackwell — from-source install (advanced)
# 1. Bootstrap vLLM (~25 min for source build)
git clone https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp.git
cd dsv4-flash-w4a16-fp8-mtp
bash scripts/bootstrap_rtx6000pro.sh
# 2. Extra pins
source ~/venv-serve/bin/activate
pip install --quiet "flashinfer-python==0.6.8.post1" "flashinfer-cubin==0.6.8.post1" \
"numba==0.65.0" "tilelang==0.1.9" "apache-tvm-ffi==0.1.9" "fastsafetensors>=0.2.2"
# 3. Apply patches
python scripts/patch_v4_forcausal_packed_mapping.py "$(python -c 'import vllm; print(vllm.__path__[0])')"
python scripts/patch_mtp_packed_mapping.py "$(python -c 'import vllm; print(vllm.__path__[0])')"
python scripts/patch_nvidia_attn_scale.py "$(python -c 'import vllm; print(vllm.__path__[0])')"
bash scripts/patch_wo_a_bf16_path.sh "$(python -c 'import vllm; print(vllm.__path__[0])')"
# 4. Download artifact (159 GiB) — already dequant'd in-artifact as of 2026-05-24,
# no local preprocessing step required.
hf download canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP \
--local-dir /scratch/weights/w4a16-fp8-mtp-gptq
# 5. Serve TP=2 (or TP=4 with 0,1,2,3)
CUDA_VISIBLE_DEVICES=0,1 bash scripts/serve_rtx6000pro.sh \
/scratch/weights/w4a16-fp8-mtp-gptq 8000 2
Required runtime env vars on SM 12.x (already set inside serve_rtx6000pro.sh but worth knowing):
export VLLM_TRITON_MLA_SPARSE=1
export VLLM_TRITON_MLA_SPARSE_HEAD_BLOCK_SIZE=4
export VLLM_USE_FLASHINFER_SAMPLER=0
Without VLLM_TRITON_MLA_SPARSE_HEAD_BLOCK_SIZE=4 the sparse-MLA Triton kernel can crash during warmup with RuntimeError: Triton Error [CUDA]: an illegal memory access in _dequantize_and_gather_k_kernel. The FlashInfer sampler is also broken on TORCH_CUDA_ARCH_LIST=12.0a — fall back to PyTorch-native via VLLM_USE_FLASHINFER_SAMPLER=0.
H200
vllm serve canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP \
--tensor-parallel-size 2 \
--kv-cache-dtype fp8 --block-size 256 \
--max-model-len 4096 \
--gpu-memory-utilization 0.80 \
--no-enable-prefix-caching \
--tokenizer-mode deepseek_v4 \
--tool-call-parser deepseek_v4 --enable-auto-tool-choice \
--reasoning-parser deepseek_v4 \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
--trust-remote-code
Quantization recipe
Table with columns: Property, Value| Property | Value |
|---|
| Dataset | HuggingFaceH4/ultrachat_200k (V4 chat template) |
| Samples | 768 |
| Max sequence length | 512 |
| Per-rank batch size | 4 |
| Calibration hardware | 8× NVIDIA H200 (p5en.48xlarge) |
| Walltime | ~15.4h (15.09h oneshot + ~16 min save) |
| Per-subgraph cadence | ~20 min/subgraph × 44 subgraphs (43 MoE + 1 MTP no-op) |
Calibration recipe identical to the W4A16-FP8 predecessor with one change: the modeling class is patched to remove mtp.* from _keys_to_ignore_on_load_unexpected before from_pretrained, so the MTP block survives the load and is written back to the artifact at BF16.
vLLM build
RTX PRO 6000 Blackwell (SM 12.0) only
Table with columns: Patch, Purpose| Patch | Purpose |
|---|
packed_modules_mapping on DeepseekV4ForCausalLM + DeepSeekV4MTP | Required as of ds4-sm120-experimental@abad5dc71 |
BF16 wo_a path for MTP block | Static weight.dtype == bfloat16 check (dynamo-safe) |
--disable-custom-all-reduce | No NVLink between RTX PRO 6000 boards |
CMakeLists USE_SABI 3.11 removal | For Python 3.10 |
(Previously this list also required a compressor/indexer FP8 → BF16 dequant preprocess step run against the local artifact. As of 2026-05-24 the dequant is baked into the published artifact — see Changes.)
H200 deployments need only the four common patches.
Honest limitations
- k=1 cap on spec-decode — current vLLM build limits
num_speculative_tokens to 1 due to DeepGemm kernel assertion next_n == 1 or next_n == 2 in smxx_fp8_fp4_paged_mqa_logits.hpp:233. vLLM passes next_n = num_speculative_tokens + 1, so practical k is 1. The FLASHINFER_MLA_SPARSE attention backend hits the same kernel-side assertion. With the assertion relaxed, expect bs=1 speedup to rise from 1.49× to ~1.85× (matching sibling NVFP4 artifact's k=2 published number).
- Concurrent thinking-mode workloads on RTX PRO 6000 produce token-corrupted output — under concurrency ≥ 2 with thinking=high (long-decode workloads like AIME), the Marlin W4A16 MoE decode kernel on SM 12.0 produces token-stream corruption (CJK / Cyrillic / garbled ASCII spliced into the model's reasoning trace). The same hardware + same vLLM build serving the NVFP4 sibling (
canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP via flashinfer_trtllm MoE) is essentially clean on the same workload (1/30 vs 14/30 corrupted at c=4 thinking). The bug is specific to the W4A16 + Marlin MoE decode path on SM 12.0. Investigation isolated through 7 controlled tests (sparse-MLA topk-chunk size, MTP-off, matmul_decode-off, eager-mode, concurrency sweep, NVFP4 vs W4A16 path comparison). for batched thinking-mode workloads, serve the NVFP4 sibling artifact instead. For sequential (c=1) thinking-mode or any batched chat-mode (no thinking), this W4A16-MTP artifact works cleanly (GSM8K-20 chat-mode sequential = 20/20 = 100%, MTP draft acceptance 92.46%). Full debug log + reproducible benches: . Filed upstream as .
Reproduction
Full pipeline at canada-quant/dsv4-flash-w4a16-fp8-mtp. From a fresh 8× H200 box:
# Phase 0 — bootstrap (venv-calib + venv-serve + vendor + apply patches)
bash scripts/bootstrap_p5en_h200.sh
# Phase 1 — download upstream + dequant to BF16-MTP source (~30 min, ~660 GB)
bash scripts/phase1_dequant.sh
# Phase 2 — GPTQ calibration (8 ranks, ~15h wall)
bash scripts/run_phase2.sh
# Phase 3 — postprocess (rename + config patch + FP32 restore + MTP aliases)
bash scripts/postprocess_phase2.sh
# Phase 4 — verify
python scripts/verify_option_y.py /scratch/weights/w4a16-fp8-mtp-gptq
# Phase 5 — serve (see Quick start above for serve command)
Upstream contributions filed during this work
Table with columns: Contribution, Description, Status| Contribution | Description | Status |
|---|
transformers — save_pretrained silent FP32 → BF16 downcast | 417 tensors specified as FP32 in DeepSeek's release spec (HC plumbing, gate bias, attn_sink, indexer/compressor ape) are silently written as BF16 by save_pretrained when model torch_dtype is BF16. Workaround: postprocess restore from BF16 source via scripts/fixup_artifact.py. Upstream filing pending | local |
vLLM — MTP loader silently skips top-level head.weight + embed.weight | calls which no-ops on non- keys; returns None → loop skips. and never reach / → uninitialized → . Workaround: postprocess injects and as duplicates. Upstream filing pending |
Changes
Table with columns: Date, Change| Date | Change |
|---|
| 2026-05-22 | Initial release on H200 (jasl/vllm@ds4-sm120-experimental@abad5dc71). GSM8K 93.71% strict, MMLU 86.88%, HumanEval 84.76%, MTP acceptance 89% on calibrated workload / 70% on random prompts |
| 2026-05-24 (morning) | RTX PRO 6000 Blackwell (SM 12.0) added. TP=2 and TP=4 both validated, chat-smoke 4/4 PASS, MTP acceptance 68-72%, MTP-on per-replica throughput 98.83 tok/s @ TP=2 / 107.32 @ TP=4. Per-replica throughput beats H200 at every batch size. vllm-project/vllm#41511 (Marlin TP > 2 bug) did not fire on this build |
| 2026-05-24 (afternoon) | Shipping-bug fix. Artifact previously shipped FP8_BLOCK compressor/indexer with .weight_scale keys; current upstream/preview-dev vLLM constructs those modules as plain BF16 (quant_config=None), so the artifact failed to load with KeyError: 'layers.10.attn.mla_attn.compressor.fused_wkv_wgate.weight_scale'. 166 compressor/indexer weights dequantized in-place (FP8 + BF16 scale → BF16, mathematically lossless) and re-uploaded. Artifact now loads cleanly on modern vLLM on both RTX PRO 6000 and H200 without local preprocessing. H200 historical numbers above remain valid: the original H200 build supported FP8 compressor; modern vLLM serving the new BF16-compressor format produces equivalent outputs (verified post-fix on RTX PRO 6000 TP=4: GSM8K-50 chat-mode 44/50 = 88.0%, matches sibling Card B's 88% strict TP=2 / 90% strict TP=4 on this hardware). |
Files in the artifact
- 4 sharded
model-*.safetensors files + model.safetensors.index.json (159 GB total)
config.json — vLLM-compatible quantization_config with MTP block excludedtokenizer.json, tokenizer_config.json, generation_config.json, chat_template.jinja — upstream DSV4-Flashrecipe.yaml — the llm-compressor GPTQ recipeREADME.md — this file
Citation
@misc{canada-quant-dsv4-flash-w4a16-fp8-mtp-2026,
title = {DeepSeek-V4-Flash W4A16-FP8 with BF16 MTP retained for vLLM speculative decoding},
author = {Canada Quant},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP}
}
License
MIT, inherited from upstream deepseek-ai/DeepSeek-V4-Flash. Review at the upstream repo before commercial deployment.
Acknowledgments