Why this exists
The shipped 510 GB checkpoint fits four GB10s only with the Engram tables on NVMe and ~7 GiB per rank left for KV (1.16 M tokens at 80 % memory utilization). Shrinking the routed experts from 4.25 to ~3.5 bpw frees ~15 GiB per rank for KV and concurrency on the same four nodes, with no measurable loss on our calibration rows.
What is in the repository
Table with columns: file, content| file | content |
|---|
model-000NN-of-00048.safetensors | same shard count and numbering as the release. Body shards (3–42) carry EXL3 expert tensors (…experts.E.w1/w3/w2.{trellis,suh,svh,mul1}) next to the untouched non-expert tensors; shards 1, 2, 43–48 are byte-identical to the release (vision, embeddings, norm/head, drafter, Engram tables). |
config.json, quantization_config.json | quant_method: "exl3", per-layer expert_bits (gate/up and down K), per-module tensor_storage; DeepSeek's original quantization block kept as original_quantization_config. Read by cuda-exl3's Exl3Config; no --quantization flag needed. |
tokenizer*, chat_template, generation_config.json, *.py, README | copied from the release |
| Engram hot-row ids | not shipped (removed 2026-09-11): a 20 M-row resident set measured −4 to −6 % single-stream on both lines (CPU hit/miss split); the frequency ledger itself (top 100 M ids per table = 92.7 % held-out coverage) is available on request and described below |
Best serving configuration so far (auto-updated 2026-09-11 11:38 Pacific)
Line A, row exl3-ROCE-A (accepted rung: ROCE=1). Settings on top of the recipe defaults: KV cache pinned = 12884901888, gpu-memory-utilization = 0.78, launcher MemFree floor (GiB) = 113, NCCL channels = 8, b12x RoCE one-shot all-reduce = 1, async scheduling = 1, max context = 1000000, max concurrent seqs = 8, DISABLED_KERNELS = FlashInferCutedslMxfp8LinearKernel,FlashInferCutlassMxfp8LinearKernel,MarlinMxfp8LinearKernel.
Table with columns: best so far | best so far |
|---|
| single stream, aggregate / per-stream tok/s | 56.1 / 62.2 |
| 4 streams aggregate tok/s | 151.4 |
| 6 streams aggregate tok/s | 199.7 |
| mean TTFT at C1 | 0.26 s |
| cold prefill 3K / 12K / 47K / 93K tok/s | 1313 / 1390 / 1440 / 1443 |
Tuning ladder (accept = +3 % single-stream or 6-stream aggregate with ≤5 % prefill loss at 47K and smokes passing; levers stack per line, winners cross-applied):
Table with columns: line, row, lever, C1/stream, C6 agg, prefill@47K, verdict| line | row | lever | C1/stream | C6 agg | prefill@47K | verdict |
|---|
| A | exl3-G0-A-base0052 | base | 61.1 | 178.7 | 1363 | base |
| A | exl3-G0-A-base0250 | base | 62.5 | 193.4 | 1440 | base |
Levers in the served configurations and where they come from (full ledger in CREDITS.md):
- pinned KV cache — our fix for vLLM sizing KV from rank 0 only under the uneven expert split
- per-rank memory gate — ours; unconditional load-time flush idea from tonyd2wild
- NCCL channels 8 — lever from tenaiaiai's dsv41-flash-4x-dgx-spark-ja recipe (https://github.com/tenaiaiai/dsv41-flash-4x-dgx-spark-ja)
- b12x one-shot RoCE all-reduce — Luke Alonso and the b12x contributors (https://github.com/local-inference-lab/b12x), vLLM shim ported from local-inference-lab/vllm
- async scheduling — vLLM project
- b12x MXFP8 dense GEMM kernel (
B12xMxfp8LinearKernel, b12x by Luke Alonso and contributors) selected over the CUTLASS path — idea from MiaAI-Lab's DeepSeek-v4.1-Flash-DGX-Sparks write-up (SGLang), re-measured on our stack
Measured — pre-serving (DeepSeek's reference forward with every routed expert replaced by its EXL3 reconstruction; 39 in-domain calibration rows × 2048 tokens, paired per row against bf16)
Table with columns: recipe, experts bpw, NLL bf16 → EXL3, paired ΔNLL (sem), rows worse| recipe | experts bpw | NLL bf16 → EXL3 | paired ΔNLL (sem) | rows worse |
|---|
| all layers K=3 | 3.01 | 1.3209 → 1.3152 | −0.0057 (0.0031) | 16 / 39 |
| this build (K=3 / K=4 mix: 19 layers gu4/down4, 3 gu3/down4, 18 gu3/down3) | 3.51 | 1.3209 → 1.3177 | −0.0032 (0.0055) | 19 / 39 |
The per-matrix weight relative error at K=3 is 0.167 and the output does not move: expert-output errors average out through top-6 routing and the residual stream. Caveat: these rows are the Hessian calibration set (in-domain chat). The held-out checks are the served numbers below.
Measured — served (TP4, vLLM + cuda-exl3, DSpark k=5, FULL_AND_PIECEWISE graphs, 300K context, gmu 0.80; bench script and prompt set v1 are the upstream recipe repo's, byte-identical)
Table with columns: upstream recipe, published boot 10 (its hardware), shipped MXFP4/FP8 checkpoint (same recipe, our 4 nodes), this build (our 4 nodes) | upstream recipe, published boot 10 (its hardware) | shipped MXFP4/FP8 checkpoint (same recipe, our 4 nodes) | this build (our 4 nodes) |
|---|
| single stream, aggregate / per-stream tok/s | 37.95 / 43.12 | 42.3 / 50.5 | 54.7 / 61.1 |
| 4 streams aggregate tok/s | 85.72 | 102.7 | 133.1 |
| 6 streams aggregate tok/s | 131.86 | 130.7 | 178.7 |
| cold prefill 3K / 12K / 47K / 93K tok/s |
This build only (no shipped-checkpoint counterpart was measured):
Table with columns: this build | this build |
|---|
| ppl probe (6 held-out texts, 4,210 tokens) | 4.043 |
| HumanEval / HumanEval+ pass@1 (greedy, evalplus) | 0.951 / 0.921 |
| needle at 219K tokens (2 keys) | PASS / PASS |
| tool-call integrity (12 calls) | PASS |
1M-context serving row (line B, pin + NCCL channels 8, --max-model-len 1000000) | C1 54.8 / 59.9, C4 141.8, C6 184.9, prefill 673 / 1007 / 1475 / 1449; 3.41 M tokens KV (3.4 full-length requests) |
| production (1M context), line A: pin + NCCL ch 8 + RoCE + async + b12x MXFP8 dense kernel | C1 55.9 / 61.4, C4 147.2, C6 192.0, TTFT 0.25, prefill 1142 / 1326 / 1385 / 1406; KV 3.41 M tokens (11 Sep 11:36) |
| production (1M context), line B: pin + NCCL ch 8 + RoCE + b12x MXFP8 dense kernel |
How it was made (Pollard-method, "route B")
- Exact bf16 upscale of the release (fp8 · 2^(ue8m0−127) 32×32 blocks; MXFP4 e2m1 LUT × per-32 ue8m0 scale), round-trip checked. There is no native bf16 release; the source is a 4.25-bit QAT checkpoint. Gate-0 measured that EXL3 on these FP4-grid weights behaves exactly like a Gaussian control, so the grid neither helps nor hurts the trellis quantizer.
- Calibration forward = DeepSeek's reference
model.py/engram.py (no framework loads this architecture), streamed layer by layer on ten GB10s: 384 × 2048 in-domain rows rendered with DeepSeek's DSML encoder; per-layer FFN input + routing dumps; Engram rows gathered by hash id; reference NLL per node.
- Per-expert Hessians (H shared by gate/up; H_down from silu(xW1)·xW3 of the routed tokens) → exllamav3's
quantize_exl3 per expert (K=3 all layers, 57 min/layer per GB10; K=4 down-only 16 min; K=4 gate/up 30 min), with a JSON ledger per layer.
- Allocation: cost(layer, matrix, K) = gain² · Σ_experts tokens · proxy_err, where gain is the layer's measured hyper-connection write gain into the residual (5–7.5× at layers 0–13, ≈1 at 19–30, 0.03–0.2 at 36–39). Greedy K=3→4 to 3.5 bpw; K=4/K=3 error ratio measured 0.254. Result: down-proj K=4 on layers 0–14, 16–18, 20–27; gate/up K=4 on layers 0–14, 16–18, 20–22, 24, 25; layers 28–39 stay K=3.
- Splice: expert tensors rewritten per body shard, everything else hardlinked from the release;
quantization_config in the cuda-exl3 layout.
Measurements and (sanitized) tooling are contributed to the Pollard Weights repository (, merged).
Engram
The two n-gram tables (203 GB fp8) are unchanged; the serving recipe reads their rows from NVMe before each forward. Two measured facts for anyone making them resident: fp4 rows (MXFP4 or NVFP4) cost no NLL (Δ −0.003 ± 0.003 while 80–100 % of elements change), and row accesses are Zipfian (top 1 M of 384 M rows = 62 % of lookups in-sample, top 5 M = 89 %); our frequency ledger (not shipped; ask) lists the top 100 M ids per table = 92.7 % held-out coverage (24.6 GiB fp8 per table = ~6.1 GiB per TP4 rank per table, half at fp4; coverage curve: 1 M 43 %, 5 M 59 %, 10 M 67 %, 20 M 74 %, 50 M 84 %, 100 M 93 %); a reference implementation (resident hits by binary search, misses to disk) is in our recipe notes.
Serving
- Recipe: tonyd2wild's DeepSeek-V4.1-Flash-vLLM-DGX-Spark (worker-first launch, Engram on disk staged before the forward, SM12x sparse-MLA page size, DSpark k=5, CUDA graphs, node-local Engram rows). Our additions: cuda-exl3 1.0.3 + a V4.1 overlay (expert naming w1/w3/w2, hybrid config delegating non-EXL3 modules to the fp8 path, clamped SwiGLU limit 10, shared experts kept fp8), the b12x one-shot RoCE all-reduce (2.5× faster than NCCL at the decode-step size on two nodes; served: +7 % C6 on one line, +2 % on the other, accepted in both bases), the GB10 spin-wait fix, and a boot ritual for GB10 unified memory (drop caches, GPU reclaim, cache-flusher during load, min_free 1 GiB, MTU 9000 jumbo check, GPU fast/slow-state probe, CUDA-visible free-memory gate, image/patch parity across ranks).
- Container path must be identical on every rank;
quantization_config.json must sit next to the weights.
- Known GB10 caveat (tonyd2wild issue #1): a hidden GPU slow state can move any single measurement by ~1.5×; probe before you bench.
Credits — this is assembled on other people's work
- DeepSeek-AI — DeepSeek-V4.1-Flash (model, weights, MIT), the reference inference code (
model.py, engram.py, kernel.py), the DSML chat encoder and deepseek-recipe, DeepGEMM, and the architecture itself (hyper-connections, CSA2 sparse attention, Engram, DSpark).
- turboderp — ExLlamaV3: the EXL3 trellis format and the
quantize_exl3 LDLQ quantizer every expert here went through.
- cuda-exl3 contributors — cuda-exl3: the vLLM plugin and CUDA kernels (EXL3 GEMM/MoE, Hadamard, MLA decode) that serve these experts; our overlay (expert naming, hybrid config, SwiGLU clamp, shared-expert fp8) is a small layer on top and is offered upstream.
- tonyd2wild — DeepSeek-V4.1-Flash-vLLM-DGX-Spark (MIT): the 4× Spark recipe this build targets, the seven patches, the launch/boot protocol, the bench protocol and prompt set our numbers use, the GB10 slow-state probe, and the node-local Engram rows idea.
Built by bot-labs-21 with AI assistance (Claude Code); every number was measured on our own hardware. Corrections to any attribution are welcome — open an issue.
License
MIT for our glue, following the base model's license; the model weights remain under DeepSeek's terms.