GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Nateemerson
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Nateemerson
Model Tree
deepseek-ai/DeepSeek-V4.1-Flash-1017
Input Modalities
Output Modalities
Supported Functionality
| Files | Contents | Source |
|---|---|---|
model-00001 to model-00043 | Routed experts in EXL3 3 bpw (MUL1 codebook, K=3, uncalibrated); everything else in DeepSeek's FP8 | PixelML/DeepSeek-V4.1-Flash-1017-EXL3-3bpw @ 3d095b72, shards 1-43, unchanged |
model-00044 to model-00046 | DSpark drafter. Its routed experts are re-encoded from MXFP4 to EXL3 3-bit (MCG codebook, uncalibrated), as in the diffbot pack; its other tensors are DeepSeek's | Made here from deepseek-ai/DeepSeek-V4.1-Flash-1017 @ 1bae0e10, shards 44-46, with diffbot's quantize_mtp_exl3.py |
model-00047, model-00048 | Engram tables in int4 with fp16 scales per 32 values (int4_fp16s32), 111 GB instead of 203 GB of FP8 | Made here from the official 1017 shards 47-48 with diffbot's convert_engram_int4.py |
config.json | The diffbot pack's config, with only quantization_config.source changed. Its model fields are identical to DeepSeek's 1017 config | |
model.safetensors.index.json | Rebuilt from the shard headers: 190,549 tensors, the same names as the diffbot pack | |
encoding/, tokenizer*, LICENSE | DeepSeek's, from the 1017 release. Prompts are formatted by encoding/ (there is no Jinja chat template) | |
build/ | The assembly script and build notes | |
results/ | Raw benchmark and quality logs from the runs below |
Why a repack is needed: 1017 has the same architecture and tensor names as V4.1-Flash, but DeepSeek retrained every weight, so all 48 files changed, including the Engram tables and the drafter. The diffbot pack's files can't be mixed with 1017 files. PixelML's repo has the 1017 experts but no Engram tables and an MXFP4 drafter.
You need two sm_120 GPUs with 96 GB each, Docker with the NVIDIA container toolkit, about 250 GiB of free host RAM and 340 GB of disk (the same as the diffbot pack).
bash
# 1. This packhf download Nateemerson/DeepSeek-V4.1-Flash-1017-EXL3-3bpw-2x-RTX-PRO-6000 --local-dir ~/models/DeepSeek-V4.1-Flash-1017-EXL3-3bpw # 2. diffbot's recipe, at the revision tested here (the recipe only, not the old weights)hf download diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000 --revision a1cf1bd227a7 \ --include "recipe/*" --local-dir ~/dsv41-recipecd ~/dsv41-recipe/recipebash docker/build.sh # serving image dsv41-flash-exl3-3bpw-sm120bash kernels/build.sh # the xmoe and xmoe_do MoE kernels # 3. Serve: OpenAI-compatible API on :8000, model name deepseek-v4.1-flashPACK=~/models/DeepSeek-V4.1-Flash-1017-EXL3-3bpw PLAN=plan-blend30-dh.json API_KEY=change-me bash serve/serve.shPACK must point at this repo's folder, because the recipe isn't inside it.PLAN=plan-blend30-dh.json is what every number below used. The recipe's default is plan-agent30-dh.json.NCCL_P2P_DISABLE=1 (GPU-to-GPU P2P hangs there), CONTAINER_MEM=236g (the host has 249 GiB;
the default is 260g) and EXTRA_ARGS="--host 127.0.0.1". Leave P2P on if it works on yours.6062be5f, rebased onto a newer vLLM nightly) was not tested with this pack.Same machine, same image, same recipe, same plan. Only the weights differ between the two columns.
Test (diffbot's recipe/bench scripts) | diffbot pack (V4.1-Flash) | This pack (1017) |
|---|---|---|
| Short-context sweep / tool calls | 11/11 / OK | 11/11 / OK |
| 2K prompt, 1 stream: decode / prefill (tok/s) | 104.8 / 4,051 | 105.0 / 4,265 |
| 2K prompt, 4 streams: decode / prefill, total (tok/s) | 219.7 / 7,900 | 212.7 / 8,783 |
| 46K prompt, 1 stream: prefill / decode (2 runs) | 8,446-8,489 / 116.0, 100.1 | 8,555-8,697 / 109.6, 100.7 |
| 46K prompt, 4 streams: prefill / decode, total |
gen30 plan
(faster on synthetic prompts) on 2× Max-Q cards at 300 W. Ours ran the blend30 plan with P2P off.Test machine:
dsv41-flash-exl3-3bpw-sm120 built from diffbot's recipe at a1cf1bd2.Both packs ran diffbot's quality probe (recipe/bench/quality-probe.py: GSM8K first 200 and HumanEval 164, greedy,
thinking off, 4 streams, max 4,096 tokens) on the same machine and setup. Only the weights differed.
| GSM8K-200 | HumanEval-164 | |
|---|---|---|
| This pack (1017, uncalibrated experts) | 98.5 % (197/200) | 95.1 % (156/164) |
| diffbot pack (V4.1-Flash, calibrated experts), same machine | 98.0 % (196/200) | 95.7 % (157/164) |
| diffbot pack, as published on its card | 98.0 % | 95.1-96.3 % (4 runs) |
The two packs are within one item of each other on both tests, and the diffbot pack's run reproduces its published numbers, so the setup matches.
Item by item:
Both tests are close to their ceiling for this model, so they show no damage from the uncalibrated experts but can't rule out a small loss.
HumanEval ran sandboxed. Each program ran in a fresh container: no network, read-only filesystem, unprivileged user, 512 MB, 15 s. That replaces the probe's direct execution on the host; the rest of the script is unchanged. All 8 failures were wrong answers, not sandbox limits.
Raw per-item outputs, including the generated code, are in results/probe-1017-pack.jsonl and
results/probe-diffbot-pack.jsonl.
What hasn't been measured:
EXL3_EXPERT_STATS + recipe/plans/make_plan.py)
may decode faster.a1cf1bd2, on one machine.All steps follow diffbot's recipe/quantize/README.md, with diffbot's scripts unchanged. build/ has the assembly
script; the times are from this machine.
Download PixelML @ 3d095b72 (all files) and DeepSeek 1017 @ 1bae0e10 (shards 44-48, config, index,
tokenizer, encoding/). Every file used was checked against the Hub's sha256.
Drafter: quantize_mtp_exl3.py --bits 3 --greedy --beam 16 --batch 8 on shards 44-46, on one GPU: 1,152 matrices in
419 s. quantize_mtp_exl3.py imports tools/pack_meta.py from sfxnz's recipe at commit 460ac89c, the commit
diffbot's quantize_mtp_exl3.diff applies to.
Engram: convert_engram_int4.py on shards 47-48: CPU only, about 9 minutes.
Assembly: build/assemble_pack_1017.py. It is diffbot's adapted to PixelML's shard layout:
Built entirely on the work of:
The weights keep DeepSeek's MIT license (LICENSE). Repacking adds no restrictions.
| 15,664 / 208.2 |
| 15,770 / 209.4 |
| Agentic turn, 1 stream: time to first token / decode / tokens per DSpark step | 1.66 s / 111.2 / 3.25 | 1.62 s / 110.0 / 3.30 |
| Agentic turn, 4 streams: time to first token / decode total / tokens per step | 4.65 s / 162.5 / 3.67 | 4.53 s / 158.0 / 3.48 |
| Needle at 100K / 250K / 448K tokens | found in 12.7 / 25.6 / 42.4 s | found in 11.9 / 23.8 / 40.1 s |
assemble_pack.py.mcg tensors its shards don't contain;config.json is the diffbot pack's, with only quantization_config.source changed.The result has exactly the diffbot pack's 190,549 tensor names, and its drafter and Engram tensors have the same dtypes, shapes and sizes as the diffbot pack's.