anemll

DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1

Bundle layout

Table with columns: Path, Contents, Runtime role
Path	Contents	Runtime role
repository root	Prepared NVIDIA NVFP4 target	W4A4 FlashInfer CUTLASS target
`dspark/`	Native `mtp.0`, `mtp.1`, and `mtp.2` shards plus a draft-only index	Three-stage DSpark draft

The root target contains 87 Safetensors files, 3,483 tensors, and 168,266,881,608 payload bytes. The draft contains 4,705 tensors and 10,862,838,300 payload bytes. bundle-manifest.json records the complete layout and provenance.

The target and draft remain two explicit model paths at runtime: mount the repository root as the target and dspark/ as the speculative model.

Requirements

two NVIDIA DGX Spark or equivalent GB10/SM121 nodes;
TP=2 with a dedicated node-to-node fabric;
enough local storage for the complete bundle on each node;
the custom vLLM runtime branch linked above;
an ARM64 runtime image built from that branch.

The checkpoint is not supported as a one-node TP=1 model. The prepared target format itself is pinned to TP=2.

Download

Run on both nodes, choosing a local SSD path with sufficient free space:

bash
hf download anemll/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1 \
  --local-dir /srv/dspark/models/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1

Runtime setup

Clone the matching runtime branch on both nodes:

bash
git clone --branch dspark-nvfp4-a4w4 \
  https://github.com/Anemll/dspark-vllm-gx10.git
cd dspark-vllm-gx10

Use an image built from this branch. The legacy 0.1.1 image does not contain the prepared bulk direct reader unless it has been rebuilt from the branch:

bash
FINAL_IMAGE=dspark-vllm-gx10:nvfp4-a4w4-v1 ./scripts/build-image.sh

Set the following values in both config/head.env and config/worker.env, in addition to their normal rank, fabric, NCCL, cache, and rendezvous fields:

bash
DSPARK_VLLM_IMAGE=dspark-vllm-gx10:nvfp4-a4w4-v1
DSPARK_MODEL_HOST=/srv/dspark/models/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1
DSPARK_DRAFT_MODEL_HOST=/srv/dspark/models/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1/dspark
SERVED_MODEL_NAME=deepseek-v4-flash-nvfp4-dspark

DSPARK_WEIGHT_LOAD_FORMAT=auto
DSPARK_MOE_BACKEND=auto
DSPARK_SPECULATION_MODE=dspark
MTP_NUM_TOKENS=5
VLLM_DSPARK_CONFIDENCE_SCHEDULER=off
VLLM_DSPARK_CONFIDENCE_THRESHOLD=0.0

VLLM_DSV4_NVFP4_CUTLASS_PREPARED_LOAD=1
VLLM_DSV4_NVFP4_CUTLASS_PREPARED_MANIFEST_SHA256=REPLACE_WITH_MANIFEST_SHA256
VLLM_DSV4_NVFP4_CUTLASS_PREPARED_DIRECT_READ=1

Use the first field of dspark-nvfp4-tp2-repack.json.sha256 as the manifest value. Keep the global MoE backend at auto: the prepared target is scoped to FlashInfer CUTLASS by its loader, while the native DSpark draft requires its own backend selection. Direct reads are the prepared-loader default; 0 selects the slower diagnostic mmap path.

Start rank 1 first, then rank 0:

bash
# WORKER / rank 1
./scripts/start-node.sh config/worker.env

# HEAD / rank 0, after rank 1 enters the rendezvous
./scripts/start-node.sh config/head.env

When passwordless worker SSH is configured in the head environment, the head can launch both ranks:

bash
./scripts/start-cluster.sh config/head.env config/worker.env

Startup verification

Follow the container log on either rank:

bash
cid="$(sudo docker ps -q --filter label=com.docker.compose.service=vllm-dspark)"
sudo docker logs -f "$cid"

A valid prepared bulk load reports:

NVFP4_PREPARED event=enabled ... io_mode=preadv;
43 event=layer_load records;
event=complete layers=43 reads=344 copies=344 ... io_mode=preadv;
43 post-load records with transforms=0 and backend FLASHINFER_CUTLASS.

After startup:

bash
curl -fsS http://HEAD_HOST:8888/health
curl -fsS http://HEAD_HOST:8888/v1/models

Validated load performance

The bulk direct reader was validated end to end on two GB10 nodes:

Table with columns: Measurement, Previous prepared path, Bulk direct reader, Improvement
Measurement	Previous prepared path	Bulk direct reader	Improvement
Slower-rank target load	558.19 s	65.23 s	8.56x
Complete head model load	595.90 s	108.54 s	5.49x
Head start to API server	n/a	231.64 s	n/a

The run reached HTTP health 200, exposed the expected model, kept both ranks out of OOM state, and returned a coherent smoke response. Raw evidence is in nvfp4-prepared-direct-read-full-3689b1c.json.

Prefill comparison

These are warmed, server-side prefill rates on the same two-node TP=2 cluster. Prefill is target-only and does not benefit from speculative decoding.

Table with columns: Input tokens, FP8/B12X production, NVFP4 W4A4, Gain, Production TTFT, W4A4 TTFT
Input tokens	FP8/B12X production	NVFP4 W4A4	Gain	Production TTFT	W4A4 TTFT
1,024	2,033.0 tok/s	2,242.5 tok/s	+10.3%	0.512 s	0.463 s
2,048	2,252.0 tok/s	2,473.2 tok/s	+9.8%	0.920 s	0.835 s
4,096	2,320.7 tok/s

The reports are same-size aggregates, not paired identical-prompt trials. See the comparison and raw-artifact links.

Decode and agentic comparisons

All rows use MTP=5, confidence scheduling off, probabilistic draft sampling, no draft/verify overlap optimization, temperature zero, and 512 requested tokens per stream.

Table with columns: Workload, C, FP8/B12X + DSpark, W4A4 + DSpark, Comparison, W4A4 mean accepted length
Workload	C	FP8/B12X + DSpark	W4A4 + DSpark	Comparison	W4A4 mean accepted length
Canonical chat/tool	4	105.48 tok/s	96.02 tok/s	-9.0%	3.127
`tool_agentic`	8	not archived cleanly	360.68 tok/s	no matched control	5.270

The canonical control is a three-trial median; its W4A4 row is one post-promotion run. The agentic row is a clean W4A4 production measurement; the older timing-instrumented concurrency study is deliberately not used as a control. The agentic prompt SHA-256 is 6173a7ae0ea3c64b364d0c405be28808efb8486c68a7011e966d31ce222c1736, and all eight W4A4 streams passed the automated no-collapse diagnostic.

The report is intentionally mixed rather than marketing a single decode number: W4A4 lost about 9% in the exploratory canonical comparison, while the high-acceptance agentic path reached 360.68 aggregate tok/s without a clean legacy control. See the full serving comparison and raw JSON links.

Limitations

This artifact is tied to the custom prepared-loader contract and TP=2.
It is not supported by stock Transformers or stock vLLM.
The target and speculative draft have separate physical layouts and must be mounted at the two paths shown above.
Decode performance depends strongly on prompt content and draft acceptance; report prompt, concurrency, acceptance, output length, and timing convention with every result.
The agentic no-collapse diagnostic is not a comprehensive quality evaluation.
The target originates from NVIDIA's NVFP4 checkpoint and the draft from the native DeepSeek V4 Flash DSpark lineage; review the upstream licenses and terms as well as the files included here.

Reproducibility

Runtime source: Anemll/dspark-vllm-gx10
Prepared format manifest: dspark-nvfp4-tp2-repack.json
Bundle manifest: bundle-manifest.json
Benchmark summary: w4a4-dspark-serving-comparison.md

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

anemll

Model Tree

Base

deepseek-ai/DeepSeek-V4-Flash-DSpark

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Bundle layout

Table with columns: Path, Contents, Runtime role
Path	Contents	Runtime role
repository root	Prepared NVIDIA NVFP4 target	W4A4 FlashInfer CUTLASS target
`dspark/`	Native `mtp.0`, `mtp.1`, and `mtp.2` shards plus a draft-only index	Three-stage DSpark draft

The target and draft remain two explicit model paths at runtime: mount the repository root as the target and dspark/ as the speculative model.

Requirements

two NVIDIA DGX Spark or equivalent GB10/SM121 nodes;
TP=2 with a dedicated node-to-node fabric;
enough local storage for the complete bundle on each node;
the custom vLLM runtime branch linked above;
an ARM64 runtime image built from that branch.

The checkpoint is not supported as a one-node TP=1 model. The prepared target format itself is pinned to TP=2.

Download

Run on both nodes, choosing a local SSD path with sufficient free space:

bash
hf download anemll/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1 \
  --local-dir /srv/dspark/models/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1

Runtime setup

Clone the matching runtime branch on both nodes:

bash
git clone --branch dspark-nvfp4-a4w4 \
  https://github.com/Anemll/dspark-vllm-gx10.git
cd dspark-vllm-gx10

Use an image built from this branch. The legacy 0.1.1 image does not contain the prepared bulk direct reader unless it has been rebuilt from the branch:

bash
FINAL_IMAGE=dspark-vllm-gx10:nvfp4-a4w4-v1 ./scripts/build-image.sh

Set the following values in both config/head.env and config/worker.env, in addition to their normal rank, fabric, NCCL, cache, and rendezvous fields:

bash
DSPARK_VLLM_IMAGE=dspark-vllm-gx10:nvfp4-a4w4-v1
DSPARK_MODEL_HOST=/srv/dspark/models/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1
DSPARK_DRAFT_MODEL_HOST=/srv/dspark/models/DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1/dspark
SERVED_MODEL_NAME=deepseek-v4-flash-nvfp4-dspark

DSPARK_WEIGHT_LOAD_FORMAT=auto
DSPARK_MOE_BACKEND=auto
DSPARK_SPECULATION_MODE=dspark
MTP_NUM_TOKENS=5
VLLM_DSPARK_CONFIDENCE_SCHEDULER=off
VLLM_DSPARK_CONFIDENCE_THRESHOLD=0.0

VLLM_DSV4_NVFP4_CUTLASS_PREPARED_LOAD=1
VLLM_DSV4_NVFP4_CUTLASS_PREPARED_MANIFEST_SHA256=REPLACE_WITH_MANIFEST_SHA256
VLLM_DSV4_NVFP4_CUTLASS_PREPARED_DIRECT_READ=1

Start rank 1 first, then rank 0:

bash
# WORKER / rank 1
./scripts/start-node.sh config/worker.env

# HEAD / rank 0, after rank 1 enters the rendezvous
./scripts/start-node.sh config/head.env

When passwordless worker SSH is configured in the head environment, the head can launch both ranks:

bash
./scripts/start-cluster.sh config/head.env config/worker.env

Startup verification

Follow the container log on either rank:

bash
cid="$(sudo docker ps -q --filter label=com.docker.compose.service=vllm-dspark)"
sudo docker logs -f "$cid"

A valid prepared bulk load reports:

NVFP4_PREPARED event=enabled ... io_mode=preadv;
43 event=layer_load records;
event=complete layers=43 reads=344 copies=344 ... io_mode=preadv;
43 post-load records with transforms=0 and backend FLASHINFER_CUTLASS.

After startup:

bash
curl -fsS http://HEAD_HOST:8888/health
curl -fsS http://HEAD_HOST:8888/v1/models

Validated load performance

The bulk direct reader was validated end to end on two GB10 nodes:

Table with columns: Measurement, Previous prepared path, Bulk direct reader, Improvement
Measurement	Previous prepared path	Bulk direct reader	Improvement
Slower-rank target load	558.19 s	65.23 s	8.56x
Complete head model load	595.90 s	108.54 s	5.49x
Head start to API server	n/a	231.64 s	n/a

Prefill comparison

These are warmed, server-side prefill rates on the same two-node TP=2 cluster. Prefill is target-only and does not benefit from speculative decoding.

Table with columns: Input tokens, FP8/B12X production, NVFP4 W4A4, Gain, Production TTFT, W4A4 TTFT
Input tokens	FP8/B12X production	NVFP4 W4A4	Gain	Production TTFT	W4A4 TTFT
1,024	2,033.0 tok/s	2,242.5 tok/s	+10.3%	0.512 s	0.463 s
2,048	2,252.0 tok/s	2,473.2 tok/s	+9.8%	0.920 s	0.835 s
4,096	2,320.7 tok/s

The reports are same-size aggregates, not paired identical-prompt trials. See the comparison and raw-artifact links.

Decode and agentic comparisons

All rows use MTP=5, confidence scheduling off, probabilistic draft sampling, no draft/verify overlap optimization, temperature zero, and 512 requested tokens per stream.

Table with columns: Workload, C, FP8/B12X + DSpark, W4A4 + DSpark, Comparison, W4A4 mean accepted length
Workload	C	FP8/B12X + DSpark	W4A4 + DSpark	Comparison	W4A4 mean accepted length
Canonical chat/tool	4	105.48 tok/s	96.02 tok/s	-9.0%	3.127
`tool_agentic`	8	not archived cleanly	360.68 tok/s	no matched control	5.270

Limitations

This artifact is tied to the custom prepared-loader contract and TP=2.
It is not supported by stock Transformers or stock vLLM.
The target and speculative draft have separate physical layouts and must be mounted at the two paths shown above.
Decode performance depends strongly on prompt content and draft acceptance; report prompt, concurrency, acceptance, output length, and timing convention with every result.
The agentic no-collapse diagnostic is not a comprehensive quality evaluation.
The target originates from NVIDIA's NVFP4 checkpoint and the draft from the native DeepSeek V4 Flash DSpark lineage; review the upstream licenses and terms as well as the files included here.

Reproducibility

Runtime source: Anemll/dspark-vllm-gx10
Prepared format manifest: dspark-nvfp4-tp2-repack.json
Bundle manifest: bundle-manifest.json
Benchmark summary: w4a4-dspark-serving-comparison.md

DeepSeek-V4-Flash-NVFP4-TP2-W4A4-v1

README

Bundle layout

Requirements

Download

Runtime setup

Startup verification

Validated load performance

Prefill comparison

Decode and agentic comparisons

Limitations

Reproducibility

Explore FriendliAI today

README

Bundle layout

Requirements

Download

Runtime setup

Startup verification

Validated load performance

Prefill comparison

Decode and agentic comparisons

Limitations

Reproducibility