autotrust

DeepSeek-V4-Flash-DSpark-4E

Deploy Dedicated

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Get help setting up a custom Dedicated Endpoints.

Talk with our engineer to get a quote for reserved GPU instances with discounts.

Why top_k=4 Instead of 6?

The original num_experts_per_tok=6 is not a power of 2. In practice, this means:

GPU tensor core utilization is suboptimal for certain MoE dispatch shapes
Memory alignment and warp scheduling are less efficient compared to power-of-2 expert counts
The routing decision per token requires computing softmax over 6 logits instead of 4, introducing unnecessary overhead

Setting top_k to 4 (a power of 2) gives the GPU's SIMT architecture a natural alignment for expert dispatch and attention masking, while activating 33% fewer parameters per token with no accuracy degradation — and in many reasoning-heavy tasks, a measurable accuracy improvement.

Key Changes from the Original

Table with columns: Configuration, Original (top_k=6), This Model (top_k=4)
Configuration	Original (top_k=6)	This Model (top_k=4)
`num_experts_per_tok`	6	4
Activated params per token	~13B	~11B
Total params	284B	284B
Routing method	noaux_tc	noaux_tc
All other weights	identical	identical

Performance Analysis

Activated Parameters

The DeepSeek-V4-Flash architecture has 284B total parameters, of which ~13B are activated per token at top_k=6. By switching to top_k=4, activated parameters drop to ~11B — a 15% reduction — because each token routes to only 4 out of 256 routed experts instead of 6, while shared experts and attention parameters remain unchanged.

Inference Speed

Table with columns: Benchmark, top_k=4, top_k=6, Speedup
Benchmark	top_k=4	top_k=6	Speedup
MMLU-Pro (12,032 questions)	73.0 s	89.0 s	+18.0%
HumanEval (164 problems)	55.8 s	64.1 s	+14.9%
Per-token generation	~0.34 s	~0.39 s	+12.8%

The speedup comes from: (a) fewer expert feed-forward computations, (b) better GPU warp utilization with power-of-2 routing, and (c) reduced softmax and gather/scatter overhead in the router.

Accuracy Impact

Table with columns: Benchmark, top_k=4, top_k=6, Δ
Benchmark	top_k=4	top_k=6	Δ
MMLU-Pro	42.30%	39.27%	+3.03%
HumanEval	94.51%	95.73%	−1.22%

On MMLU-Pro, top_k=4 significantly outperforms top_k=6 (+3.03%), suggesting that routing to fewer, more confidently selected experts improves factual retrieval. On HumanEval, top_k=6 retains a narrow edge (+1.22%), indicating code generation benefits from broader expert diversity.

Overall: top_k=4 delivers superior throughput with competitive or better accuracy on knowledge-heavy tasks, making it the recommended default.

Evaluation Results

MMLU-Pro (chat mode, max_tokens=20)

Table with columns: Configuration, Accuracy, Generation Time
Configuration	Accuracy	Generation Time
DSpark top_k=4	42.30%	73.0 s
DSpark top_k=6	39.27%	89.0 s
4E top_k=4 (reference)	41.75%	122.0 s

HumanEval (thinking mode, max_tokens=4096)

Table with columns: Configuration, Pass@1, Generation Time
Configuration	Pass@1	Generation Time
DSpark top_k=4	94.51%	55.78 s
DSpark top_k=6	95.73%	64.07 s
4E top_k=4 (reference)	95.73%	56.83 s

Key Findings

MMLU-Pro: top_k=4 outperforms top_k=6 by +3.03% (42.30% vs 39.27%) — the additional expert diversity in top_k=6 hurts multiple-choice knowledge retrieval.
HumanEval: top_k=6 slightly outperforms top_k=4 (+1.22%), showing code generation benefits from more experts.
Speed: top_k=4 generates ~13–15% faster across both benchmarks, with 33% fewer activated parameters per token.

vLLM Compatibility: Required Code Modification

⚠️ Important: The checkpoint's tid2eid (hash-based expert routing table) weights have shape [vocab_size, 6] (trained with num_experts_per_tok=6). To run with num_experts_per_tok=4, vLLM's model loading code must be patched.

The Fix (in vLLM source)

In /home/user/.local/lib/python3.11/site-packages/vllm/models/deepseek_v4/nvidia/model.py, locate the load_weights method and add a shape-mismatch handler for tid2eid weights:

python
# Inside the else block (~line 1135), before weight_loader():
if "tid2eid" in name and loaded_weight.shape != param.shape:
    loaded_weight = loaded_weight[:, :param.shape[1]].contiguous()

This slices the checkpoint's 6-column tid2eid tensor to 4 columns, matching the config's num_experts_per_tok=4. Without this patch, vLLM raises:

markdown
AssertionError: Attempted to load weight (torch.Size([129280, 6])) into parameter (torch.Size([129280, 4]))

Configuration

Set num_experts_per_tok=4 in config.json:

json
"num_experts_per_tok": 4

Model Details

Table with columns: Property, Value
Property	Value
Architecture	DeepSeekV4ForCausalLM (MoE)
Total Parameters	284B
Activated Parameters	~11B (top_k=4)
Expert Precision	FP4 (MXFP4)
Other Parameters	FP8
Context Length	1,048,576 tokens
DSpark Module	Markov speculative decoding (layers 40–42)
Recommended top_k	4

Usage

Use with vLLM (with the tid2eid patch above):

python
from vllm import LLM, SamplingParams

llm = LLM(
    model="autotrust/DeepSeek-V4-Flash-DSpark-4E",
    trust_remote_code=True,
    kv_cache_dtype="fp8",
    max_model_len=32768,
)

sampling_params = SamplingParams(
    temperature=0.0,
    max_tokens=20,
)

outputs = llm.generate(["Your prompt here"], sampling_params)

For proper message encoding, use the encoding module included in this repository.

License

MIT

Model provider

autotrust

Model tree

Base

this model

Modalities

Input

Text

Output

Text

Pricing

Dedicated Endpoints

View details

Supported Functionality

Model APIs

Dedicated Endpoints

Container

More information

Model card

Explore FriendliAI today

Get started Talk to an engineer

Why top_k=4 Instead of 6?

The original num_experts_per_tok=6 is not a power of 2. In practice, this means:

GPU tensor core utilization is suboptimal for certain MoE dispatch shapes
Memory alignment and warp scheduling are less efficient compared to power-of-2 expert counts
The routing decision per token requires computing softmax over 6 logits instead of 4, introducing unnecessary overhead

Key Changes from the Original

Table with columns: Configuration, Original (top_k=6), This Model (top_k=4)
Configuration	Original (top_k=6)	This Model (top_k=4)
`num_experts_per_tok`	6	4
Activated params per token	~13B	~11B
Total params	284B	284B
Routing method	noaux_tc	noaux_tc
All other weights	identical	identical

Performance Analysis

Activated Parameters

Inference Speed

Table with columns: Benchmark, top_k=4, top_k=6, Speedup
Benchmark	top_k=4	top_k=6	Speedup
MMLU-Pro (12,032 questions)	73.0 s	89.0 s	+18.0%
HumanEval (164 problems)	55.8 s	64.1 s	+14.9%
Per-token generation	~0.34 s	~0.39 s	+12.8%

The speedup comes from: (a) fewer expert feed-forward computations, (b) better GPU warp utilization with power-of-2 routing, and (c) reduced softmax and gather/scatter overhead in the router.

Accuracy Impact

Table with columns: Benchmark, top_k=4, top_k=6, Δ
Benchmark	top_k=4	top_k=6	Δ
MMLU-Pro	42.30%	39.27%	+3.03%
HumanEval	94.51%	95.73%	−1.22%

Overall: top_k=4 delivers superior throughput with competitive or better accuracy on knowledge-heavy tasks, making it the recommended default.

Evaluation Results

MMLU-Pro (chat mode, max_tokens=20)

Table with columns: Configuration, Accuracy, Generation Time
Configuration	Accuracy	Generation Time
DSpark top_k=4	42.30%	73.0 s
DSpark top_k=6	39.27%	89.0 s
4E top_k=4 (reference)	41.75%	122.0 s

HumanEval (thinking mode, max_tokens=4096)

Table with columns: Configuration, Pass@1, Generation Time
Configuration	Pass@1	Generation Time
DSpark top_k=4	94.51%	55.78 s
DSpark top_k=6	95.73%	64.07 s
4E top_k=4 (reference)	95.73%	56.83 s

Key Findings

MMLU-Pro: top_k=4 outperforms top_k=6 by +3.03% (42.30% vs 39.27%) — the additional expert diversity in top_k=6 hurts multiple-choice knowledge retrieval.
HumanEval: top_k=6 slightly outperforms top_k=4 (+1.22%), showing code generation benefits from more experts.
Speed: top_k=4 generates ~13–15% faster across both benchmarks, with 33% fewer activated parameters per token.

vLLM Compatibility: Required Code Modification

The Fix (in vLLM source)

In /home/user/.local/lib/python3.11/site-packages/vllm/models/deepseek_v4/nvidia/model.py, locate the load_weights method and add a shape-mismatch handler for tid2eid weights:

python
# Inside the else block (~line 1135), before weight_loader():
if "tid2eid" in name and loaded_weight.shape != param.shape:
    loaded_weight = loaded_weight[:, :param.shape[1]].contiguous()

This slices the checkpoint's 6-column tid2eid tensor to 4 columns, matching the config's num_experts_per_tok=4. Without this patch, vLLM raises:

markdown
AssertionError: Attempted to load weight (torch.Size([129280, 6])) into parameter (torch.Size([129280, 4]))

Configuration

Set num_experts_per_tok=4 in config.json:

json
"num_experts_per_tok": 4

Model Details

Table with columns: Property, Value
Property	Value
Architecture	DeepSeekV4ForCausalLM (MoE)
Total Parameters	284B
Activated Parameters	~11B (top_k=4)
Expert Precision	FP4 (MXFP4)
Other Parameters	FP8
Context Length	1,048,576 tokens
DSpark Module	Markov speculative decoding (layers 40–42)
Recommended top_k	4

Usage

Use with vLLM (with the tid2eid patch above):

python
from vllm import LLM, SamplingParams

llm = LLM(
    model="autotrust/DeepSeek-V4-Flash-DSpark-4E",
    trust_remote_code=True,
    kv_cache_dtype="fp8",
    max_model_len=32768,
)

sampling_params = SamplingParams(
    temperature=0.0,
    max_tokens=20,
)

outputs = llm.generate(["Your prompt here"], sampling_params)

For proper message encoding, use the encoding module included in this repository.

License

MIT

DeepSeek-V4-Flash-DSpark-4E

Get help setting up a custom Dedicated Endpoints.

README

Why top_k=4 Instead of 6?

Key Changes from the Original

Performance Analysis

Activated Parameters

Inference Speed

Accuracy Impact

Evaluation Results

MMLU-Pro (chat mode, max_tokens=20)

HumanEval (thinking mode, max_tokens=4096)

Key Findings

vLLM Compatibility: Required Code Modification

The Fix (in vLLM source)

Configuration

Model Details

Usage

License

Explore FriendliAI today

README

Why top_k=4 Instead of 6?

Key Changes from the Original

Performance Analysis

Activated Parameters

Inference Speed

Accuracy Impact

Evaluation Results

MMLU-Pro (chat mode, max_tokens=20)

HumanEval (thinking mode, max_tokens=4096)

Key Findings

vLLM Compatibility: Required Code Modification

The Fix (in vLLM source)

Configuration

Model Details

Usage

License