Why top-k = 4 is reasonable
DeepSeek-V4-Pro routes each token to 6 of 384 fine-grained experts (plus 1 shared expert),
scaled by routed_scaling_factor = 2.5 with noaux_tc biased top-k selection and
sqrtsoftplus scoring. Two properties make the top-6 → top-4 reduction low-risk:
- Router mass is front-loaded. With normalized top-k probabilities (
norm_topk_prob: true),
the highest-scoring experts dominate the mixture; the 5th and 6th experts contribute the
smallest, most redundant residual. Dropping them removes the least informative routing mass.
- Fine-grained MoE is over-provisioned for many tokens. With 384 experts and a shared
expert always active, most tokens are well covered by their top few specialists; the tail
experts largely refine rather than redirect.
The result is a compute/quality knob that is nearly free at k=4 and only degrades sharply
below it — which is exactly what the measurements show.
Evaluation results
Independently re-evaluated with vLLM 0.24.0 (offline LLM), tensor-parallel across 8×B300,
kv_cache_dtype=fp8, chat / non-think mode, greedy decoding (temperature=0), Pass@1,
max_tokens=8192. Code is executed against the official unit tests (HumanEval = 164 problems,
MBPP = full test split, 500 problems). The only variable across rows is num_experts_per_tok.
Table with columns: num_experts_per_tok, HumanEval Pass@1, MBPP Pass@1, Expert matmuls / tokennum_experts_per_tok | HumanEval Pass@1 | MBPP Pass@1 | Expert matmuls / token |
|---|
| 6 (original) | 96.34 % (158/164) | 83.8 % (419/500) | 1.00× |
| 4 (this model) | 96.34 % (158/164) | 82.8 % (414/500) | 0.67× |
| 2 (for reference) | 87.80 % (144/164) | 75.6 % (378/500) | 0.33× |
Takeaways
- 6 → 4 is nearly lossless: HumanEval is identical (158/164) and MBPP drops only 1.0 pt,
while expert computation and expert-weight memory traffic fall by one third.
- 4 → 2 is not free: HumanEval −8.5 pt and MBPP −8.2 pt, and generations become longer / less
confident. top-k=4 is the sweet spot; going below it is not recommended.
Numbers are self-hosted Pass@1 in non-think mode and are not directly comparable to the official
Think-High/Think-Max scores on the upstream card.
Efficiency
The routed-expert FFN is the dominant per-token cost in this architecture. Since that work scales
linearly with num_experts_per_tok, k=6 → k=4 yields:
- ~33 % fewer routed-expert matmuls and ~33 % less expert-weight memory bandwidth per token
(the decisive factor for MoE decode throughput / GGUF CPU inference),
- attention, shared expert, embeddings and norms are unchanged,
- lower activated-parameter count per token (shared + attention + 4 routed experts instead of 6).
Required vLLM patch
Stock vLLM validates that every loaded tensor matches the model's parameter shape. Because this
checkpoint keeps the original 6-column tid2eid routing table while the config now declares
4 experts-per-token, loading fails with:
AssertionError: Attempted to load weight (torch.Size([129280, 6]))
into parameter (torch.Size([129280, 4]))
Fix (one line). In vLLM's DeepSeek-V4 loader — for the build used here
…/site-packages/vllm/models/deepseek_v4/nvidia/model.py (path may vary by version; search for
def load_weights) — inside the generic else branch, before the weight_loader(param, loaded_weight)
call, slice the routing table to the configured expert count:
else:
if is_pp_missing_parameter(name, self):
continue
param = params_dict[name]
if "tid2eid" in name and loaded_weight.shape != param.shape:
loaded_weight = loaded_weight[:, : param.shape[1]].contiguous()
weight_loader = getattr(
param, "weight_loader", default_weight_loader
)
weight_loader(param, loaded_weight)
loaded_params.add(name)
continue
This keeps the top-4 hash-routing entries per token and discards the unused 5th/6th columns,
matching num_experts_per_tok = 4 in config.json.
Auto-apply (copy-paste). Idempotent script that locates your installed vLLM and inserts the
patch in the right place:
python - <<'PY'
import pathlib, vllm
p = pathlib.Path(vllm.__file__).parent / "models/deepseek_v4/nvidia/model.py"
src = p.read_text()
if "tid2eid" in src:
print("already patched:", p); raise SystemExit
needle = (" param = params_dict[name]\n"
" weight_loader = getattr(\n")
patch = (" param = params_dict[name]\n"
" if \"tid2eid\" in name and loaded_weight.shape != param.shape:\n"
" loaded_weight = loaded_weight[:, : param.shape[1]].contiguous()\n"
" weight_loader = getattr(\n")
assert needle in src, "loader block not found -- check your vLLM version / file path"
p.write_text(src.replace(needle, patch, 1))
print("patched:", p)
PY
The exact file path can differ between vLLM builds (e.g.
vllm/model_executor/models/deepseek_v4.py in some releases). The script resolves it from the
installed vllm package; if the loader block isn't found, open the file, find def load_weights,
and add the two tid2eid lines before the generic weight_loader(param, loaded_weight) call.
Running it
from vllm import LLM, SamplingParams
llm = LLM(
model="autotrust/DeepSeek-V4-Pro-4Expert",
tokenizer="autotrust/DeepSeek-V4-Pro-4Expert",
trust_remote_code=True,
tensor_parallel_size=8,
dtype="auto",
kv_cache_dtype="fp8",
max_model_len=32768,
gpu_memory_utilization=0.90,
)
out = llm.generate(["Write a Python function that returns the n-th Fibonacci number."],
SamplingParams(temperature=0.0, max_tokens=512,
stop=["<｜end▁of▁sentence｜>"]))
print(out[0].outputs[0].text)
Notes:
config.json already ships with "num_experts_per_tok": 4.- Prompts use the DeepSeek-V4 chat format from the bundled
encoding/
module (encode_messages(...)), not a Jinja template.
kv_cache_dtype="fp8" is required by the model's fp8_ds_mla MLA kv-cache layout.- On Blackwell (B300 / sm_100/103) the fp8 path JIT-compiles DeepGEMM/flashinfer kernels on first
run; ensure
CUDA_HOME points at a CUDA-13 toolkit.
Architecture
Table with columns: Property, Value| Property | Value |
|---|
| Base model | deepseek-ai/DeepSeek-V4-Pro (weights unchanged) |
| Architecture | DeepSeek-V4 (MoE + MLA, hybrid CSA/HCA attention, mHC) |
| Layers | 61 |
| Hidden size | 7168 |
| Attention heads | 128 (MLA, head_dim=512, qk_rope_head_dim=64) |
| Routed experts | 384 |
| Active experts / token | (was 6) |
Q4_K GGUF variant
A CPU/Metal-friendly Q4_K GGUF of this top-k=4 configuration was produced with the
ds4 4expert toolchain (see gguf-tools/). The GGUF
declares expert_used_count = 4, and both the template generator and the ds4 shape-selector
were extended to accept the Pro shape at top-k=4 (the upstream branch only enabled top-k=4 for
the Flash shape). The tid2eid table is column-sliced 6 → 4 during quantization, mirroring the
vLLM patch above.
Quantization policy: routed experts Q4_K; attention projections / shared expert / output head
Q8_0; embedding and dense/HC/indexer tensors F16; norms/scales F32.
Provenance & honesty
- Not a retrained model. These are the original DeepSeek-V4-Pro weights. The "4Expert" name
refers solely to the inference-time routing width (
num_experts_per_tok = 4).
- Evaluation above is self-hosted, non-think, greedy Pass@1 on HumanEval + MBPP and reflects the
6→4 routing change only; it is not a reproduction of the official benchmark suite or modes.
- Base weights and this repository are distributed under the original MIT License.
Citation
@misc{deepseekai2026deepseekv4,
title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
author={DeepSeek-AI},
year={2026}
}