What the 64k MTP patch does
- Restricts the draft head from 248,320 vocabulary rows to 64,026 kept rows.
- Pads the draft head to 64,064 rows for the quantized GEMM kernel.
- Gathers the draft head from the target
lm_head and quantizes it at runtime
to Marlin NVFP4 W4A16.
- Quantizes the five MTP linear projections at runtime to Marlin NVFP4.
- Leaves the target model and the verification
lm_head untouched.
The result is a much smaller draft step, allowing larger speculative token
counts. The verified starting point is num_speculative_tokens: 7 on B200 and
8 on DGX Spark.
Important: 64k MTP requires the included plugin. Running a plain
vllm serve command with only --speculative-config uses native MTP, not the
64k/Marlin draft path.
Requirements
- vLLM 0.31.0 (the plugin uses vLLM internals and may break on other versions)
- Blackwell GPU: B200 (
sm_100) verified; DGX Spark / GB10 (sm_120/121) is the
recipe's target platform
- Python 3.10+
- Enough memory for the 20.6 GB checkpoint plus KV cache
- PyTorch and CUDA matching the vLLM 0.31.0 build
This MTP patch is for vLLM. It is not an SGLang or TensorRT-LLM plugin.
1. Download the model
export MODEL_DIR="$HOME/models/Qwen3.8-27B-NVFP4-DGX-SPARK-V2" HF_HUB_ENABLE_HF_TRANSFER=1 \hf download autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK-V2 \ --local-dir "$MODEL_DIR" cd "$MODEL_DIR"
2. Install vLLM and the MTP plugin
Use a clean virtual environment:
python3 -m venv .venvsource .venv/bin/activatepython -m pip install -U pip pip install "vllm==0.31.0"pip install -e ./mtp64k/plugin
Verify that the plugin is installed:
python -c "import spark_mtp_draft; print(spark_mtp_draft.__file__)"
3. Set the 64k MTP environment
export SPARK_DRAFT_HOME="$PWD/mtp64k/scripts"export SPARK_DRAFT_VOCAB="$PWD/mtp64k/artifacts/draft_vocab_64k.pt"export SPARK_DRAFT_HEAD_Q=marlinexport SPARK_DRAFT_MTP_Q=marlin
SPARK_DRAFT_VOCAB is also the activation switch. If it is empty, the plugin
stays inactive. SPARK_DRAFT_HOME can be omitted when the plugin was installed
with pip install -e ./mtp64k/plugin from this repository layout, but setting
it explicitly is recommended.
4. Serve with vLLM
B200
vllm serve "$MODEL_DIR" \ --served-model-name qwen3.8-27b-nvfp4 \ --speculative-config '{"method":"mtp","num_speculative_tokens":7,"use_local_argmax_reduction":true}' \ --gpu-memory-utilization 0.60 \ --max-model-len 32768 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3
DGX Spark / GB10
Use the same command but start with num_speculative_tokens: 8:
vllm serve "$MODEL_DIR" \ --served-model-name qwen3.8-27b-nvfp4 \ --speculative-config '{"method":"mtp","num_speculative_tokens":8,"use_local_argmax_reduction":true}' \ --gpu-memory-utilization 0.60 \ --max-model-len 32768 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3
0.60 GPU memory utilization is the safe starting point on unified-memory
Spark systems. On a discrete B200, raise it only after confirming that the
model and KV cache fit. Increase --max-model-len only if the deployment has
enough KV-cache memory.
Confirm that 64k MTP is active
The server log must contain a line like:
[real-draft] head rows=64064 head_q=marlin mtp_q=marlin layers=5
If that line is absent, the plugin did not activate. The model may still start
with native MTP, but the 64k draft vocabulary and Marlin draft head are not in
use.
5. Send an OpenAI-compatible request
curl http://127.0.0.1:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "qwen3.8-27b-nvfp4", "messages": [{"role": "user", "content": "Write a Python LRU cache."}], "temperature": 0, "max_tokens": 512, "chat_template_kwargs": {"enable_thinking": false} }'
Thinking mode is enabled by the chat template by default. Disable it per request
with chat_template_kwargs.enable_thinking: false.
Keep client max_tokens below the served --max-model-len; the prompt and
completion must fit in the same context.
Offline Python usage
For the in-process vLLM API, apply the patch before creating LLM, and disable
vLLM V1 multiprocessing so the patch is present in the engine process:
import osimport sys os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"sys.path.insert(0, "./mtp64k/scripts") import real_draftreal_draft.install( "./mtp64k/artifacts/draft_vocab_64k.pt", "marlin", "marlin",) from vllm import LLM, SamplingParams llm = LLM( model=".", speculative_config={ "method": "mtp", "num_speculative_tokens": 7, "use_local_argmax_reduction": True, }, gpu_memory_utilization=0.60, max_model_len=32768, kv_cache_dtype="fp8",) out = llm.chat( [{"role": "user", "content": "Reply with exactly: OK"}], SamplingParams(temperature=0, max_tokens=16),)print(out[0].outputs[0].text)
Run this from the downloaded repository root.
Tuning notes
- B200: start at
num_speculative_tokens: 7.
- DGX Spark / GB10: start at
num_speculative_tokens: 8.
- Sampling temperature above ~0.7: reduce to roughly
4–6 speculative
tokens and re-benchmark; higher entropy lowers draft acceptance.
- Quantization choices:
marlin is the verified default. The scripts also
expose cutlass, fp8, and bf16; use them only after testing on your exact
vLLM build.
- Context length:
32768 is the conservative starting point. Raising it
increases KV-cache memory; --kv-cache-dtype fp8 is recommended.
- DGX Spark OOM: keep
--gpu-memory-utilization 0.60, lower
--max-model-len, or reduce concurrency.
Repository layout
.├── config.json / tokenizer* / chat_template.jinja├── model-00001-of-00002.safetensors├── model-00002-of-00002.safetensors├── model-mtp-fc.safetensors├── model.safetensors.index.json├── mtp64k/│ ├── artifacts/│ │ └── draft_vocab_64k.pt│ ├── scripts/│ │ ├── real_draft.py│ │ ├── nvfp4_ops.py│ │ └── spark_mtp_draft.py│ ├── plugin/│ │ ├── pyproject.toml│ │ └── spark_mtp_draft.py│ └── manifest.json├── LICENSE└── README.md
mtp64k/manifest.json contains the packaged file hashes and recommended
runtime settings.
Troubleshooting
The server starts but there is no [real-draft] line
Check all of the following:
python -c "import spark_mtp_draft; print(spark_mtp_draft.__file__)"test -f "$SPARK_DRAFT_VOCAB" && echo "vocab exists"
Then restart the server with the environment from section 3. Environment
variables must be inherited by the vLLM engine process.
MTP runs but there is no expected speedup
- Confirm
head rows=64064 appears in the log.
- Confirm
use_local_argmax_reduction: true is present in the speculative
config.
- Try a lower temperature and re-benchmark.
- Compare with a single stream; batching changes the relative benefit.
CUDA driver error 803
On the B200 host used for validation, the host driver needed the system CUDA
driver library to be preloaded:
export LD_PRELOAD=/lib/x86_64-linux-gnu/libcuda.so.1
This is host-specific. Do not apply it blindly if your driver stack does not
need it.
Validation
The packaged repository was served directly from its download directory with
vLLM 0.31.0 on a B200. The MTP patch loaded successfully:
[real-draft] head rows=64064 head_q=marlin mtp_q=marlin layers=5
An OpenAI-compatible smoke request returned OK with finish_reason=stop.
License and sources
Apache-2.0. See LICENSE.