azampatti

Qwen3.5-122B-A7B-Int4

README

License: apache-2.0

⚠️ Read this first — you need the hybrid vLLM patches

This checkpoint keeps its shared expert in BF16 while everything else is INT4. Stock vLLM sends any module with bits >= 16 to UnquantizedLinearMethod, which drops those 144 projections off the fused quantized path. It still runs — it just runs ~27% slower.

Measured on identical hardware, same model, same vLLM version (0.26.1rc1.dev1123), patches on vs off:

Table with columns: stock vLLM, patched
	stock vLLM	patched
throughput	49.0 tok/s	70.8 tok/s

Get the patches here:

https://github.com/azampatti/vllm-hybrid-int4-fp8-patches

They apply on top of an existing vLLM image and leave the base image untouched.

Speed

🚀 70.8 tok/s on long-form code generation

Full bench.sh sweep, single stream, DGX Spark, patched vLLM, MTP depth 3:

Table with columns: workload, output tokens, time, throughput
workload	output tokens	time	throughput
Q&A	256	4.19 s	61.0 tok/s
Code	488	7.06 s	69.1 tok/s
JSON	1024	15.99 s	64.0 tok/s
Math	64	1.07 s

Caveat on all speed numbers. These are DGX Spark (GB10 / SM121) figures. Everything here was built and tested only on a DGX Spark, on a vLLM image derived from eugr/spark-vllm-docker. No other hardware or base image has been validated. Your numbers will differ.

What was done

1. Top-k=4 expert pruning

The base model routes each token to 8 of its 256 experts. This build routes to 4, halving routed-expert compute and weight traffic per token. Nothing is deleted — all 256 experts are still present and the router still chooses among them; it simply selects half as many per token.

Done naively this costs real accuracy: the model was trained expecting eight opinions per token and now gets four.

2. Shared-expert healing

Every layer also has a shared expert that is always active, regardless of routing. That module is the natural place to absorb what the four dropped experts were contributing, so it was retrained in BF16 by distillation against the unpruned base model while the rest of the network stayed frozen.

The shared expert is deliberately left unquantized (BF16). Quantizing it was measured and rejected — it is the one module carrying the trained correction, and quantization noise on it is comparable in size to the correction itself. That choice is exactly why the hybrid patches above are required for full speed.

After healing, quality matches the original base model.

3. MTP draft-head retraining

The multi-token-prediction head used for speculative decoding is itself a 256-expert MoE layer, so cutting the target's top-k cut the draft head with it. The head was retrained with KL distillation against the target model and the best checkpoint is shipped here. Speculative decoding is distribution-preserving — it changes speed, never output quality.

Serve with num_speculative_tokens: 3 (depth 3); the launch command below does.

Model details

Table

Total parameters	~122 B
Active per token	~6.4 B (~4.9 B in the 48 MoE layers + ~1.5 B embeddings & LM head)
Layers	48
Routed experts	256, top-4 active per token (base: top-8)
Shared expert	1, always active, BF16
Hidden size	3072
Attention	32 heads / 2 KV heads (GQA), head dim 256
Vocabulary	248,320

The A7B in the name follows the base model's A10B convention. Counted directly from config.json, this build activates ~6.4 B parameters per token including embeddings and the LM head, or ~4.9 B across the 48 transformer layers alone.

Launching

Replace <hybrid-image> with the tag you built from the patch repo. The model is pulled from the Hugging Face cache — no local path needed.

bash
docker run --privileged --gpus all -d \
  --name vllm-qwen35-a7b \
  --net=host \
  --ipc=host \
  -v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_MTP_TOP_K=8 \
  <hybrid-image> \
  vllm serve azampatti/Qwen3.5-122B-A7B-Int4 \
    --served-model-name Qwen3.5-122B-A7B-Int4 \
    --port 8000 \
    --host 0.0.0.0 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}' \
    --max-model-len 200K \
    --gpu-memory-utilization 0.75 \
    --load-format fastsafetensors \
    --attention-backend TRITON_ATTN \
    --dtype bfloat16 \
    --kv-cache-dtype fp8 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --max-num-seqs 8 \
    --max-num-batched-tokens 16384 \
    --override-generation-config '{"temperature":0.5,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

Then:

bash
curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.5-122B-A7B-Int4",
       "messages":[{"role":"user","content":"Write a binary search in Rust."}]}'

Notes on the flags

VLLM_MTP_TOP_K=8 is provided by the patches. The MTP draft head is a 256-expert MoE layer in its own right, and this lets it route more widely than the top-4 target. It affects speed only.
The chat template ships with the model (chat_template.jinja) and is loaded automatically. It carries the tool-calling format this build expects — pass --chat-template only if you intend to override it.
--reasoning-parser qwen3 and --tool-call-parser qwen3_coder are required for thinking blocks and tool calls to be parsed into their own response fields rather than appearing as raw text.
--kv-cache-dtype fp8 and --gpu-memory-utilization 0.75 are sized for a 128 GB DGX Spark at 200K context. Raise the utilization if you have headroom.

License

Apache 2.0, inherited from Qwen3.5-122B-A10B.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider