1. 📖 Model Introduction
This is Qwen3.6-35B-A3B (a mixture-of-experts vision-language model) compressed with Collaborative Deep Compression
(CDC): the compression stages are designed to work together, in a fixed order, and each is judged on the modalities
the model serves, not on text alone.
Two stages were applied:
- Expert pruning (pillar A): each MoE layer keeps the 240 of its 256 routed experts that matter most for real
text, image and video traffic. The shared expert is untouched.
- 4-bit experts (pillar C): the remaining routed experts are quantized to NVFP4 (4-bit weights and activations).
Attention, the shared expert, the router and the vision tower stay in bf16.
2. 📋 Model Summary
Table | |
|---|
| Base model | Qwen3.6-35B-A3B (Qwen3_5MoeForConditionalGeneration), image + video + text input |
| Architecture | MoE, 40 decoder layers (hybrid linear + full attention), top-8 routing plus a shared expert, expert intermediate size 512 |
| Experts per layer | 240 routed (from 256) + 1 shared |
| Precision | routed experts NVFP4 W4A4 (NVIDIA ModelOpt); attention, shared expert, router, vision tower bf16 |
| Size on disk | 22 GB (bf16 base: 67 GB) |
| Weights in GPU memory | 21.4 GiB (bf16 base: 65.5 GiB) |
| Serving | vLLM ≥ 0.30.1 (native NVFP4 MoE kernel); Blackwell GPU for W4A4 (measured on GB200) |
3. 🔬 Deep Compression Method
Table with columns: Pillar, Stage, Applied here, What it did| Pillar | Stage | Applied here | What it did |
|---|
| A | Pruning-aware compression | ✅ | Kept 240 / 256 routed experts per layer |
| B | Clustering-aware compression | ❌ | Not applied (see below) |
| C | Quantization-aware compression | ✅ | Routed experts → NVFP4 W4A4 |
| D | Serving | ✅ | vLLM native NVFP4 MoE path |
Pillar A: multimodal expert pruning. Each routed expert is scored by how much it contributes when it is used: its
router weight times the size of its output, averaged over the tokens routed to it. The scores come from 256 text
conversations plus 256 image, multi-image and video conversations. The 16 lowest-scoring experts per layer are removed,
with no retraining; the kept experts carry 95% of the routing weight. The same recipe was first developed on
Gemma-4-26B, and it transfers: keeping 240 alone is within ±0.4 points of the original on 7 of 8 benchmarks.
Pillar C: experts-only NVFP4. The routed experts hold almost all of the parameters and tolerate 4-bit weights and
activations: experts-only NVFP4 stays within ±0.7 points of bf16 on the vision benchmarks. Attention and the shared
expert stay in bf16.
Why not pillar B (weight clustering) here. Clustered (codebook) weights need a lookup-table matrix kernel that
vLLM does not ship, so a clustered model could not be served at speed.
4. 📊 Evaluation Results
Accuracy (%) on the same serving stack. This model's row is the original score plus the mean paired change over 3 pruning-calibration seeds.
This model keeps 99.8% of the original model's performance on average (at least 98.9% on every benchmark) at about one third of the weight memory.
Table with columns: MMLU, AI2D, ChartQA, MMBench, MMMU, MMStar, OCRBench, MVBench (video) | MMLU | AI2D | ChartQA | MMBench | MMMU | MMStar | OCRBench | MVBench (video) |
|---|
| Original bf16 model | 82.15 | 88.12 | 88.52 | 91.43 | 66.67 | 67.40 | 90.90 | 74.54 |
| This model | 81.66 | 87.92 | 88.17 | 91.18 | 65.96 | 67.98 | 90.63 | 74.85 |
| Performance retained | 99.4% | 99.8% | 99.6% | 99.7% | 98.9% | 100.9% | 99.7% | 100.4% |
Retained = this model's score ÷ the original's score. Values above 100% are within evaluation noise: treat them as "no measurable loss", not as an improvement.
Detailed change vs. the original, on the same serving stack (percentage points; paired item-by-item; thinking
disabled; mean ± s.d. over 3 pruning-calibration seeds):
Table with columns: MMLU, AI2D, ChartQA, MMBench, MMMU, MMStar, OCRBench, MVBench (video)| MMLU | AI2D | ChartQA | MMBench | MMMU | MMStar | OCRBench | MVBench (video) |
|---|
| −0.49 ± 0.19 | −0.20 ± 0.18 | −0.35 ± 0.40 | −0.25 ± 0.22 | −0.71 ± 0.77 | +0.58 ± 0.32 | −0.27 ± 0.25 | +0.31 ± 0.27 |
- MMLU costs about half a point. Every vision benchmark is within noise.
- Re-evaluating one checkpoint moves these benchmarks by about 1 point, so differences below that are not
meaningful.
5. ⚡ Efficiency
One GB200 GPU, tensor-parallel 1, vLLM 0.30.1rc1, GPU memory utilisation 0.85. Throughput is the mean of 3 runs,
interleaved with the bf16 base in one session (64 requests, 256 generated tokens each).
Table with columns: bf16 base, This model, Change | bf16 base | This model | Change |
|---|
| Batched text requests (req/s) | 8.97 | 12.12 | +35% |
| Batched image requests (req/s, two image sets) | 19.16 / 12.91 | 26.03 / 15.85 | +36% / +23% |
| Batched video requests (req/s) | 0.79 | 0.79 | ≈ |
| Single-stream decode (tokens/s) | 298 | 313 | +5% |
| Energy efficiency (tokens per joule) | 12.8 | pending | |
| Weights in GPU memory | 65.5 GiB | 21.4 GiB | −67% |
| KV-cache capacity at the same memory budget (32 K context) | 3.69 M tokens | 5.51 M tokens | +49% |
| Max concurrent requests at 32 K context | 113 | 168 | +49% |
| Max concurrent requests at 128 K context | 32.7 | 49.4 | +51% |
- Single-stream speed changes little: only about 3 B parameters are active per token, so the model is already fast.
The gain is in batch throughput and in how many long-context requests fit in memory.
- Video is limited by encoding the frames, which stays in bf16.
6. 🚀 Deployment
pip install "vllm>=0.30.1"vllm serve trannhiem/TranNhiem-DeepCompression-Qwen3.6-35B-A3B --max-model-len 32768
- Hardware: native 4-bit (W4A4) needs a Blackwell GPU (B200 / GB200 / B300, RTX PRO 6000 Blackwell, DGX Spark, RTX 50-series). Measured on GB200 only.
- No MTP head: the pruned checkpoint does not include the base model's multi-token-prediction weights, so
MTP speculative decoding is not available.
7. 💻 Usage
Qwen3.6 reasons by default. Our evaluations disable thinking; enable it if you want reasoning traces.
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")resp = client.chat.completions.create( model="trannhiem/TranNhiem-DeepCompression-Qwen3.6-35B-A3B", messages=[{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/document.png"}}, {"type": "text", "text": "Summarise this document in three bullet points."}, ]}], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, max_tokens=512,)print(resp.choices[0].message.content)
8. ⚠️ Limitations
- All measurements are on one GPU type (GB200). Speed and energy on other hardware will differ.
- Accuracy is reported on 8 benchmarks (knowledge, diagrams, charts, general and expert vision, OCR, video) with
thinking disabled. The model was not re-evaluated for safety beyond the base model.
- Pruning removes 16 routed experts per layer permanently. Rare skills that only those experts served may degrade more
than these benchmarks show.
9. 📜 License
Apache-2.0, inherited from the base model (license).
This is a modified version of Qwen3.6-35B-A3B: 16 routed experts per layer were removed and the remaining routed
experts were quantized to NVFP4.
10. 📝 Citation
@article{cdc2026, title = {Collaborative Deep Compression for Foundation Models}, author = {Tran, Van Nhiem}, year = {2026}, note = {Paper link coming soon}}
Trần Nhiệm · rick.tvn@foxconn.com WhatsApp: +886 934 311751