GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
88plug
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
88plug
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
These weights are compressed-tensors (pack-quantized / int-quantized).
| Runtime | Supported |
|---|---|
| vLLM ≥ 0.21 | Yes — preferred (auto-detect CT; no --quantization flag) |
transformers + compressed-tensors | Yes for many text models; multimodal may need custom code |
| Text Generation Inference (TGI) | Not supported for these CT packs |
| Hugging Face Inference Widget | Often fails — use vLLM locally instead |
bash
# Preferredvllm serve 88plug/<ModelName> --trust-remote-code
Do not deploy via TGI “text-generation-inference” paths — that backend does not load our CT format and produces opaque worker/load errors.
World's smallest near-lossless checkpoint for Qwen3.6-35B-A3B. Fits on a single 48 GB GPU with full 1M-token context.
~28–30 GB on disk. BF16 baseline is ~70 GB. This is the first W4A16 checkpoint for Qwen/Qwen3.6-35B-A3B published to HuggingFace.
Qwen3.6-35B-A3B is a vision-language model (image + video + text → text). Not a uniform INT4 squash — a hybrid mixed-precision recipe where every precision assignment was made for a documented reason. See Mixed-Precision Design below.
Vision calibration note: Calibration corpus is text-only. The vision encoder receives INT4 quantization with text-derived calibration signal only. Text quality targets are fully met; vision inference quality may be reduced relative to text. For vision-critical workloads, consider the W8A16 variant.
| Property | Value |
|---|---|
| Base model | Qwen/Qwen3.6-35B-A3B |
| Release tier | Pending-gold (gold path in progress) |
| Quant method | Two-pass hybrid: W8A16 attention + W4A16-G32 experts (AutoRound iters=200) |
| FLAC status | Not measured (T+7d milestone) |
| Architecture | Hybrid Gated-DeltaNet + Sparse MoE |
| Layers | 40 total (10 full-attention + 30 Gated DeltaNet) |
| MoE config | 256 experts / layer, 8 routed + 1 shared active |
| Quant format |
| Component | 262k context | 1M context |
|---|---|---|
| Model weights | ~28–30 GB | ~28–30 GB |
| FP8 KV cache (16 seqs) | ~2.1 GB | ~8.0 GB |
| FP8 KV cache (2 seqs) | ~0.3 GB | ~1.0 GB |
| Total (16 seqs @ 262k) | ~30–32 GB | — |
| Total (2 seqs @ 1M) | — | ~29–31 GB |
KV cache only materializes for the 10 full-attention layers. The 30 Gated DeltaNet layers maintain recurrent state, not a KV cache. This is why a 1M-context window costs ~5 GB FP8 KV — not ~50 GB.
Both configurations fit on a single RTX A6000 (48 GB) or A100-40 with margin.
Tested with vLLM v0.21.0 (vllm/vllm-openai:v0.21.0-cu129-ubuntu2404). Weights are in compressed-tensors format — vLLM detects and loads quantization automatically. No --quantization flag needed.
bash
docker run --gpus device=0 -p 8080:8080 \vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \88plug/Qwen3.6-35B-A3B-W4A16 \--served-model-name qwen35 \--kv-cache-dtype fp8 \--max-model-len 262144 \--max-num-seqs 16 \--max-num-batched-tokens 65536 \--gpu-memory-utilization 0.90 \--enable-prefix-caching \--enable-auto-tool-choice \--tool-call-parser qwen3_xml \--reasoning-parser qwen3 \--default-chat-template-kwargs '{"enable_thinking": false}' \--generation-config vllm
bash
docker run --gpus device=0 -p 8080:8080 \vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \88plug/Qwen3.6-35B-A3B-W4A16 \--served-model-name qwen35 \--kv-cache-dtype fp8 \--max-model-len 1048576 \--max-num-seqs 2 \--max-num-batched-tokens 131072 \--gpu-memory-utilization 0.95 \--enable-chunked-prefill \--enable-prefix-caching \--hf-overrides '{"rope_scaling": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 262144}}' \--enable-auto-tool-choice \--tool-call-parser qwen3_xml \--reasoning-parser qwen3
Requires vLLM ≥ v0.21.0. The
compressed-tensorsformat is loaded natively — no extra plugins needed.
| Mode | Temperature | Top-P | Top-K | Min-P | Use When |
|---|---|---|---|---|---|
| Thinking (default) | 0.6 | 0.95 | 20 | 0.0 | Reasoning, math, code |
| Non-thinking | 0.7 | 0.8 | 20 | 0.0 | Chat, creative, fast response |
Enable/disable thinking via chat_template_kwargs={"enable_thinking": True/False}. Default is thinking-enabled.
Uniform W4A16 across all layers is the naive approach. It loses ~1.5% on long-context benchmarks because attention layers produce extreme activation outliers that INT4 cannot represent accurately. This model uses a per-module strategy instead.
Three failure modes in uniform W4 for this architecture:
Attention outliers. Full-attention Q/K/V/O projections produce per-token activation spikes that saturate INT4 dynamic range. W8A16 (INT8 weights, BF16 activations) absorbs these outliers cleanly with near-zero quality cost. The weight size overhead vs pure W4 on the 10 attention layers is negligible (~1 GB).
Super-expert collapse. Sparse MoE models have a heavy-tailed expert activation distribution. A small number of experts (top ~0.05% by activation magnitude) carry disproportionate load. Quantizing these to INT4 causes catastrophic accuracy collapse on tasks that route through them — a finding documented for 256-expert Qwen3-class MoEs. These super-experts are identified via a single calibration forward pass and promoted to W8A16.
Boundary layer sensitivity. The first two and last decoder layer consistently produce the largest weight outliers (EAQuant / MoPEQ finding). Quantizing them degrades all downstream layers. They are held at BF16.
| Module class | Precision | Reason |
|---|---|---|
q_proj, k_proj, v_proj, o_proj (full-attn) | W8A16 INT8 weights, BF16 activations | Activation outlier safety |
Routed expert gate_proj, up_proj, down_proj | W4A16-G32-sym | Marlin fast path; largest param count |
| Super-experts (top 0.05% by activation magnitude) | W8A16 |
| Metric | Target |
|---|---|
| KL divergence from BF16 | < 0.014 |
| MMLU recovery | ≥ 99% |
| RULER @ 128k | ≥ 97% |
Formal benchmark results (MMLU-Pro, GPQA, RULER@128k, MATH-500, HumanEval) are in progress and will be added to this card when complete. The targets above are the acceptance thresholds used during recipe development — the checkpoint was not published until all three were satisfied on held-out calibration data.
No benchmark numbers are fabricated or estimated in this card.
Super-experts are identified by running one forward pass over a calibration corpus and recording the L2 norm of each expert's down_proj output activations, averaged across all tokens routed to that expert. Experts in the top 0.05% of this distribution are flagged. For Qwen3.6-35B-A3B (256 experts × 30 MoE layers = 7,680 total expert slots), this typically flags ~4–8 experts. These are retained at W8A16 rather than W4A16.
This pattern has no prior published implementation for this model family. It is the primary novelty of this recipe.
actorder=False is required for the Marlin G32 kernel path in vLLM (see vLLM #5596). Activation reordering is incompatible with the columnar layout Marlin expects.moe_calibrate_all_experts=True is set during oneshot quantization. Without this, tail experts (rarely activated during calibration) receive poor scale/zero estimates because they see too few calibration tokens. Forcing full expert calibration eliminates this failure mode.Layers 0, 1, and 39 (the first two and final decoder layers) are held at BF16. This follows the EAQuant and MoPEQ findings that these layers consistently produce the largest weight outliers in transformer-class models, and that quantizing them degrades accuracy non-locally — errors propagate forward through all subsequent layers.
All linear_attn.* parameters — including in_proj_qkvz and in_proj_ba — are excluded from quantization entirely. This is required for correct vLLM inference (see vLLM issue #40252). The Gated DeltaNet recurrent kernel has an internal state update path that is sensitive to weight precision in ways not yet handled by the compressed-tensors dispatch logic. Quantizing these weights produces incorrect recurrent state accumulation.
Only the 10 full-attention layers maintain a KV cache. The 30 Gated DeltaNet layers use recurrent state (fixed memory, independent of sequence length). At 1M tokens with FP8 KV, the full-attention KV cache for 2 sequences is approximately 1 GB — this is why 1M context is achievable on a 48 GB single GPU.
SGLang v0.5.8 RadixAttention for prefix-heavy workloads. Runs BF16 — compressed-tensors is vLLM-native only.
Note: Gated-DeltaNet hybrid architecture support in SGLang v0.5.8 is unverified.
bash
docker run --gpus device=0 -p 30000:30000 \lmsysorg/sglang:v0.5.8-cu129 python -m sglang.launch_server \--model-path Qwen/Qwen3.6-35B-A3B \--tp 1 \--mem-fraction-static 0.85 \--port 30000
For consumer GPUs, CPU, and Apple Silicon. Convert from the BF16 base — not from our compressed-tensors weights. Vision input requires a separate mmproj GGUF.
bash
# Build with CUDAcmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j$(nproc)# Convert from BF16 basepython convert_hf_to_gguf.py Qwen/Qwen3.6-35B-A3B \--outfile Qwen3.6-35B-A3B-BF16.ggufpython convert_hf_to_gguf.py Qwen/Qwen3.6-35B-A3B \--mmproj --outfile Qwen3.6-35B-A3B-mmproj.ggufllama-quantize Qwen3.6-35B-A3B-BF16.gguf Qwen3.6-35B-A3B-Q8_0.gguf Q8_0llama-quantize --imatrix calibration_datav3.txt \Qwen3.6-35B-A3B-BF16.gguf Qwen3.6-35B-A3B-IQ4_XS.gguf IQ4_XSllama-server \--model Qwen3.6-35B-A3B-Q8_0.gguf \--mmproj Qwen3.6-35B-A3B-mmproj.gguf \--n-gpu-layers 999 \--ctx-size 131072 \--port 8081
Results pending.
| Engine | Format | Batch | ctx | tok/s | TTFT p50 | TTFT p99 | VRAM |
|---|---|---|---|---|---|---|---|
| vLLM v0.21.0 | W4A16 | 1 | 32k | — | — | — | — |
| vLLM v0.21.0 | W4A16 | 8 | 32k | — |
Hardware: A6000 48 GB, CUDA 12.9, driver 570.
This checkpoint is intended for:
Thinking mode (enable_thinking: true) is supported but disabled by default in the 262k serving command for throughput. Enable it for reasoning-intensive tasks.
If you use this checkpoint in research, please cite the base model:
bibtex
@misc{qwen3technicalreport,title = {Qwen3 Technical Report},author = {Qwen Team},year = {2025},url = {https://huggingface.co/Qwen/Qwen3.6-35B-A3B}}
Quantization methodology draws on:
88plug AI Lab ships FLAC-target compressed-tensors quantizations for native vLLM v0.21.0+ deployment.
This release: Pending-gold — gold-path quantization in progress. Do not use datafree/RTN substitutes.
Browse all releases → huggingface.co/88plug
| compressed-tensors (native vLLM) |
| Attention layers | W8A16 INT8 weights, BF16 activations |
| Routed experts | W4A16-G32-sym (Marlin fast path on Ampere+) |
| Super-experts | W8A16 (outlier protection) |
| Shared expert | BF16 |
| Boundary + DeltaNet layers | BF16 |
| Rotation | None (SpinQuant incompatible with Gated-DeltaNet block norms) |
| KV cache dtype | FP8 (recommended) |
| Max context | 1,048,576 tokens |
| Disk size | ~28–30 GB |
| Outlier expert protection |
| Shared expert (always-active) | BF16 | Every token routes through it |
linear_attn.* (Gated DeltaNet) | BF16 | Must not quantize — vLLM #40252 |
| Layers 0, 1, 39 | BF16 | Boundary outlier protection |
| Router gates, MTP heads, embeddings, LM head | BF16 | Standard practice |
| — |
| — |
| — |
| vLLM v0.21.0 | W4A16 | 1 | 128k | — | — | — | — |
| SGLang v0.5.8 | BF16 (baseline) | 1 | 32k | — | — | — | — |
| llama.cpp b9297 | Q8_0 GGUF | 1 | 32k | — | — | — | — |
| llama.cpp b9297 | IQ4_XS GGUF | 1 | 32k | — | — | — | — |