fasolack

Qwen3.6-35B-A3B-W4A16

Deploy Dedicated

README

License: apache-2.0

Model overview

Base model: Qwen/Qwen3.6-35B-A3B — a hybrid GatedDeltaNet (linear-attention) + full-attention multimodal MoE: 40 layers (30 linear-attn + 10 full-attn), 256 experts per layer with top-8 routing, ~36B total / ~3B active, plus a vision tower and an MTP head. 262K context.
Quantization: int4 W4A16, group size 128, symmetric, on the language-decoder Linear layers including all 30,720 expert matrices.
Format: compressed-tensors (pack-quantized), auto-detected by vLLM — no --quantization flag.
Quantizer: Intel AutoRound (arXiv:2309.05516), SignRound block reconstruction.

Why this fits two 24 GB cards well

~21 GB of weights, so it serves at TP=2 on a pair of 3090s with room to spare.
~3B active parameters per token. Decode is bandwidth-bound on the active slice, not on all 36B, so it generates substantially faster than a dense model of similar quality.
Long context is cheap here. Only 10 of 40 layers keep a KV cache at all, with 2 KV heads at head_dim 256 → ~20 KiB/token at fp16, ~10 KiB at fp8. The full 262144 context costs about 2.6 GB with --kv-cache-dtype fp8. The remaining 30 layers keep a fixed-size recurrent state regardless of sequence length.

Calibration

Table with columns: Slice, Weight, Source
Slice	Weight	Source
Tool calling	45%	`NousResearch/hermes-function-calling-v1`
Code	35%	`ise-uiuc/Magicoder-OSS-Instruct-75K`
General	20%	`NeelNanda/pile-10k`

1024 samples × 2048 tokens, 100 tuning iterations (reduced from 200 after timing the trial run: ~210s/block at 20 iters extrapolated to ~23h for 200 iterations across all 40 blocks on this hardware, over the ~12h budget; 100 iterations was the RUNBOOK's own documented fallback for this case), seed 42.

The general slice is larger than you might expect for a domain-focused build, and it is doing two jobs. The usual one: calibrating on a 100% narrow distribution costs general ability. The MoE-specific one: generic text spreads router mass across more experts. A purely narrow mix concentrates routing on a subset and leaves the tail of the expert grid effectively untuned.

Tool-calling samples are rendered through the model's own chat template with the tool schemas passed in, and Hermes-style JSON tool calls are rewritten into Qwen's XML style, so the calibration text contains the same tokens the model emits at inference.

Expert coverage

Measured over 128 calibration samples, counting top-8 routing hits per (layer, expert) pair:

Table with columns: Metric, Value
Metric	Value
Experts below 100 routing hits	0.19% (19 of 10,240 (layer, expert) pairs)
Median hits per expert	7,050
Minimum hits in any layer	14

Full per-layer histograms are in expert_coverage.json in this repo.

Experts in the starved tail receive little tuning signal. On this model's target workload they also receive little inference traffic, so the practical impact is bounded — but it is a real property of any domain-calibrated MoE quant, and it is reported here rather than left implicit.

The BF16 keep-list

Two independent ways to silently ruin this model, both guarded here:

Table with columns: Kept BF16, Why
Kept BF16	Why
`mlp.gate` (the MoE router, one per layer)	Selects 8 of 256 experts. Quantizing it perturbs which experts fire — categorically worse than perturbing weights inside an expert, and it shows up as diffuse quality loss rather than obvious breakage.
`linear_attn.in_proj_a` / `in_proj_b`	Per-layer recurrence-control projections of the 30 GatedDeltaNet layers. Tiny, but they steer the recurrent state. AutoRound's docs state Mamba is unsupported, so it will not skip these for you.
`mlp.shared_expert_gate`	Gating scalar for the always-on shared expert.
`visual.*`	Vision tower — vLLM requires BF16.

Everything else is quantized, including all mlp.experts.N.{gate,up,down}_proj, mlp.shared_expert.*, linear_attn.{in_proj_qkv,in_proj_z,out_proj}, and the full-attention q/k/v/o.

The keep-list is not passed as glob patterns — AutoRound's layer_config matching is regex, so a leading * silently matches nothing while it quantizes the layer anyway. This build enumerates the real module tree on a meta device, passes exact names, and asserts counts derived from config.json: routers must equal the layer count (40), in_proj_a/b must equal the linear-attention layer count (30), and no expert matrix may appear in the keep-list. "A rule matched something" is too weak a check when one over-match would leave 30,720 expert matrices in BF16.

Deployment (vLLM)

bash
vllm serve fasolack/Qwen3.6-35B-A3B-W4A16 \
    --served-model-name qwen3.6-35b-a3b-w4a16 \
    --tensor-parallel-size 2 \
    --max-model-len 262144 \
    --kv-cache-dtype fp8 \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml

--tool-call-parser qwen3_xml, not hermes — this model emits the XML style.
TP=2 wants a real interconnect. With only ~3B active parameters the per-layer compute is small, so the all-reduce is a larger share of each step than it would be for a dense model. On NVLink-bridged cards vLLM's custom all-reduce engages and this is fast; straddling PCIe-only cards without P2P gives up much of the benefit.
If startup OOMs during CUDA-graph capture, add --enforce-eager.

Evaluation

Measured with EleutherAI lm-evaluation-harness against the served endpoint, using the model's recommended sampling.

Note on stop tokens: this model's own reasoning scaffold naturally writes the word "Question" while restating the problem, which collides with GSM8K's (and other tasks') default stop-list (['Question:', '</s>', '<|im_end|>']) and truncates generation almost immediately. Measured with until=<|im_end|> overriding the task default (evaluate.sh's GEN_KWARGS) -- without this override GSM8K measures 3.6-4.4% instead of the real number below. This is a model/harness interaction, not a quantization artifact: reproduced identically against a hand-built single-turn request bypassing lm-eval entirely.

Note on HumanEval methodology: this model's reasoning habit is strong enough that it breaks out of chat-formatted code-completion prompts too -- even lm-eval's own humaneval_instruct task (which pre-seeds the assistant turn inside an open ```python fence) still produced prose instead of code, scoring 0%. Measured instead via local-completions against vLLM's raw /v1/completions endpoint (no chat template at all, direct code continuation) -- the standard way this benchmark was originally designed to run. Spot-checked several failing samples: all were syntactically complete, genuine logic misses (not truncated or malformed output), confirming this is a clean measurement.

Note on HumanEval sampling -- two rows, on purpose: lm-eval's task config sets do_sample: False, but that flag is dead for the API backend used here -- openai_completions.py pops and discards it; only temperature reaches the actual request. So without an explicit override, HumanEval ran with real temperature=1.0 stochastic sampling, one sample per problem (n=1, no averaging) -- a materially riskier setup for code than the deterministic greedy decoding that single-number published pass@1 figures conventionally use, since neither the code nor its decoding got a second chance. Re-ran with temperature=0 (greedy): it recovered ~7 points. GSM8K/MMLU-Pro are far less sensitive to single-sample variance (a stray token rarely invalidates a whole multi-step answer or a multiple-choice pick), so they weren't re-run at greedy.

Table with columns: Task, BF16 base, This build (int4 W4A16)
Task	BF16 base	This build (int4 W4A16)
GSM8K	96.59% ± 0.50%	95.45% ± 0.57%
HumanEval (pass@1, temperature=1.0, n=1)	44.51% ± 3.89%	52.44% ± 3.91%
HumanEval (pass@1, greedy)	63.41% ± 3.77%	59.76% ± 3.84%
MMLU-Pro (100/subject)	83.00% ± 0.98%	82.57% ± 0.99%

BF16 baseline measured on a one-off TP=4 rig (this model's own recommended deployment is TP=2 on the NVLink pair, per the note above -- TP=4 was only used here because the 67 GiB BF16 checkpoint needs all four GPUs to fit). Interconnect topology affects serving speed, not the numbers themselves -- these are the actual model outputs, nothing about the comparison is approximate. Getting a usable throughput out of that rig took real debugging; see STATUS.md if you're ever serving this architecture at TP=4 without NVLink yourself.

Reading the table: GSM8K and MMLU-Pro are both close between the two columns, with BF16 very slightly ahead on each -- consistent with what you'd expect from a well-calibrated quantization (small, roughly noise-level cost). HumanEval is the outlier -- and specifically the temperature=1.0 row, not the greedy one: BF16 greedy actually comes out above the quantized build's greedy score (63.41% vs 59.76%), the more apples-to-apples comparison since neither run's decoding has a second chance either way. The temp=1.0 gap (44.51% vs 52.44%) is well within combined stderr (~5.5 pts) for a 164-problem, single-sample benchmark -- read it as sampling noise, not a quantization win, given the greedy numbers point the other way.

Limitations and intended use

Calibrated narrowly on purpose. Rounding is optimised for coding and tool calling. Outside that distribution it may be marginally worse than a generically-calibrated W4A16 build of the same model. See the expert-coverage section for the MoE-specific dimension of this.
Weight-only quantization. Activations stay BF16 → maximal compatibility, no activation-quant speedup. On Blackwell hardware an NVFP4 (W4A4) build will be faster.
The vision tower is untouched and no image data appears in the calibration set (quant_nontext_module=False), so multimodal behaviour should track the base model — but it is not separately evaluated here.
Inherits the base model's limitations and biases. See the base model card.

Reproducibility

python
from auto_round import AutoRound

# layer_config is built by enumerating the real module tree on a meta device and
# marking mlp.gate / shared_expert_gate / in_proj_a / in_proj_b / visual.* /
# mtp.* / lm_head as {"bits": 16}, with count assertions against config.json.
ar = AutoRound(
    "Qwen/Qwen3.6-35B-A3B",
    scheme="W4A16",
    dataset=texts,                  # the 1024-sample coding/tool-calling mix above
    nsamples=1024, seqlen=2048, iters=100,  # reduced from 200, see Calibration section
    batch_size=1, gradient_accumulate_steps=4,
    low_gpu_mem_usage=True,
    quant_nontext_module=False,
    trust_remote_code=True, seed=42,
    layer_config=BF16_KEEP_LIST,
)
ar.quantize_and_save(output_dir="Qwen3.6-35B-A3B-W4A16", format="llm_compressor")

Toolchain: auto-round ≥0.15.0, transformers with qwen3_5_moe support, compressed-tensors, torch + CUDA, Python 3.12. Quantized on RTX 3090s.

Citation

bibtex
@article{cheng2023optimize,
  title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
  author={Cheng, Wenhua and others},
  journal={arXiv:2309.05516},
  year={2023}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

fasolack

Model Tree

Base

Qwen/Qwen3.6-35B-A3B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality