GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Intel
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Intel
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
This model is a INT2/4 bits mixed weight-only quantized version of Qwen/Qwen3.6-35B-A3B, generated with AutoRound.
The checkpoint uses the compressed-tensors WNA16 format for vLLM-compatible inference:
compressed-tensors, pack-quantizedThis checkpoint requires compressed-tensors>=0.17.1. Please also follow the license and usage policy of the original base model.
The following command follows the AutoRound setup used for this checkpoint:
bash
auto_round \--model Qwen/Qwen3.6-35B-A3B \--bits 4 \--group_size 128 \--layer_config "{.*experts.*:{bits:2,group_size:64}}" \--enable_alg_ext \--enable_torch_compile \--iters 1000 \--nsamples 512 \--format auto_round,llm_compressor \--output_dir ./Qwen3.6-35B-A3B-mix
This sets the default quantization scheme to W4G128 and overrides routed expert layers to W2G64.
The command exports both AutoRound format and llm-compressor format. AutoRound format is useful for AutoRound-native workflows and currently supports evaluation through the Hugging Face backend, for example lm_eval --model hf. For faster and more efficient serving/evaluation, use the llm-compressor export, which produces the compressed-tensors checkpoint consumed by vLLM.
This model requires vLLM support for compressed-tensors WNA16 MoE checkpoints with the Humming backend. The support is tracked in vLLM PR #48918.
Until the PR is available in your installed vLLM release, install vLLM from the PR branch:
bash
pip install -v "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/48918/head"pip install "compressed-tensors>=0.17.1"
Serve the model:
bash
vllm serve <ORG_OR_USER>/<MODEL_ID> \--dtype bfloat16 \--tensor-parallel-size 1 \--max-model-len 8192 \--gpu-memory-utilization 0.85 \--trust-remote-code \--served-model-name qwen-mixed-2bit \--gdn-prefill-backend triton
For this compressed-tensors WNA16 checkpoint, a supported vLLM build selects the Humming MoE path automatically. Do not pass --moe-backend humming unless your vLLM build explicitly requires it.
Example request:
bash
curl http://localhost:8000/v1/chat/completions \-H "Content-Type: application/json" \-d '{"model": "qwen-mixed-2bit","messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Summarize mixed-precision MoE quantization in one sentence."}],"temperature": 0,"max_tokens": 256}'
EvalScope is the recommended evaluation path for this checkpoint because it evaluates the served vLLM model through the OpenAI-compatible API.
Install EvalScope in the same environment as the compatible vLLM build:
bash
pip install evalscope
Start vLLM:
bash
vllm serve <ORG_OR_USER>/<MODEL_ID> \--dtype bfloat16 \--tensor-parallel-size 1 \--max-model-len 8192 \--gpu-memory-utilization 0.85 \--trust-remote-code \--served-model-name qwen-mixed-2bit \--gdn-prefill-backend triton \--port 8000
Run EvalScope:
bash
evalscope eval \--model qwen-mixed-2bit \--api-url http://127.0.0.1:8000/v1 \--eval-type openai_api \--api-key EMPTY \--datasets mmlu gsm8k mmlu_pro \--generation-config '{"temperature": 0, "max_tokens": 2048, "timeout": 3000}' \--eval-batch-size 32 \--work-dir ./outputs/evalscope_qwen_mixed_2bit
lm_eval --model vllm can be useful for quick comparisons, but for this model card EvalScope is preferred for accuracy reporting.
bash
python -m lm_eval \--model vllm \--model_args "pretrained=<ORG_OR_USER>/<MODEL_ID>,tensor_parallel_size=1,dtype=bfloat16,max_model_len=8192,gpu_memory_utilization=0.85,trust_remote_code=True,add_bos_token=True,gdn_prefill_backend=triton" \--tasks mmlu,gsm8k,mmlu_pro \--gen_kwargs "max_gen_toks=2048" \--batch_size auto \--seed 42 \--output_path ./outputs/lmeval_vllm_mixed_2bit
Report formal scores from the full EvalScope run later.
Quantization can change model behavior and may reduce accuracy on some tasks. This checkpoint is optimized for experiments with mixed INT2/INT4 MoE inference and requires a vLLM build with compressed-tensors WNA16 MoE Humming support. Users should run task-specific quality and safety evaluations before deployment.
The model may produce factually incorrect, biased, unsafe, or otherwise undesirable outputs. Downstream users are responsible for validating the model for their use case and for following the base model license and applicable policies.
bibtex
@article{cheng2023optimize,title={Optimize weight rounding via signed gradient descent for the quantization of llms},author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},journal={arXiv preprint arXiv:2309.05516},year={2023}}