GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
RedHatAI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
RedHatAI
Model Tree
Qwen/Qwen3.5-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
This model is a quantized version of Qwen/Qwen3.5-35B-A3B. It was evaluated on several tasks to assess its quality in comparison to the unquantized model.
This model was obtained by quantizing the weights and activations of Qwen/Qwen3.5-35B-A3B to Int8 data type, ready for inference with vLLM.
This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
Only the weights and activations of the linear operators are quantized using LLM Compressor. Layers such as the visual encoder, linear attention (Gated DeltaNet), MoE router gates, shared experts, token embeddings and MTP layers are kept in original precision.
This model can be deployed efficiently using vLLM.
text
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --language-model-onlytext
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3Image input is supported by the upstream architecture but was not evaluated for this release.
text
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coderTool calling was not evaluated for this release.
text
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'Send requests to the server:
python
from openai import OpenAI openai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,) model = "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8" messages = [ {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},] outputs = client.chat.completions.create( model=model, messages=messages,) generated_text = outputs.choices[0].message.contentprint(generated_text)The checkpoint retains native MTP weights in BF16. This speculative-decoding configuration was not tested with the quantized checkpoint.
This model was created by applying LLM Compressor with w8a8 integer quantization using the GPTQModifier calibrating on the ultrachat dataset, as presented in the code snippet below.
python
from pathlib import Path from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration from llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.utils import load_context MODEL_ID = "Qwen/Qwen3.5-35B-A3B"DATASET = "ultrachat"SPLIT = "train_sft[:512]"save_dir = Path("gptq-ultrachat")save_dir.parent.mkdir(parents=True, exist_ok=True) with load_context(Qwen3_5MoeForConditionalGeneration): model = Qwen3_5MoeForConditionalGeneration.from_pretrained( MODEL_ID, device_map="auto_offload", )processor = AutoProcessor.from_pretrained(MODEL_ID) recipe = GPTQModifier( targets="Linear", scheme="W8A8", ignore=[ "lm_head", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate.*", "re:.*norm.*", "re:.*embed_tokens.*", "re:.*visual.*", "re:.*conv1d.*", ],) oneshot( model=model, processor=processor, dataset=DATASET, splits=SPLIT, recipe=recipe, max_seq_length=2048, num_calibration_samples=512, moe_calibrate_all_experts=True,) model.save_pretrained(str(save_dir), save_compressed=True)processor.save_pretrained(str(save_dir))print(f"Saved GPTQ UltraChat checkpoint to {save_dir}")The BF16 reference and this GPTQ W8A8 checkpoint were served with vLLM in --language-model-only mode. Accuracy scores are means over seeds 1234, 2345, and 3456. Recovery is the quantized score divided by the BF16 score.
Perplexity was measured on the full 62-document test set from the registered wikitext task. Lower is better.
| Metric | Qwen/Qwen3.5-35B-A3B (BF16) | RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 | Change |
|---|---|---|---|
| Word perplexity | 7.46346 | 7.50737 | +0.5883% |
| Byte perplexity | 1.45628 | 1.45788 | +0.1100% |
| Bits per byte | 0.542285 | 0.543868 | +0.2921% |
The accuracy evaluations used the evaluation frameworks lm-evaluation-harness and Lighteval. GSM8K Platinum and IFEval used their registered 0-shot prompts; Math500 and GPQA Diamond used the registered Lighteval tasks. MMLU used mmlu_cot_llama with 5-shot examples formatted as multi-turn chat and a maximum of 8,192 generated tokens. All accuracy tasks used temperature 1.0, top-p 0.95, top-k 20, and presence penalty 1.5.
The checkpoint was served with vLLM 0.30.1rc1.dev674+g87954c5b0, tensor parallel size 2, the Qwen 3 reasoning parser, and text-only mode.
bash
vllm serve "$MODEL_ID" \ --served-model-name "$SERVED_MODEL" \ --port "$PORT" \ --tensor-parallel-size 2 \ --max-model-len 69632 \ --reasoning-parser qwen3 \ --language-model-onlyThe same serving and evaluation settings were used for the BF16 reference by replacing the model ID with Qwen/Qwen3.5-35B-A3B.
lm_eval, 0-shot)bash
lm_eval --model local-chat-completions \ --tasks ifeval \ --model_args "model=${SERVED_MODEL},max_length=69632,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path "./eval/ifeval_seed${SEED}.json" \ --seed "$SEED" \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=32000"lm_eval, 5-shot)Restart vLLM with a 20,480-token context for this task, then run:
bash
for SEED in 1234 2345 3456; do lm_eval --model local-chat-completions \ --tasks mmlu_cot_llama \ --model_args "model=${SERVED_MODEL},max_length=20480,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 5 \ --apply_chat_template \ --fewshot_as_multiturn \ --output_path "./eval/mmlu_cot_llama_seed${SEED}.json" \ --seed "$SEED" \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=8192"donelighteval, 0-shot)bash
mkdir -p ./evalfor SEED in 1234 2345 3456; do CONFIG="./eval/lighteval_seed${SEED}.yaml" cat > "$CONFIG" <<EOFmodel_parameters: provider: hosted_vllm model_name: hosted_vllm/${SERVED_MODEL} base_url: ${API_BASE}/v1 api_key: "" timeout: 3600 concurrent_requests: 64 generation_parameters: max_new_tokens: 65536 presence_penalty: 1.5 seed: ${SEED} temperature: 1.0 top_k: 20 top_p: 0.95EOF lighteval endpoint litellm "$CONFIG" 'math_500|0' \ --output-dir "./eval/math500_seed${SEED}" --save-details lighteval endpoint litellm "$CONFIG" 'gpqa:diamond|0' \ --output-dir "./eval/gpqa_diamond_seed${SEED}" --save-detailsdoneThe PPL task used lm-eval's local-completions model against the vLLM /v1/completions endpoint, with the Qwen3.5 tokenizer, max_length=20480, num_concurrent=64, max_retries=3, and timeout=3600. It used a separate server configured for a 20,480-token context:
bash
lm_eval --model local-completions \ --tasks wikitext \ --model_args 'model=RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8,max_length=20480,base_url=http://localhost:8000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer=Qwen/Qwen3.5-35B-A3B,tokenizer_backend=huggingface,timeout=3600' \ --num_fewshot 0 \ --seed 1234 \ --output_path ./eval/wikitext2_ppl/resultsfrom openai import OpenAI openai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,) model = "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8" messages = [ {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},] outputs = client.chat.completions.create( model=model, messages=messages,) generated_text = outputs.choices[0].message.contentprint(generated_text)from pathlib import Path from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration from llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.utils import load_context MODEL_ID = "Qwen/Qwen3.5-35B-A3B"DATASET = "ultrachat"SPLIT = "train_sft[:512]"save_dir = Path("gptq-ultrachat")save_dir.parent.mkdir(parents=True, exist_ok=True) with load_context(Qwen3_5MoeForConditionalGeneration): model = Qwen3_5MoeForConditionalGeneration.from_pretrained( MODEL_ID, device_map="auto_offload", )processor = AutoProcessor.from_pretrained(MODEL_ID) recipe = GPTQModifier( targets="Linear", scheme="W8A8", ignore=[ "lm_head", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate.*", "re:.*norm.*", "re:.*embed_tokens.*", "re:.*visual.*", "re:.*conv1d.*", ],) oneshot( model=model, processor=processor, dataset=DATASET, splits=SPLIT, recipe=recipe, max_seq_length=2048, num_calibration_samples=512, moe_calibrate_all_experts=True,) model.save_pretrained(str(save_dir), save_compressed=True)processor.save_pretrained(str(save_dir))print(f"Saved GPTQ UltraChat checkpoint to {save_dir}")vllm serve "$MODEL_ID" \ --served-model-name "$SERVED_MODEL" \ --port "$PORT" \ --tensor-parallel-size 2 \ --max-model-len 69632 \ --reasoning-parser qwen3 \ --language-model-onlylm_eval --model local-chat-completions \ --tasks ifeval \ --model_args "model=${SERVED_MODEL},max_length=69632,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path "./eval/ifeval_seed${SEED}.json" \ --seed "$SEED" \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=32000"for SEED in 1234 2345 3456; do lm_eval --model local-chat-completions \ --tasks mmlu_cot_llama \ --model_args "model=${SERVED_MODEL},max_length=20480,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 5 \ --apply_chat_template \ --fewshot_as_multiturn \ --output_path "./eval/mmlu_cot_llama_seed${SEED}.json" \ --seed "$SEED" \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=8192"donemkdir -p ./evalfor SEED in 1234 2345 3456; do CONFIG="./eval/lighteval_seed${SEED}.yaml" cat > "$CONFIG" <<EOFmodel_parameters: provider: hosted_vllm model_name: hosted_vllm/${SERVED_MODEL} base_url: ${API_BASE}/v1 api_key: "" timeout: 3600 concurrent_requests: 64 generation_parameters: max_new_tokens: 65536 presence_penalty: 1.5 seed: ${SEED} temperature: 1.0 top_k: 20 top_p: 0.95EOF lighteval endpoint litellm "$CONFIG" 'math_500|0' \ --output-dir "./eval/math500_seed${SEED}" --save-details lighteval endpoint litellm "$CONFIG" 'gpqa:diamond|0' \ --output-dir "./eval/gpqa_diamond_seed${SEED}" --save-detailsdonelm_eval --model local-completions \ --tasks wikitext \ --model_args 'model=RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8,max_length=20480,base_url=http://localhost:8000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer=Qwen/Qwen3.5-35B-A3B,tokenizer_backend=huggingface,timeout=3600' \ --num_fewshot 0 \ --seed 1234 \ --output_path ./eval/wikitext2_ppl/results