GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
RedHatAI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
RedHatAI
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
This model is a quantized version of Qwen/Qwen3.8-27B.
This model was obtained by applying GPTQ with AWQ activation smoothing via LLM Compressor, quantizing the weights and activations of the linear operators to the MXFP4 (FP4) data type and the KV cache to FP8, ready for inference with vLLM.
This optimization reduces the number of bits per parameter from 16 to 4 for the quantized weights and activations, reducing disk size and GPU memory requirements by approximately 75% (non-quantized components are kept in their original BF16 precision).
text
vllm serve RedHatAI/Qwen3.8-27B-MXFP4 \ --tensor-parallel-size 1 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xmlAdditional model-specific options (long-context scaling, speculative decoding via the built-in MTP head, etc.) are documented in the vLLM recipe for Qwen3.8-27B.
This model was created by applying LLM Compressor with calibration samples from perfectblend, as presented in the code snippet below.
python
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration from llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.modifiers.transform.awq import AWQModifierfrom llmcompressor.utils import load_context MODEL_ID = "Qwen/Qwen3.8-27B" # Load model.with load_context(Qwen3_5ForConditionalGeneration): model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID)processor = AutoProcessor.from_pretrained(MODEL_ID) recipe = [ AWQModifier(duo_scaling="both"), GPTQModifier( targets="Linear", scheme="MXFP4", ignore=[ "re:visual.*", "re:model.visual.*", r"re:.*lm_head", "re:.*embed_tokens$", r"re:.*linear_attn\.in_proj_a$", r"re:.*linear_attn\.in_proj_b$", ], kv_cache_scheme={ "num_bits": 8, "type": "float", "symmetric": True, "strategy": "tensor", "dynamic": False, "observer": "static_minmax", }, ),] # Apply quantization.oneshot( model=model, processor=processor, recipe=recipe, dataset="perfectblend", splits="train[:512]", max_seq_length=4096, num_calibration_samples=512, moe_calibrate_all_experts=True,) # Save to disk in compressed-tensors format.SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-MXFP4"model.save_pretrained(SAVE_DIR)processor.save_pretrained(SAVE_DIR)This model was evaluated on GSM8K Platinum, IFEval, MATH-500, AIME 2025, and GPQA Diamond using lm-evaluation-harness and lighteval, all served with vLLM (OpenAI-compatible API). Evaluations were run on 1× NVIDIA B200.
All tasks were evaluated 0-shot with seeds 1234/2345/3456 (3 repetitions); AIME 2025 used 8 seeds (1234/2345/3456/4567/5678/6789/7890/8901). The model was served with vLLM (the Deployment command) and the results were obtained using the following commands:
text
lm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=RedHatAI/Qwen3.8-27B-MXFP4,max_length=69632,base_url=http://127.0.0.1:3233/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,seed=1234,max_gen_toks=32000" \ --apply_chat_template \ --output_path results_gsm8k_platinum.jsonRun once per seed (changing --seed and the seed in gen_kwargs each time); use --tasks ifeval for IFEval.
litellm_config.yaml:
yaml
model_parameters: provider: hosted_vllm model_name: hosted_vllm/RedHatAI/Qwen3.8-27B-MXFP4 base_url: http://127.0.0.1:3233/v1 concurrent_requests: 32 timeout: 3600 generation_parameters: temperature: 1.0 top_p: 0.95 top_k: 20 seed: 1234 max_new_tokens: 65536Run once per seed (changing seed in the config each time):
text
lighteval endpoint litellm litellm_config.yaml 'math_500|0' \ --output-dir results/ --save-details lighteval endpoint litellm litellm_config.yaml 'aime25|0' \ --output-dir results/ --save-details lighteval endpoint litellm litellm_config.yaml 'gpqa:diamond|0' \ --output-dir results/ --save-detailsWe also assess performance of this model against a comparable amd/Qwen3.8-27B-Quark-AWQ-MXFP4 checkpoint and observe the following results:
Evaluations were collected through Inspect-AI, and the results above are average scores of three seeds except for MMLU-Pro and DeepSWE where we run only one seed due to extensive runtime of the benchmark.
Both checkpoints were evaluated in the same environment: vLLM vllm-0.31.0+rocm723 and on AMD MI355X GPUs.
vllm serve RedHatAI/Qwen3.8-27B-MXFP4 \ --tensor-parallel-size 1 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xmlfrom transformers import AutoProcessor, Qwen3_5ForConditionalGeneration from llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.modifiers.transform.awq import AWQModifierfrom llmcompressor.utils import load_context MODEL_ID = "Qwen/Qwen3.8-27B" # Load model.with load_context(Qwen3_5ForConditionalGeneration): model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID)processor = AutoProcessor.from_pretrained(MODEL_ID) recipe = [ AWQModifier(duo_scaling="both"), GPTQModifier( targets="Linear", scheme="MXFP4", ignore=[ "re:visual.*", "re:model.visual.*", r"re:.*lm_head", "re:.*embed_tokens$", r"re:.*linear_attn\.in_proj_a$", r"re:.*linear_attn\.in_proj_b$", ], kv_cache_scheme={ "num_bits": 8, "type": "float", "symmetric": True, "strategy": "tensor", "dynamic": False, "observer": "static_minmax", }, ),] # Apply quantization.oneshot( model=model, processor=processor, recipe=recipe, dataset="perfectblend", splits="train[:512]", max_seq_length=4096, num_calibration_samples=512, moe_calibrate_all_experts=True,) # Save to disk in compressed-tensors format.SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-MXFP4"model.save_pretrained(SAVE_DIR)processor.save_pretrained(SAVE_DIR)lm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=RedHatAI/Qwen3.8-27B-MXFP4,max_length=69632,base_url=http://127.0.0.1:3233/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,seed=1234,max_gen_toks=32000" \ --apply_chat_template \ --output_path results_gsm8k_platinum.jsonlighteval endpoint litellm litellm_config.yaml 'math_500|0' \ --output-dir results/ --save-detailslighteval endpoint litellm litellm_config.yaml 'aime25|0' \ --output-dir results/ --save-detailslighteval endpoint litellm litellm_config.yaml 'gpqa:diamond|0' \ --output-dir results/ --save-detailsmodel_parameters: provider: hosted_vllm model_name: hosted_vllm/RedHatAI/Qwen3.8-27B-MXFP4 base_url: http://127.0.0.1:3233/v1 concurrent_requests: 32 timeout: 3600 generation_parameters: temperature: 1.0 top_p: 0.95 top_k: 20 seed: 1234 max_new_tokens: 65536