GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
RedHatAI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
RedHatAI
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
This model is an updated quantized version of Qwen/Qwen3.8-27B, using a mixed-precision FP4/FP8 scheme with an unquantized language-model head and updated quantization scales. See Evaluation for accuracy results.
This model was produced by applying mixed-precision quantization to Qwen/Qwen3.8-27B. MLP projections are quantized to FP4, attention projections and the final MLP layers are quantized to FP8, and the KV cache is quantized to FP8, while the language-model head is kept in full precision to preserve output quality. The quantization scales were updated by calibrating on a 512-sample subset of the perfectblend dataset with a recipe that combines AWQ and GPTQ.
Only the weights and activations of the linear operators within the transformer blocks are quantized using LLM Compressor. The checkpoint is ~24.7 GB on disk (versus ~54 GB in BF16), reducing disk size and GPU memory requirements by roughly 70%.
markdown
vllm serve RedHatAI/Qwen3.8-27B-NVFP4 \--reasoning-parser qwen3 \--enable-auto-tool-choice \--tool-call-parser qwen3_xml \--speculative-config '{"model":"RedHatAI/Qwen3.8-27B-speculator.dspark","num_speculative_tokens":8,"method":"dspark"}'
This model was created by applying LLM Compressor with calibration samples from perfectblend, as presented in the code snippet below.
python
from compressed_tensors.quantization.quant_scheme import (FP8_DYNAMIC,NVFP4,QuantizationScheme,)from transformers import AutoProcessor, Qwen3_5ForConditionalGenerationfrom llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.modifiers.transform.awq import AWQModifierfrom llmcompressor.utils import load_contextMODEL_ID = "Qwen/Qwen3.8-27B"# Load model.with load_context(Qwen3_5ForConditionalGeneration):model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID)processor = AutoProcessor.from_pretrained(MODEL_ID)recipe = [AWQModifier(duo_scaling="both"),GPTQModifier(config_groups={"attention": QuantizationScheme(targets=[r"re:.*self_attn\.(q|k|v|o)_proj$",r"re:.*linear_attn\.(in_proj_qkv|in_proj_z|out_proj)$",r"re:.*layers\.(56|57|58|59|60|61|62|63)\.mlp\..*(gate|up|down)_proj$",],**FP8_DYNAMIC,),"mlp": QuantizationScheme(targets=[r"re:.*mlp\..*(gate|up|down)_proj$"],**NVFP4,),},ignore=["re:visual.*","re:model.visual.*","re:.*lm_head",],kv_cache_scheme={"num_bits": 8,"type": "float","symmetric": True,"strategy": "tensor","dynamic": False,"observer": "static_minmax",},),]# Apply quantization.oneshot(model=model,processor=processor,recipe=recipe,dataset="perfectblend",splits="train[:512]",max_seq_length=4096,num_calibration_samples=512,moe_calibrate_all_experts=True,)# Save to disk in compressed-tensors format.SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-NVFP4-GPTQ-AWQ"model.save_pretrained(SAVE_DIR)processor.save_pretrained(SAVE_DIR)
This model was evaluated on GSM8K Platinum, MATH-500, AIME 2025, GPQA Diamond, and IFEval using lm-evaluation-harness (https://github.com/neuralmagic/lm-evaluation-harness) and lighteval (https://github.com/neuralmagic/lighteval), and on SWE Bench using Inspect AI (https://github.com/UKGovernmentBEIS/inspect_ai), served with vLLM (OpenAI-compatible API). Evaluations were run on 1x B200 GPU.
Recovery vs. BF16 baseline
The results were obtained using the following commands. Each benchmark was run multiple times with different random seeds — 3 repetitions for GSM8K Platinum, MATH-500, GPQA Diamond, and IFEval, and 8 repetitions for AIME 2025 — and the reported score is the mean across seeds.
GSM8K Platinum & IFEval (lm-eval, 0-shot)
Run once per seed:
markdown
lm_eval --model local-chat-completions \--tasks gsm8k_platinum_cot_llama \--model_args "model=RedHatAI/Qwen3.8-27B-NVFP4,max_length=69632,base_url=http://127.0.0.1:3235/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 \--apply_chat_template \--output_path results_gsm8k_platinum.json \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,max_gen_toks=32000,seed=1234"
markdown
lm_eval --model local-chat-completions \--tasks ifeval \--model_args "model=RedHatAI/Qwen3.8-27B-NVFP4,max_length=69632,base_url=http://127.0.0.1:3235/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 \--apply_chat_template \--output_path results_ifeval.json \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,max_gen_toks=32000,seed=1234"
MATH-500, AIME 2025, GPQA Diamond (lighteval, 0-shot)
litellm_config.yaml:
markdown
model_parameters:provider: hosted_vllmmodel_name: hosted_vllm/RedHatAI/Qwen3.8-27B-NVFP4base_url: http://127.0.0.1:3235/v1api_key: ''timeout: 3600concurrent_requests: 32generation_parameters:temperature: 1.0max_new_tokens: 65536top_p: 0.95top_k: 20seed: 1234
Run once per seed (changing seed in the config each time):
markdown
lighteval endpoint litellm litellm_config.yaml 'math_500@1@3|0' --output-dir results/ --save-detailslighteval endpoint litellm litellm_config.yaml 'aime25@1@8|0' --output-dir results/ --save-detailslighteval endpoint litellm litellm_config.yaml 'gpqa:diamond@1@3|0' --output-dir results/ --save-details