GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
RedHatAI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
RedHatAI
Model Tree
Qwen/Qwen3-32B
Input Modalities
Output Modalities
Supported Functionality
This model was obtained by quantizing activations and weights of Qwen3-32B to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%.
Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. The llm-compressor library is used for quantization.
This model can be deployed efficiently using the vLLM backend, as shown in the example below.
python
from vllm import LLM, SamplingParamsfrom transformers import AutoTokenizer model_id = "RedHatAI/Qwen3-32B-FP8-dynamic"number_gpus = 1sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, min_p=0, max_tokens=256) messages = [ {"role": "user", "content": prompt}] tokenizer = AutoTokenizer.from_pretrained(model_id) messages = [{"role": "user", "content": "Give me a short introduction to large language model."}] prompts = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) llm = LLM(model=model_id, tensor_parallel_size=number_gpus) outputs = llm.generate(prompts, sampling_params) generated_text = outputs[0].outputs[0].textprint(generated_text)vLLM aslo supports OpenAI-compatible serving. See the documentation for more details.
python
from llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.transformers import oneshotfrom transformers import AutoModelForCausalLM, AutoTokenizer # Load modelmodel_stub = "Qwen/Qwen3-32B"model_name = model_stub.split("/")[-1] model = AutoModelForCausalLM.from_pretrained(model_stub) tokenizer = AutoTokenizer.from_pretrained(model_stub) # Configure the quantization algorithm and schemerecipe = QuantizationModifier( ignore=["lm_head"], targets="Linear", scheme="FP8_dynamic",) # Apply quantizationoneshot( model=model, recipe=recipe,) # Save to disk in compressed-tensors formatsave_path = model_name + "-FP8-dynamic"model.save_pretrained(save_path)tokenizer.save_pretrained(save_path)print(f"Model and tokenizer saved to: {save_path}")The model was evaluated on the OpenLLM leaderboard tasks (versions 1 and 2), using lm-evaluation-harness, and on reasoning tasks using lighteval. vLLM was used for all evaluations.
lm-evaluation-harness
text
lm_eval \ --model vllm \ --model_args pretrained="RedHatAI/Qwen3-32B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=2 \ --tasks openllm \ --apply_chat_template\ --fewshot_as_multiturn \ --batch_size autotext
lm_eval \ --model vllm \ --model_args pretrained="RedHatAI/Qwen3-32B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=2 \ --tasks mgsm \ --apply_chat_template\ --batch_size autotext
lm_eval \ --model vllm \ --model_args pretrained="RedHatAI/Qwen3-32B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=16384,enable_chunk_prefill=True,tensor_parallel_size=2 \ --tasks leaderboard \ --apply_chat_template\ --fewshot_as_multiturn \ --batch_size autolighteval
lighteval_model_arguments.yaml
yaml
model_parameters: model_name: RedHatAI/Qwen3-32B-FP8-dynamic dtype: auto gpu_memory_utilization: 0.9 tensor_parallel_size: 2 max_model_length: 40960 generation_parameters: temperature: 0.6 top_k: 20 min_p: 0.0 top_p: 0.95 max_new_tokens: 32768text
lighteval vllm \ --model_args lighteval_model_arguments.yaml \ --tasks lighteval|aime24|0|0 \ --use_chat_template = truetext
lighteval vllm \ --model_args lighteval_model_arguments.yaml \ --tasks lighteval|aime25|0|0 \ --use_chat_template = truetext
lighteval vllm \ --model_args lighteval_model_arguments.yaml \ --tasks lighteval|math_500|0|0 \ --use_chat_template = truetext
lighteval vllm \ --model_args lighteval_model_arguments.yaml \ --tasks lighteval|gpqa:diamond|0|0 \ --use_chat_template = truetext
lighteval vllm \ --model_args lighteval_model_arguments.yaml \ --tasks extended|lcb:codegeneration \ --use_chat_template = truefrom vllm import LLM, SamplingParamsfrom transformers import AutoTokenizer model_id = "RedHatAI/Qwen3-32B-FP8-dynamic"number_gpus = 1sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, min_p=0, max_tokens=256) messages = [ {"role": "user", "content": prompt}] tokenizer = AutoTokenizer.from_pretrained(model_id) messages = [{"role": "user", "content": "Give me a short introduction to large language model."}] prompts = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) llm = LLM(model=model_id, tensor_parallel_size=number_gpus) outputs = llm.generate(prompts, sampling_params) generated_text = outputs[0].outputs[0].textprint(generated_text)from llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.transformers import oneshotfrom transformers import AutoModelForCausalLM, AutoTokenizer # Load modelmodel_stub = "Qwen/Qwen3-32B"model_name = model_stub.split("/")[-1] model = AutoModelForCausalLM.from_pretrained(model_stub) tokenizer = AutoTokenizer.from_pretrained(model_stub) # Configure the quantization algorithm and schemerecipe = QuantizationModifier( ignore=["lm_head"], targets="Linear", scheme="FP8_dynamic",) # Apply quantizationoneshot( model=model, recipe=recipe,) # Save to disk in compressed-tensors formatsave_path = model_name + "-FP8-dynamic"model.save_pretrained(save_path)tokenizer.save_pretrained(save_path)print(f"Model and tokenizer saved to: {save_path}")lm_eval \ --model vllm \ --model_args pretrained="RedHatAI/Qwen3-32B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=2 \ --tasks openllm \ --apply_chat_template\ --fewshot_as_multiturn \ --batch_size autolm_eval \ --model vllm \ --model_args pretrained="RedHatAI/Qwen3-32B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=2 \ --tasks mgsm \ --apply_chat_template\ --batch_size autolm_eval \ --model vllm \ --model_args pretrained="RedHatAI/Qwen3-32B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=16384,enable_chunk_prefill=True,tensor_parallel_size=2 \ --tasks leaderboard \ --apply_chat_template\ --fewshot_as_multiturn \ --batch_size automodel_parameters: model_name: RedHatAI/Qwen3-32B-FP8-dynamic dtype: auto gpu_memory_utilization: 0.9 tensor_parallel_size: 2 max_model_length: 40960 generation_parameters: temperature: 0.6 top_k: 20 min_p: 0.0 top_p: 0.95 max_new_tokens: 32768lighteval vllm \ --model_args lighteval_model_arguments.yaml \ --tasks lighteval|aime24|0|0 \ --use_chat_template = truelighteval vllm \ --model_args lighteval_model_arguments.yaml \ --tasks lighteval|aime25|0|0 \ --use_chat_template = truelighteval vllm \ --model_args lighteval_model_arguments.yaml \ --tasks lighteval|math_500|0|0 \ --use_chat_template = truelighteval vllm \ --model_args lighteval_model_arguments.yaml \ --tasks lighteval|gpqa:diamond|0|0 \ --use_chat_template = truelighteval vllm \ --model_args lighteval_model_arguments.yaml \ --tasks extended|lcb:codegeneration \ --use_chat_template = true