GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
RedHatAI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
RedHatAI
Model Tree
Base
google/gemma-4-31B-it
Quantized
this model
Input Modalities
Text
Image
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
Container
This model is a quantized version of google/gemma-4-31B-it. It was evaluated on several tasks to assess its quality in comparison to the unquantized model.
This model was obtained by quantizing the weights and activations of google/gemma-4-31B-it to FP4 data type using the NVFP4 format, ready for inference with vLLM. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%.
Weights are quantized with FP4 (group_size=16), and activations are quantized with FP4 using local per-group scaling. Only the weights and activations of the linear operators within transformer blocks are quantized using LLM Compressor. Vision tower, embedding, and output head layers are kept in their original precision.
This model can be deployed using vLLM. For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the Gemma 4 vLLM usage guide.
markdown
vllm serve RedHatAI/gemma-4-31B-it-NVFP4 \--max-model-len 32768 \--gpu-memory-utilization 0.90 \--enable-auto-tool-choice \--reasoning-parser gemma4 \--tool-call-parser gemma4 \--chat-template examples/tool_chat_template_gemma4.jinja \--limit-mm-per-prompt '{"image": 4, "audio": 1}' \--async-scheduling
Tip: For text-only workloads, pass
--limit-mm-per-prompt '{"image": 0, "audio": 0}'to skip vision encoder memory allocation and free up GPU memory for a longer context window.
python
from openai import OpenAIopenai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8000/v1"client = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)model = "RedHatAI/gemma-4-31B-it-NVFP4"messages = [{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},]outputs = client.chat.completions.create(model=model,messages=messages,extra_body={"chat_template_kwargs": {"enable_thinking": True}},)generated_text = outputs.choices[0].message.contentprint(generated_text)
This model was created by applying NVFP4 quantization with LLM Compressor, as presented in the code snippet below.
python
from datasets import load_datasetfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom llmcompressor import applyfrom llmcompressor.modifiers.quantization import QuantizationModifierMODEL_ID = "google/gemma-4-31B-it"SAVE_DIR = MODEL_ID.split("/")[1] + "-NVFP4"model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto", device_map="auto")tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)ds = load_dataset("mgoin/ultrachat_200k_s3", split="train_sft")calibration_data = [ex["prompt"] for ex in ds.select(range(512))]recipe = QuantizationModifier(targets=["Linear"],ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"],scheme="NVFP4",)apply(model=model, tokenizer=tokenizer, recipe=recipe, calibration_data=calibration_data)model.save_pretrained(SAVE_DIR, save_compressed=True)tokenizer.save_pretrained(SAVE_DIR)
This model was evaluated on GSM8K Platinum, MMLU-Pro, IFEval, MATH-500, AIME 2025, GPQA Diamond, LiveCodeBench v6, and BFCLv4 (function calling) using lm-evaluation-harness, lighteval, and BFCL — all served with vLLM (OpenAI-compatible API). Accuracy results are reported both without and with thinking enabled; BFCLv4 was evaluated with thinking enabled.
The results were obtained using the following commands:
Each benchmark was run 3 times with different random seeds (1234, 2345, 3456) and the scores were averaged; AIME 2025 used 8 seeds.
vLLM server:
markdown
vllm serve RedHatAI/gemma-4-31B-it-NVFP4 \--served-model-name gemma-4-31b-it-NVFP4 \--max-model-len 32768 \--gpu-memory-utilization 0.90 \--language-model-only \--enable-auto-tool-choice \--reasoning-parser gemma4 \--tool-call-parser gemma4 \--chat-template examples/tool_chat_template_gemma4.jinja \--async-scheduling \--default-chat-template-kwargs '{"enable_thinking": true}'
Note: To reproduce the results without thinking, remove
--default-chat-template-kwargs '{"enable_thinking": true}'. To run without tool calling, remove--enable-auto-tool-choice,--tool-call-parser gemma4, and--reasoning-parser gemma4.
markdown
lm_eval --model local-chat-completions \--tasks gsm8k_platinum_cot_llama \--model_args "model=gemma-4-31b-it-NVFP4,max_length=32768,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=1200" \--num_fewshot 0 \--apply_chat_template \--output_path results_gsm8k_platinum.json \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"
markdown
lm_eval --model local-chat-completions \--tasks mmlu_pro_chat \--model_args "model=gemma-4-31b-it-NVFP4,max_length=32768,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=1200" \--num_fewshot 0 \--apply_chat_template \--output_path results_mmlu_pro.json \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"
markdown
lm_eval --model local-chat-completions \--tasks ifeval \--model_args "model=gemma-4-31b-it-NVFP4,max_length=32768,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=1200" \--num_fewshot 0 \--apply_chat_template \--output_path results_ifeval.json \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"
litellm_config.yaml:
yaml
model_parameters:provider: hosted_vllmmodel_name: hosted_vllm/gemma-4-31b-it-NVFP4base_url: http://0.0.0.0:8000/v1api_key: ''timeout: 3600concurrent_requests: 32generation_parameters:temperature: 1.0max_new_tokens: 65536top_p: 0.95top_k: 64seed: 1234
Run once per seed (changing seed in the config each time):
markdown
lighteval endpoint litellm litellm_config.yaml 'math_500|0' \--output-dir results/ --save-detailslighteval endpoint litellm litellm_config.yaml 'aime25|0' \--output-dir results/ --save-detailslighteval endpoint litellm litellm_config.yaml 'gpqa:diamond|0' \--output-dir results/ --save-detailslighteval endpoint litellm litellm_config.yaml 'lcb:codegeneration_v6|0' \--output-dir results/ --save-details
BFCL requires the model to be registered in the leaderboard codebase before running evaluation.
Step 1 — Register the model in bfcl_eval/constants/model_config.py
Add the following entry to api_inference_model_map:
python
"gemma-4-31b-it-NVFP4": ModelConfig(model_name="gemma-4-31b-it-NVFP4",display_name="Gemma-4-31b-it-NVFP4 (FC)",url="https://huggingface.co/RedHatAI/gemma-4-31B-it-NVFP4",org="Google",license="Apache 2.0",model_handler=OpenAICompletionsHandler,input_price=None,output_price=None,is_fc_model=True,underscore_to_dot=True,),
Step 2 — Add the key to bfcl_eval/constants/supported_models.py
Add "gemma-4-31b-it-NVFP4" to the SUPPORTED_MODELS list.
Step 3 — Start the vLLM server (use the command at the top of this section; the --served-model-name flag ensures BFCL can find the model by its registered slug).
Step 4 — Generate responses and evaluate
markdown
bfcl generate --model gemma-4-31b-it-NVFP4 --test-category allbfcl evaluate --model gemma-4-31b-it-NVFP4 --test-category all