Quantization Details

This model was quantized using NVIDIA ModelOpt with the following configurations:

Excluded from Quantization

The following modules are kept in full precision (BF16) to preserve accuracy:

• lm_head
• model.embed_vision*
• All self_attn layers (layers 0–59)

Usage

You can deploy this model using vLLM with the modelopt quantization backend. Please ensure you refer to the vLLM documentation for Gemma 4 for advanced serving options.

bash
Copy code
vllm serve vrfai/gemma-4-31B-it-nvfp4 \
  --quantization modelopt_fp4 \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --async-scheduling \
  --trust-remote-code

Quantization Script

The recipes and scripts used to quantize this model can be found in the following repository:

• VinRobotics/model-quantization-recipes