What Is Quantization
Quantization is a technique that reduces the precision of a generative AI model’s parameters, optimizing memory usage and inference speed while maintaining response quality.
Friendli Container Supports
- Online quantization: Quantize your model on the fly at serving time. You don’t need to prepare pre-quantized weights in advance. Launch the model with the
--quantization option, and the system dynamically quantizes it as the container starts.
- Serving a pre-quantized model: Serve a model that has been already quantized beforehand. In this mode, you use model weights that were already quantized and load them during serving.
Serving a Model with Online Quantization
If you want to serve your own model but need to quantize it or adjust its precision, Friendli Container offers online quantization. You don’t need to prepare a quantized model in advance.
Once your model is ready, you can serve it with online quantization by adding the --quantization argument when running Friendli Container.
--quantization (8bit|4bit|16bit): Applies online quantization with the specified precision. It automatically detects your hardware and selects a suitable quantization scheme.
- Use
--quantization 8bit for NVIDIA Ada, Hopper, and Blackwell GPUs.
- Use
--quantization 4bit for NVIDIA Hopper and Blackwell GPUs.
To dequantize a model to 16-bit precision, use --quantization 16bit.
Example: deepseek-ai/DeepSeek-R1 with 4-Bit Online Quantization on NVIDIA H200 GPUs
To serve online quantized models efficiently, you must run a policy search to explore the optimal execution policy. Learn how to run the policy search at Running Policy Search. Serving a Pre-Quantized Model
If you have already quantized and uploaded a model to the Hugging Face Hub, Friendli Container supports the model with the following options:
Example: openai/gpt-oss-120b on NVIDIA B200 GPU
To serve pre-quantized models efficiently, you must run a policy search to explore the optimal execution policy. Learn how to run the policy search at Running Policy Search. 
