GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
nintwentydo
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
nintwentydo
Model Tree
mistral-experimental/pixtral-12b
Input Modalities
Output Modalities
Supported Functionality
Quant of mistral-community/pixtral-12b using LLM Compressor for optimised inference on VLLM.
FP8 dynamic quant on language model, and FP8 quant of KV cache. multi_modal_projector and vision_tower left in FP16 since it's a small part of the model.
Calibrated on 2048 ultrachat samples.
Example VLLM usage
text
vllm serve nintwentydo/pixtral-12b-FP8-dynamic-FP8-KV-cache --quantization fp8 --kv-cache-dtype fp8Supported on Nvidia GPUs with compute capability > 8.9 (Ada Lovelace, Hopper).
Edit: Something seems to be wrong with the tokenizer. If you have any issues add --tokenizer mistral-community/pixtral-12b to your VLLM command line args.