GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
JuanGiner
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
JuanGiner
Model Tree
mistralai/Voxtral-Mini-3B-2507
Input Modalities
Output Modalities
Supported Functionality
Kept as a reference variant to compare against audio-calibrated builds. If your production input is audio, prefer AWQ4-ARG (Spanish specialist) or AWQ4-GEN (multi-language).
Data: FLEURS + Common Voice 15 + Multilingual LibriSpeech + ylacombe/google-argentinian-spanish (es-AR). Hardware: 2x RTX 4090, vLLM 0.25. Bold = best 4-bit variant.
| Metric | Voxtral-Mini bf16 (baseline) | AWQ4-ARG | AWQ4-EN | AWQ4-GEN |
|---|---|---|---|---|
| WER es | 5.82 | 4.68 | 4.60 | 4.55 |
| WER de | 6.06 | 7.13 | 6.66 | 8.16 |
| WER fr | 8.01 | 10.59 | 6.70 | 9.33 |
| WER it | 6.50 | 7.02 | 7.01 | 6.94 |
| WER pt | 7.16 | 7.21 | 6.14 | 6.08 |
| WER nl | 9.24 | 7.95 | 11.29 | 10.59 |
| WER en (no MLS) | 6.65 | 7.00 | 6.96 | 7.01 |
| WER avg-6 | 7.13 | 7.43 | 7.07 | 7.61 |
| WER es-AR | 4.69 | 5.00 | 5.11 | 5.04 |
| VRAM (weights) | 8.8 GB | 4.36 GB | 4.36 GB | 4.36 GB |
| Throughput (tok/s, higher=better) | 170.9 | 185.0 | 186.3 | 186.5 |
This card is the AWQ4-EN variant.
bash
# 1. In your vLLM venv, add the deps Voxtral needsuv pip install --python /path/to/vllm/venv/python "mistral-common[soundfile]"# 2. Servevllm serve JuanGiner/AWQ4-EN \--tokenizer-mode mistral \--config-format mistral \--max-model-len 8192 \--gpu-memory-utilization 0.9 \--port 8000
Then send audio to POST /v1/audio/transcriptions in OpenAI-compatible format.
512 samples of English text (HuggingFaceH4/ultrachat_200k), no audio in calib. Scales get tuned for the manifold of text-token activations, not audio-encoder outputs.
Quantization was done with llmcompressor 0.10 (AWQ modifier, W4A16 scheme, group_size=128, duo_scaling=True),
targeting Linear modules inside LlamaDecoderLayer. The compressed-tensors format matches what vLLM 0.25.x expects
for AWQ models (weights re-fused for the vLLM Voxtral loader).
Base: mistralai/Voxtral-Mini-3B-2507, Apache-2.0.
This quantized derivative is released under the same license.