GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
WiktorMatuszek
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
WiktorMatuszek
Model Tree
abacusai/Smaug-Mini
Input Modalities
Output Modalities
Supported Functionality
0.47.0general/ptq/fp8_default-kv_fp8_castThe checkpoint is stored in ModelOpt's Hugging Face format and includes hf_quant_config.json.
Example vLLM invocation:
bash
vllm serve WiktorMatuszek/smaug-mini-fp8 \ --quantization modelopt \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coderThe source Smaug-Mini model supports up to a 262,144-token context. Set --max-model-len to a value appropriate for your available memory and workload.
Smaug-Mini inherits its MTP head from the underlying Qwen model. The official Smaug-Mini model card states that this head was not retrained after the language-trunk fine-tune and recommends leaving MTP speculative decoding disabled. Standard decoding is unaffected.
In our earlier serving benchmark on one RTX PRO 6000, using greedy decoding over 256 held-out prompts with 512 generated tokens and 16 concurrent requests, the unmodified native MTP head measured:
| Verifier | Mean accepted length | Per-position acceptance |
|---|---|---|
| FP8 | 3.15 | 0.86 / 0.71 / 0.57 |
| MXFP8 | 3.14 | 0.86 / 0.71 / 0.57 |
At 128 concurrent requests with a 16k maximum output length, the same test setup measured roughly 1,000 tok/s with MTP enabled versus 1,400–1,560 tok/s without MTP. These are serving-performance measurements, not model-quality scores; they are included to document why MTP is not recommended for this Smaug-Mini release.
The full capability evaluation for the unquantized model is published on the Smaug-Mini model card. This repository does not claim an independent rerun of that benchmark suite. Quantization can change outputs, so evaluate the checkpoint on your own workload before deployment.
Apache-2.0, following the source checkpoint. Smaug-Mini is published by Abacus.AI; this quantization is an independent community conversion.