GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
88plug
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
88plug
Model Tree
Input Modalities
Output Modalities
Supported Functionality
These weights are compressed-tensors (pack-quantized / int-quantized).
| Runtime | Supported |
|---|---|
| vLLM ≥ 0.21 | Yes — preferred (auto-detect CT; no --quantization flag) |
transformers + compressed-tensors | Yes for many text models; multimodal may need custom code |
| Text Generation Inference (TGI) | Not supported for these CT packs |
| Hugging Face Inference Widget | Often fails — use vLLM locally instead |
bash
# Preferredvllm serve 88plug/<ModelName> --trust-remote-codeDo not deploy via TGI “text-generation-inference” paths — that backend does not load our CT format and produces opaque worker/load errors.
INT8 post-training quantization of openbmb/MiniCPM-o-4.5 — a compact omni model with vision (SigLIP2), audio (Whisper), and speech synthesis (CosyVoice2) built on a Qwen3-8B backbone. ~9 GB on disk. Runs on any 16 GB GPU.
| Property | Value |
|---|---|
| Base model | openbmb/MiniCPM-o-4.5 |
| Release tier | Provisional (datafree RTN — re-quant scheduled) |
| Quant method | datafree RTN W8A16 (weight-only INT8) |
| FLAC status | Not measured (T+7d milestone) |
| Architecture | Qwen3-8B LLM + SigLIP2 vision + Whisper audio + CosyVoice2 TTS |
| Quant format | compressed-tensors (native vLLM) |
| Quantized | model.llm transformer layers |
| Configuration | BF16 | W8A16 |
|---|---|---|
| Weights | ~18 GB | ~9 GB |
| Min GPU | 1× A100 40GB | 1× RTX 3090 24GB |
Tested with vLLM v0.21.0 (vllm/vllm-openai:v0.21.0-cu129-ubuntu2404). Weights are in compressed-tensors format — vLLM detects and loads quantization automatically. No --quantization flag needed.
bash
docker run --gpus device=0 -p 8080:8080 \ vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \ 88plug/MiniCPM-o-4.5-W8A16 \ --kv-cache-dtype fp8 \ --max-model-len 32768 \ --gpu-memory-utilization 0.90Weights are in compressed-tensors format — no --quantization flag needed. Requires vLLM ≥ v0.21.0. Mainline vLLM returns text only; CosyVoice2 TTS output is not supported.
Mainline llama.cpp supports MiniCPM-V (vision + text). For full CosyVoice2 speech output, use the tc-mb/llama.cpp-omni fork. Convert from BF16 base.
bash
python convert_hf_to_gguf.py openbmb/MiniCPM-o-4.5 \ --outfile MiniCPM-o-4.5-BF16.gguf llama-quantize MiniCPM-o-4.5-BF16.gguf MiniCPM-o-4.5-Q8_0.gguf Q8_0llama-quantize --imatrix calibration_datav3.txt \ MiniCPM-o-4.5-BF16.gguf MiniCPM-o-4.5-IQ4_XS.gguf IQ4_XS llama-server \ --model MiniCPM-o-4.5-Q8_0.gguf \ --n-gpu-layers 999 \ --ctx-size 32768 \ --port 8081Results pending.
| Engine | Format | Batch | ctx | tok/s | TTFT p50 | TTFT p99 | VRAM |
|---|---|---|---|---|---|---|---|
| vLLM v0.21.0 | W8A16 | 1 | 32k | — | — | — | — |
| vLLM v0.21.0 | W8A16 | 8 | 32k | — |
Hardware: A6000 48 GB, CUDA 12.9, driver 570.
| Component | Precision | Reason |
|---|---|---|
model.llm.* transformer layers | W8A16 INT8 | Quantized |
| Vision encoder (SigLIP2) | BF16 | Excluded |
| Audio encoder (Whisper) | BF16 | Excluded |
| CosyVoice2 TTS | BF16 | Excluded |
| Embeddings, LM head, norms | BF16 | Standard practice |
| Metric | Target |
|---|---|
| KL divergence vs BF16 | < 0.005 |
| MMLU recovery | ≥ 99.7% |
This is the first compressed-tensors W8A16 checkpoint for MiniCPM-o-4.5. It halves VRAM usage while retaining native vLLM serving with audio and vision input.
| Quant | Method | Size | GPU Compatibility | Notes |
|---|---|---|---|---|
| 88plug W8A16 (this) | compressed-tensors RTN W8A16 | ~9 GB | Any Ampere+ ≥16 GB | First W8A16; native vLLM; LLM backbone quantized |
| Community GGUF Q4_K_M | llama.cpp GGUF | ~5 GB | CPU / any GPU | Vision via mmproj; no CosyVoice2 in mainline |
| Community GGUF Q8_0 | llama.cpp GGUF | ~9 GB | Any GPU ≥10 GB |
model.llm transformer layers are quantized. Vision encoder (SigLIP2), audio encoder (Whisper), and CosyVoice2 TTS components stay BF16.tc-mb/llama.cpp-omni fork for speech synthesis.bibtex
@misc{minicpmo, title = {MiniCPM-o: A GPT-4o Level Multimodal LLM on Your Phone}, author = {MiniCPM Team, OpenBMB}, year = {2025}, url = {https://huggingface.co/openbmb/MiniCPM-o-4.5}}88plug AI Lab ships compressed-tensors quantizations for native vLLM v0.21.0+ deployment.
This release: Provisional tier — datafree RTN (weight-only rounding, no calibration corpus). A gold AutoRound re-quant is scheduled; 88plug architecture forbids new provisional W4A16 uploads.
Browse all releases → huggingface.co/88plug
@misc{minicpmo, title = {MiniCPM-o: A GPT-4o Level Multimodal LLM on Your Phone}, author = {MiniCPM Team, OpenBMB}, year = {2025}, url = {https://huggingface.co/openbmb/MiniCPM-o-4.5}}| Kept BF16 |
| vision encoder, audio encoder, TTS components |
| Disk size | ~9 GB |
| Min GPU | 1× RTX 3090 24GB |
| — |
| — |
| — |
| llama.cpp b9297 | Q8_0 GGUF | 1 | 32k | — | — | — | — |
| llama.cpp b9297 | IQ4_XS GGUF | 1 | 32k | — | — | — | — |
| Near-lossless; same TTS limitation |
| BF16 baseline | None | ~18 GB | 1× A100 40GB | Reference; requires high-VRAM GPU |