GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
xanguera
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
xanguera
Model Tree
pipecat-ai/phonellm-alpha-1
Input Modalities
Output Modalities
Supported Functionality
Neither official PhoneLLM Alpha 1 release fits 3x RTX 3090: they're Ampere (no NVFP4 support), and the BF16 checkpoint alone is ~59GB — too tight for KV cache headroom across only 3x24GB cards. This checkpoint quantizes the MoE experts and general Linear layers to INT8 (channel-wise symmetric weights, dynamic per-token symmetric int8 activations) while keeping attention/Mamba projections, norms, gates, and lm_head in BF16, matching NVIDIA's own precedent for this architecture.
Coherent generations, and correct <tool_call> emission with accurate parameter extraction, confirmed via a say/do test. This checkpoint has not been run against PhoneBench — no formal accuracy comparison against the BF16 or NVFP4 releases is published here.
bash
vllm serve xanguera/phonellm-alpha-1-int8 \--trust-remote-code \--tensor-parallel-size 2 \--max-model-len 8192 \--reasoning-parser-plugin nano_v3_reasoning_parser.py \--reasoning-parser nano_v3
Important: the reasoning-parser plugin is required. Without it, the model's <think>...</think> reasoning text leaks directly into the visible response content (a literal </think> shows up mid-sentence) instead of landing in a separate reasoning field — setting enable_thinking: false via chat_template_kwargs alone is not sufficient. This only shows up when testing the actual served HTTP endpoint; vLLM's offline batch API doesn't exercise this code path.
Recommended inference settings match the base model: temperature=0, thinking disabled.
This model is a quantization of pipecat-ai/phonellm-alpha-1, released under the BSD 2-Clause License, itself a derivative of NVIDIA Nemotron 3 Nano 30B-A3B, licensed under the NVIDIA Nemotron Open Model License.
Per Section 3 (Redistribution) of the NVIDIA Nemotron Open Model License, this repository includes a copy of that license and retains the NVIDIA copyright and attribution notices carried in the bundled model code. This quantization is released under the BSD 2-Clause License, consistent with the terms under which the base model was released; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.
Built on PhoneLLM Alpha 1 by the Pipecat team at Daily, itself built on NVIDIA Nemotron 3 Nano 30B-A3B.