GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
stuchapin
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
stuchapin
Model Tree
InternScience/Sage-A1.5
Input Modalities
Output Modalities
Supported Functionality
20.3 GB total checkpoint size on disk (~18.35 GiB model weights in VRAM), leaving room for up to ~252,000 tokens of FP8 KV cache on a single 24 GB RTX 3090.mlp.gate (the 256-expert MoE router) and shared_expert_gate across all 40 layers are preserved in 16-bit (BF16) so expert routing fidelity matches the unquantized model.0.15.1 (100 iterations, seqlen=2048, 256 multi-turn agentic web/extraction calibration samples) using symmetric int4 with group_size=128 and auto_round:auto_gptq packing (the same recipe family as Intel's Qwen3.6-35B-A3B-int4-mixed-AutoRound).model.safetensors.index.json 1-to-1.| Parameter | Value |
|---|---|
| Base Model | InternScience/Sage-A1.5 (35B-A3B hybrid linear/full attention MoE) |
| Quantization Method | Intel AutoRound 0.15.1 (quant_method: auto-round) |
| Precision / Packing | W4A16 symmetric int4, group_size: 128, auto_round:auto_gptq |
| Unquantized Layers (16-bit) | model.layers.*.mlp.gate (MoE router), model.layers.*.mlp.shared_expert_gate, norms & embeddings |
| Calibration | 256 samples, seqlen: 2048, iters: 100 on multi-turn tool-use & structured extraction trajectories |
| Checkpoint Size | 20.3 GB across 6 .safetensors shards (~18.35 GiB in GPU memory) |
Because quant_method: auto-round uses model.language_model.* tensor paths with the original Qwen3_5MoeForConditionalGeneration config, pass --language-model-only when serving text/agent workloads in vLLM:
bash
vllm serve stuchapin/Sage-A1.5-W4A16-AutoRound \ --served-model-name sage-a1.5 \ --language-model-only \ --gpu-memory-utilization 0.93 \ --max-model-len 32768 \ --max-num-seqs 16 \ --kv-cache-dtype fp8 \ --mamba-ssm-cache-dtype float16 \ --enable-prefix-caching \ --mamba-cache-mode align \ --async-scheduling \ --max-num-batched-tokens 2048 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coderIf running inside Docker on WSL2:
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (WSL2's paravirtualized CUDA driver does not support expandable_segments:True).-e VLLM_WSL2_ENABLE_PIN_MEMORY=1 and --shm-size=8g.--gpu-memory-utilization 0.93 with --kv-cache-dtype fp8 and --mamba-ssm-cache-dtype float16, vLLM allocates ~252,000 tokens of KV cache.Measured on a single RTX 3090 (vLLM 0.28, --kv-cache-dtype fp8, --enable-prefix-caching --mamba-cache-mode align) on ~12,000-token structured web-extraction & tool-calling prompts (300 output tokens, thinking disabled):
Concurrent Requests (N) | Dense 27B W4A16 Baseline (jobs/min) | Sage-A1.5-W4A16-AutoRound (jobs/min) | Speedup | p95 Latency (Baseline vs. Sage-A1.5) |
|---|---|---|---|---|
| 1 | 8.1 | 23.7 | 2.9× | 7.4 s vs. 2.5 s |
| 4 | 9.4 | 47.3 | 5.0× | 25.7 s vs. 5.1 s |
| 8 | 9.4 | 59.7 | 6.4× | 51.2 s vs. 8.0 s |
| 16 | 9.3 | 68.2 | 7.3× | 102.9 s vs. 14.1 s |
Like the base InternScience/Sage-A1.5 checkpoint, thinking mode is controlled via chat_template_kwargs:
python
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="sage-a1.5", messages=[{"role": "user", "content": "Summarize the key differences between GQA and MLA."}], extra_body={"chat_template_kwargs": {"enable_thinking": False}},)print(response.choices[0].message.content)This model is a quantized derivative of InternScience/Sage-A1.5 and is released under the Apache-2.0 license. Please refer to the base model repository and arXiv:2610.28578 for full details on the Sage-A1.5 training recipe and architecture.
vllm serve stuchapin/Sage-A1.5-W4A16-AutoRound \ --served-model-name sage-a1.5 \ --language-model-only \ --gpu-memory-utilization 0.93 \ --max-model-len 32768 \ --max-num-seqs 16 \ --kv-cache-dtype fp8 \ --mamba-ssm-cache-dtype float16 \ --enable-prefix-caching \ --mamba-cache-mode align \ --async-scheduling \ --max-num-batched-tokens 2048 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coderfrom openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="sage-a1.5", messages=[{"role": "user", "content": "Summarize the key differences between GQA and MLA."}], extra_body={"chat_template_kwargs": {"enable_thinking": False}},)print(response.choices[0].message.content)