GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
LeoHyperlink
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
LeoHyperlink
Model Tree
InternScience/Sage-A1.5
Input Modalities
Output Modalities
Supported Functionality
| Property | Specification |
|---|---|
| Base checkpoint | InternScience/Sage-A1.5, text use (--language-model-only) |
| Parameters | ~36B (Qwen3_5MoeForConditionalGeneration: 40 layers, 256 experts, 8 active) |
| Architecture | MoE decoder (hidden 2048, tied word embeddings), plus MTP and a vision tower (both BF16, text serve unused) |
| Context length | 8192 tokens at serve time (--max-model-len 8192) |
| Quantization | W4A16 compressed-tensors, pack-quantized (int4 symmetric, group 128, no activation quantization) |
| License | Apache-2.0 (follows the base model) |
Serve with vLLM (XPU fork on Intel GPUs; upstream CUDA vLLM also reads compressed-tensors):
bash
pip install "vllm>=0.32" # XPU fork (vllm-xpu) on Intel GPUsvllm serve LeoHyperlink/Sage-1.5-W4A16 --trust-remote-code \ --max-model-len 8192 --gpu-memory-utilization 0.9 --language-model-onlyTransformers can load it for offline use (compressed-tensors + llmcompressor packages), but expect CPU-class
throughput there; the fused path above is the vLLM one.
Sampling parameters (the validated settings — all numbers below used these):
temperature=0, thinking disabled via
--default-chat-template-kwargs '{"enable_thinking": false}'.Single-stream decode, 128 streamed tokens (temperature 0, prompt: "Explain why the sky is blue in one short paragraph"),
OpenAI-compatible /v1/chat/completions streaming, tokens counted after time-to-first-token:
| Checkpoint | Decode | TTFT | Backend |
|---|---|---|---|
| This repo (Sage W4A16-CT) | 104.3 t/s | 0.06 s | XPUwNa16LinearKernel (fused) |
| intel-ai/Qwen3.6-35B-A3B-MXFP4-Mixed-FP8Attn-CT-AutoRound (same box and protocol, reference) | 60.0 t/s | 0.08 s | XPUExpertsMxFp4 + XPUMxFp8LinearKernel |
Correctness spot-checks on this checkpoint: 8/8 concurrent prompts coherent, plus an 80-request 4-way-concurrency soak
with zero garbage outputs. A/B notes: FP8 KV cache + prefix caching did not move single-stream decode
(102.4 vs 104.3 t/s, noise); the tuner-suggested VLLM_XPU_LM_HEAD_INT8=1 does not exist in the serving build below
and was rejected.
Three sound-harness proxy scores — every one measured on the same live serve as the speed numbers above (thinking off, temperature 0):
| Setting | Value |
|---|---|
| Tool | Intel auto-round 0.15.1, auto_round:llm_compressor (CT pack-quantized) export |
| Weights | int4 symmetric, group 128 |
| Activations | none quantized (fp16 compute) |
| Calibration | pile-10k default (128 samples x 2048 tokens), auto-round default optimizer settings |
| torch.compile | explicitly disabled (Triton make_ttgir inductor failure on XPU) |
| Quantized | 31030 / 31181 layers |
| Kept in BF16 | lm_head, per-layer mlp.shared_expert_gate, visual tower |
| MTP | NOT quantized: 785 raw BF16 tensors passed through to model_extra_tensors.safetensors |
| Quant time | ~8.7 h on one Intel Arc Pro B70 |
vllm-xpu-kernels with libgrouped_gemm_xe_2 and fused_moe_interface), torch 2.x +XPU, oneCCL--trust-remote-code --max-model-len 8192 --gpu-memory-utilization 0.9 --language-model-only --kv-cache-dtype fp8 --enable-prefix-caching
(text-only; the visual tower ships BF16 and was not exercised), GPU pinned via ONEAPI_DEVICE_SELECTOR=level_zero:N.
Note: ZE_AFFINITY_MASK=0.0 crashes engine init on this build (int parsing of "0.0"); this build binds localhost only.local-chat-completions against the live endpoint (8-way concurrency),
IFStruct upstream harness against the same endpoint.model-0000[1-4]-of-00004.safetensors (20 GB) + model.safetensors.index.json — quantized weightsmodel_extra_tensors.safetensors (1.7 GB) — raw BF16 MTP/vision passthrough tensors, kept for faithfulness.
vLLM 0.32.0 cannot load them alongside the CT main checkpoint; text-only serving does not need them.quantization_config.json — compressed-tensors pack-quantized spec (int4 symmetric g128, no activation quant)config.json, tokenizer files, preprocessor/processor configs, chat_template.jinja, generation_config.json.eval_results/gsm8k.yaml, .eval_results/ifstruct.yaml — machine-readable scores (Hub eval-results schema)lm_head
sustained-concurrency corruption trap on some builds, so soak your own workload before production.No new training was done for this quant — if you use it, cite the base model (InternScience/Sage-A1.5) and link back to this repo for the quantization recipe and serving numbers.
pip install "vllm>=0.32" # XPU fork (vllm-xpu) on Intel GPUsvllm serve LeoHyperlink/Sage-1.5-W4A16 --trust-remote-code \ --max-model-len 8192 --gpu-memory-utilization 0.9 --language-model-only