GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ISTA-DASLab
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ISTA-DASLab
Model Tree
moonshotai/Kimi-K2.5
Input Modalities
Output Modalities
Supported Functionality
| Field | Value |
|---|---|
| Weight dtype | NVFP4 (E2M1) |
| Weight group size | 16 |
| Scale dtype | FP8 E4M3, per-group |
| Global scale | FP32, per-tensor |
| Sparsity | Paired 4:8 (NVIDIA Blackwell) |
| Quantized + sparsified layers | Non-shared MoE experts (gate_proj, up_proj, down_proj) |
| Uncompressed | lm_head, self_attn.*, shared_experts.*, router, embeddings |
| Format | compressed-tensors (NVFP4PackedCompressor) |
Paired-4:8 sparsity. Every 8 contiguous elements form 4 pairs of 2; exactly 2 of the 4 pairs are nonzero. The zeroed positions are stored as FP4 zero codes inside weight_packed, so the sparsity structure is implicit — there is no separate bitmask tensor in the file.
Per-linear keys:
weight_packed — FP4 values, full K dimensionweight_scale — FP8 E4M3 per-16 group scalesweight_global_scale — FP32 per-tensor scaleThe recipe below follows the upstream vLLM guide for Kimi-K2.5: https://recipes.vllm.ai/moonshotai/Kimi-K2.5. Refer to that page for advanced options (long context, prefix caching, structured output) and version-specific notes.
bash
uv pip install -U vllm --torch-backend=auto vllm serve ISTA-DASLab/Kimi-K2.5-P48NVFP4-Preview \ --tensor-parallel-size 4 \ --mm-encoder-tp-mode data \ --trust-remote-code \ --tool-call-parser kimi_k2 \ --reasoning-parser kimi_k2Flag notes:
--tensor-parallel-size 4 — fits on 4× B200 (192 GB HBM3e each). Bump on smaller GPUs.--trust-remote-code — required: Kimi-K2.5 ships custom modeling code.--tool-call-parser kimi_k2, --reasoning-parser kimi_k2 — enable K2.5's tool-use and reasoning chat templates.--mm-encoder-tp-mode data — multimodal encoder TP mode; harmless for text-only inference.Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.
All evaluations run with lm-evaluation-harness v0.4.11 against a vLLM 0.21.0 server on 4× B200.
| Benchmark | Setup | Base (BF16) | SparseGPT + GPTQ one-shot | This release | Δ (vs base) |
|---|---|---|---|---|---|
| ARC-Challenge | acc_norm, 25-shot | 74.23 | 62.54 | 69.20 | −5.03 |
| HellaSwag | acc_norm, 10-shot | 91.86 | 84.90 | 88.71 | −3.15 |
| MMLU | acc, 5-shot | 89.57 | 81.83 | 85.49 | −4.08 |
| TruthfulQA | mc2, 0-shot | 62.54 | 55.83 | 59.66 | −2.88 |
| Winogrande | acc, 5-shot | 82.48 | 79.95 | 83.90 | +1.42 |
| GSM8K | exact_match, 5-shot | 94.39 | 79.98 | 88.93 | −5.46 |
| Average | 82.51 | 74.17 | 79.31 | −3.20 |
Recovery: 79.31 / 82.51 = 96.12% of base-model average accuracy.
SparseGPT + GPTQ one-shot baseline. Reference point at the same compression target: SparseGPT picks the paired-4:8 mask, GPTQ quantizes the masked weights to NVFP4 (89.89% recovery).
For questions or open a discussion on this preview, please fill free to reach out to kwanhee.lee@postech.ac.kr.
uv pip install -U vllm --torch-backend=auto vllm serve ISTA-DASLab/Kimi-K2.5-P48NVFP4-Preview \ --tensor-parallel-size 4 \ --mm-encoder-tp-mode data \ --trust-remote-code \ --tool-call-parser kimi_k2 \ --reasoning-parser kimi_k2