GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
ISTA-DASLab
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
ISTA-DASLab
Model Tree
moonshotai/Kimi-K2.5
Input Modalities
Output Modalities
Supported Functionality
| Field | Value |
|---|---|
| Weight dtype | NVFP4 (E2M1) |
| Weight group size | 16 |
| Weight scale dtype | FP8 E4M3, per-group |
| Weight global scale | FP32, per-tensor |
| Activation dtype | NVFP4 (E2M1), dynamic: "local" |
| Activation group size | 16 |
| Activation group scale dtype | FP8 E4M3, per-group (computed per batch at runtime) |
| Activation global scale | FP32, per-tensor (stored in checkpoint, per expert linear) |
| Sparsity | Paired 4:8 (NVIDIA Blackwell) |
| Quantized + sparsified layers | Non-shared MoE experts (gate_proj, up_proj, down_proj) |
| Uncompressed | lm_head, self_attn.*, shared_experts.*, router, embeddings |
| Format | compressed-tensors (NVFP4PackedCompressor) |
Paired-4:8 sparsity. Every 8 contiguous elements form 4 pairs of 2; exactly 2 of the 4 pairs are nonzero.
The zeroed positions are stored as FP4 zero codes inside weight_packed, so the sparsity structure is implicit — there is no separate bitmask tensor in the file.
Per-linear keys:
weight_packed — FP4 values, full K dimensionweight_scale — FP8 E4M3 per-16 group weight scalesweight_global_scale — FP32 per-tensor weight global scaleinput_global_scale — FP32 per-tensor activation global scaleThe weight format is a standard NVFP4 checkpoint — any inference stack with compressed-tensors NVFP4 support loads it directly. The vLLM flags below cover Kimi-K2.5-specific runtime needs (custom model code, chat-template parsers).
The recipe below follows the upstream vLLM guide for Kimi-K2.5: https://recipes.vllm.ai/moonshotai/Kimi-K2.5. Refer to that page for advanced options (long context, prefix caching, structured output) and version-specific notes. Tested on 4xB200.
bash
uv pip install -U vllm --torch-backend=autoVLLM_USE_FLASHINFER_MOE_FP4=1 vllm serve ISTA-DASLab/Kimi-K2.5-P48-NVFP4-W4A4-Preview \--tensor-parallel-size 4 \--mm-encoder-tp-mode data \--trust-remote-code \--tool-call-parser kimi_k2 \--reasoning-parser kimi_k2
Flag notes:
VLLM_USE_FLASHINFER_MOE_FP4=1 — enables FlashInfer's NVFP4×NVFP4 grouped GEMM path for MoE experts.Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.
All evaluations run with lm-evaluation-harness v0.4.11 against a vLLM 0.21.0 server on 4× B200 with VLLM_USE_FLASHINFER_MOE_FP4=1.
| Benchmark | Setup | Base (BF16) | SparseGPT + GPTQ one-shot | Ours | Δ vs base |
|---|---|---|---|---|---|
| ARC-Challenge | acc_norm, 25-shot | 74.23 | 62.54 | 68.43 | −5.80 |
| HellaSwag | acc_norm, 10-shot | 91.86 | 84.90 | 88.70 | −3.16 |
| MMLU | acc, 5-shot |
Recovery: 79.01 / 82.51 = 95.76% of base-model average accuracy.
SparseGPT + GPTQ one-shot baseline. Reference point at the same compression target: SparseGPT picks the paired-4:8 mask, GPTQ quantizes the masked weights to NVFP4 (89.89% recovery, no activation quant).
This preview ships the dense NVFP4 storage format with paired-4:8 zeros embedded as FP4 zero codes. That keeps the checkpoint compatible with current compressed-tensors and vLLM loaders out of the box, but leaves two opportunities on the table:
ElementE) that CUTLASS / cuSPARSELt sparse-NVFP4 kernels expect. This cuts the on-disk and HBM footprint of the expert weights roughly in half. The paired-4:8 mask is structurally preserved in the current dense FP4 codes, so the conversion can run as an offline post-processing step on top of the released checkpoint.Both are tracked for the next release, not this preview.
For questions or open a discussion on this preview, please fill free to reach out to kwanhee.lee@postech.ac.kr.
| 89.57 |
| 81.83 |
| 85.45 |
| −4.12 |
| TruthfulQA | mc2, 0-shot | 62.54 | 55.83 | 60.33 | −2.21 |
| Winogrande | acc, 5-shot | 82.48 | 79.95 | 83.35 | +0.87 |
| GSM8K | exact_match, 5-shot | 94.39 | 79.98 | 87.79 | −6.60 |
| Average | 82.51 | 74.17 | 79.01 | −3.50 |