GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ISTA-DASLab
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ISTA-DASLab
Model Tree
moonshotai/Kimi-K2.5
Input Modalities
Output Modalities
Supported Functionality
| Field | Value |
|---|---|
| Weight dtype | NVFP4 (E2M1) |
| Weight group size | 16 |
| Weight scale dtype | FP8 E4M3, per-group |
| Weight global scale | FP32, per-tensor |
| Activation dtype | NVFP4 (E2M1), dynamic: "local" |
| Activation group size | 16 |
| Activation group scale dtype | FP8 E4M3, per-group (computed per batch at runtime) |
| Activation global scale | FP32, per-tensor (stored in checkpoint, per expert linear) |
| Sparsity | Paired 4:8 (NVIDIA Blackwell) |
| Quantized + sparsified layers | Non-shared MoE experts (gate_proj, up_proj, down_proj) |
| Uncompressed | lm_head, self_attn.*, shared_experts.*, router, embeddings |
| Format | compressed-tensors (NVFP4PackedCompressor) |
Paired-4:8 sparsity. Every 8 contiguous elements form 4 pairs of 2; exactly 2 of the 4 pairs are nonzero.
The zeroed positions are stored as FP4 zero codes inside weight_packed, so the sparsity structure is implicit — there is no separate bitmask tensor in the file.
Per-linear keys:
weight_packed — FP4 values, full K dimensionweight_scale — FP8 E4M3 per-16 group weight scalesweight_global_scale — FP32 per-tensor weight global scaleinput_global_scale — FP32 per-tensor activation global scaleThe weight format is a standard NVFP4 checkpoint — any inference stack with compressed-tensors NVFP4 support loads it directly. The vLLM flags below cover Kimi-K2.5-specific runtime needs (custom model code, chat-template parsers).
The recipe below follows the upstream vLLM guide for Kimi-K2.5: https://recipes.vllm.ai/moonshotai/Kimi-K2.5. Refer to that page for advanced options (long context, prefix caching, structured output) and version-specific notes. Tested on 4xB200.
bash
uv pip install -U vllm --torch-backend=auto VLLM_USE_FLASHINFER_MOE_FP4=1 vllm serve ISTA-DASLab/Kimi-K2.5-P48-NVFP4-W4A4-Preview \ --tensor-parallel-size 4 \ --mm-encoder-tp-mode data \ --trust-remote-code \ --tool-call-parser kimi_k2 \ --reasoning-parser kimi_k2Flag notes:
VLLM_USE_FLASHINFER_MOE_FP4=1 — enables FlashInfer's NVFP4×NVFP4 grouped GEMM path for MoE experts.Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.
All evaluations run with lm-evaluation-harness v0.4.11 against a vLLM 0.21.0 server on 4× B200 with VLLM_USE_FLASHINFER_MOE_FP4=1.
| Benchmark | Setup | Base (BF16) | SparseGPT + GPTQ one-shot | Ours | Δ vs base |
|---|---|---|---|---|---|
| ARC-Challenge | acc_norm, 25-shot | 74.23 | 62.54 | 68.43 | −5.80 |
| HellaSwag | acc_norm, 10-shot | 91.86 | 84.90 | 88.70 | −3.16 |
| MMLU | acc, 5-shot | 89.57 | 81.83 | 85.45 | −4.12 |
| TruthfulQA | mc2, 0-shot | 62.54 | 55.83 | 60.33 | −2.21 |
| Winogrande | acc, 5-shot | 82.48 | 79.95 | 83.35 | +0.87 |
| GSM8K | exact_match, 5-shot | 94.39 | 79.98 | 87.79 | −6.60 |
| Average | 82.51 | 74.17 | 79.01 | −3.50 |
Recovery: 79.01 / 82.51 = 95.76% of base-model average accuracy.
SparseGPT + GPTQ one-shot baseline. Reference point at the same compression target: SparseGPT picks the paired-4:8 mask, GPTQ quantizes the masked weights to NVFP4 (89.89% recovery, no activation quant).
This preview ships the dense NVFP4 storage format with paired-4:8 zeros embedded as FP4 zero codes. That keeps the checkpoint compatible with current compressed-tensors and vLLM loaders out of the box, but leaves two opportunities on the table:
ElementE) that CUTLASS / cuSPARSELt sparse-NVFP4 kernels expect. This cuts the on-disk and HBM footprint of the expert weights roughly in half. The paired-4:8 mask is structurally preserved in the current dense FP4 codes, so the conversion can run as an offline post-processing step on top of the released checkpoint.Both are tracked for the next release, not this preview.
For questions or open a discussion on this preview, please fill free to reach out to kwanhee.lee@postech.ac.kr.
uv pip install -U vllm --torch-backend=auto VLLM_USE_FLASHINFER_MOE_FP4=1 vllm serve ISTA-DASLab/Kimi-K2.5-P48-NVFP4-W4A4-Preview \ --tensor-parallel-size 4 \ --mm-encoder-tp-mode data \ --trust-remote-code \ --tool-call-parser kimi_k2 \ --reasoning-parser kimi_k2