GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ISTA-DASLab
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
ISTA-DASLab
Model Tree
moonshotai/Kimi-K2.5
Input Modalities
Output Modalities
Supported Functionality
This checkpoint does not load in upstream vLLM. The MoESQ repository installs a patched
vLLM v0.30.0 (the paired48_nvfp4 MoE backend and sparse-storage loader) together with the
kernels. It needs an NVIDIA Blackwell SM100 (B200, GB200) or SM120 (RTX 5090, RTX PRO 6000) GPU and a
CUDA toolkit >= 12.8; SM103 (B300) and SM121 (DGX Spark) are not supported.
bash
git clone --recurse-submodules https://github.com/IST-DASLab/MoESQ.git && cd MoESQbash integrations/vllm/install.sh && source .venv-vllm/bin/activate # 8x B200, tensor + expert parallelvllm serve ISTA-DASLab/Kimi-K2.5-P48NVFP4-MoESQ --trust-remote-code \ --tensor-parallel-size 8 --enable-expert-parallel --kv-cache-dtype bfloat16 # or data + expert parallel (requires DeepEP)vllm serve ISTA-DASLab/Kimi-K2.5-P48NVFP4-MoESQ --trust-remote-code \ --data-parallel-size 8 --enable-expert-parallel --all2all-backend deepep_low_latency \ --kv-cache-dtype bfloat16 --max-num-batched-tokens 256vLLM selects the backend automatically. See Serving with vLLM for other parallel layouts.
The six tasks are ARC-Challenge (25-shot), HellaSwag (10-shot), MMLU (5-shot), TruthfulQA-MC2 (0-shot), Winogrande (5-shot) and GSM8K (5-shot, greedy). Scores use lm-evaluation-harness with the full test sets, as mean ± std over 3 few-shot seeds (1234, 0, 1). Recovery is computed per seed against dense, then averaged.
| Model | Avg | Recovery vs dense |
|---|---|---|
| Kimi-K2.5 (dense) | 82.30 ± 0.18 | — |
| This model (MoESQ) | 79.25 ± 0.07 | 96.30 ± 0.23 % |
| SparseGPT + GPTQ (same W4A4 paired-4:8 target) | 73.88 ± 0.18 | 89.77 ± 0.34 % |
| OBR (same target) | 74.80 ± 0.11 | 90.89 ± 0.24 % |
| JSQ (same target) | 67.84 ± 0.08 | 82.43 ± 0.26 % |
The metric is average pass@1 over N samples per problem (AIME25 N=10, GPQA-Diamond N=5, MATH-500 N=5). Sampling uses temperature 1.0 and top_p 0.95, with up to 65,536 new tokens. Both models were run with this same protocol. ± is the lighteval standard error across problems; it was not recorded for the dense run.
| Benchmark | Kimi-K2.5 (dense) | This model (MoESQ) | Recovery |
|---|---|---|---|
| AIME25 | 95.67 | 84.67 ± 5.27 | 88.5 % |
| GPQA-Diamond | 89.49 | 76.46 ± 2.51 | 85.4 % |
| MATH-500 | 96.36 | 96.28 ± 0.74 | 99.9 % |
| Field | Value |
|---|---|
| Weights | NVFP4 (E2M1), one FP8-E4M3 scale per 32 dense K elements, FP32 per-tensor global scale |
| Activations | NVFP4, dynamic per-32 group scales, FP32 per-tensor global scale stored per expert linear |
| Sparsity | Paired 4:8 along K: in every 8 consecutive elements (4 pairs), exactly 2 pairs are kept |
| Compressed layers | Routed MoE experts (gate_proj, up_proj, down_proj) in layers 1–60 |
| Left in BF16 | lm_head, embeddings, attention, norms, router (mlp.gate), shared experts, layer-0 dense MLP, vision tower / projector |
The scale group is 32 because the Blackwell sparse NVFP4 MMA (SM100 and SM120) requires one scale per 32 dense K elements, which is 16 surviving elements after the 4:8 prune.
paired48_sparse, pair-bitmask v1)Packed NVFP4 stores two adjacent K elements per byte, so one pair is one byte. A paired-4:8 weight therefore has exactly 2 nonzero bytes in every 4-byte chunk. Each routed-expert linear stores:
| Tensor | dtype / shape | Contents |
|---|---|---|
weight_sparse_packed | uint8 [out, K/4] | the 2 kept bytes of every 4, in K order |
weight_sparse_mask | uint8 [out, K/16] | 4 bits per 4-byte chunk (exactly 2 set; bit i means byte i is kept); the low nibble is the lower-K chunk |
weight_scale | float8_e4m3 [out, K/32] | block scales (linear layout) |
quantization_config carries "paired48_sparse": {"layout": "pair-bitmask", "version": 1}.
If a 4-byte chunk has fewer than 2 nonzero bytes, its lowest-index zero bytes are marked as
kept. The encoding is lossless: converting to and from dense NVFP4 is exact, and every tensor
in this upload was checked with a round trip. At load time vLLM rebuilds each layer's dense
packed weight and compresses it into the kernel's own layout. Load-time memory is therefore
the sparse checkpoint plus one dense layer, not the whole dense model.
The full MoESQ config for this run is in moe_sq_config.yaml.
gate_weight_exponent = 2).bibtex
@article{moesq, title = {TODO}, author = {TODO}, journal = {arXiv preprint arXiv:TODO}, year = {TODO}}For questions, open a discussion here or contact kwanhee.lee@postech.ac.kr.
| Format | compressed-tensors, nvfp4-pack-quantized, with a paired48_sparse storage marker |
weight_global_scale, input_global_scale | float32 | per-tensor global scales |
git clone --recurse-submodules https://github.com/IST-DASLab/MoESQ.git && cd MoESQbash integrations/vllm/install.sh && source .venv-vllm/bin/activate # 8x B200, tensor + expert parallelvllm serve ISTA-DASLab/Kimi-K2.5-P48NVFP4-MoESQ --trust-remote-code \ --tensor-parallel-size 8 --enable-expert-parallel --kv-cache-dtype bfloat16 # or data + expert parallel (requires DeepEP)vllm serve ISTA-DASLab/Kimi-K2.5-P48NVFP4-MoESQ --trust-remote-code \ --data-parallel-size 8 --enable-expert-parallel --all2all-backend deepep_low_latency \ --kv-cache-dtype bfloat16 --max-num-batched-tokens 256@article{moesq, title = {TODO}, author = {TODO}, journal = {arXiv preprint arXiv:TODO}, year = {TODO}}