GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ISTA-DASLab
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
ISTA-DASLab
Model Tree
Qwen/Qwen3-30B-A3B
Input Modalities
Output Modalities
Supported Functionality
This checkpoint does not load in upstream vLLM. The MoESQ repository installs a patched
vLLM v0.30.0 (the paired48_nvfp4 MoE backend and sparse-storage loader) together with the
kernels. It needs an NVIDIA Blackwell SM100 (B200, GB200) or SM120 (RTX 5090, RTX PRO 6000) GPU and a
CUDA toolkit >= 12.8; SM103 (B300) and SM121 (DGX Spark) are not supported.
bash
git clone --recurse-submodules https://github.com/IST-DASLab/MoESQ.git && cd MoESQbash integrations/vllm/install.sh && source .venv-vllm/bin/activate vllm serve ISTA-DASLab/Qwen3-30B-A3B-P48NVFP4-MoESQ # 1x B200# multi-GPU: add --tensor-parallel-size N --enable-expert-parallelvLLM selects the backend automatically.
The six tasks are ARC-Challenge (25-shot), HellaSwag (10-shot), MMLU (5-shot), TruthfulQA-MC2 (0-shot), Winogrande (5-shot) and GSM8K (5-shot, greedy). Scores use lm-evaluation-harness with the full test sets, mean ± sd over few-shot seeds 1234, 0 and 1, served with the MoESQ vLLM integration on 1× B200. Recovery is relative to dense.
| Model | Avg | Recovery vs dense |
|---|---|---|
| Qwen3-30B-A3B (dense, BF16) | 73.43 ± 0.25 | — |
| This model (MoESQ) | 66.41 ± 0.15 | 90.4 % |
| OBR (same W4A4 paired-4:8 target) | 62.12 ± 0.33 | 84.6 % |
| SparseGPT + GPTQ (same target) | 61.71 ± 0.09 | 84.0 % |
Per task, this model: ARC-C 60.78 · GSM8K 82.36 · HellaSwag 64.66 · MMLU 71.13 · TruthfulQA-MC2 51.73 · Winogrande 67.80.
Gate and up projections carry separate NVFP4 global scales. The MoESQ vLLM patch folds their ratio into the block scales at load time. A vLLM build without that fold applies the gate scale to both projections; this checkpoint's weights scored 65.20 ± 0.08 on such a build under the same protocol.
| Field | Value |
|---|---|
| Weights | NVFP4 (E2M1), one FP8-E4M3 scale per 32 dense K elements, FP32 per-tensor global scale |
| Activations | NVFP4, dynamic per-32 group scales, FP32 per-tensor global scale stored per expert linear |
| Sparsity | Paired 4:8 along K: in every 8 consecutive elements (4 pairs), exactly 2 pairs are kept |
| Compressed layers | Routed MoE experts (gate_proj, up_proj, down_proj) in all 48 layers |
| Left in BF16 | lm_head, embeddings, attention, norms, router (mlp.gate) |
The scale group is 32 because the Blackwell sparse NVFP4 MMA (SM100 and SM120) requires one scale per 32 dense K elements, which is 16 surviving elements after the 4:8 prune.
paired48_sparse, pair-bitmask v1)Packed NVFP4 stores two adjacent K elements per byte, so one pair is one byte. A paired-4:8 weight therefore has exactly 2 nonzero bytes in every 4-byte chunk. Each routed-expert linear stores:
| Tensor | dtype / shape | Contents |
|---|---|---|
weight_sparse_packed | uint8 [out, K/4] | the 2 kept bytes of every 4, in K order |
weight_sparse_mask | uint8 [out, K/16] | 4 bits per 4-byte chunk (exactly 2 set; bit i means byte i is kept); the low nibble is the lower-K chunk |
weight_scale | float8_e4m3 [out, K/32] | block scales (linear layout) |
quantization_config carries "paired48_sparse": {"layout": "pair-bitmask", "version": 1}.
The encoding is lossless: converting to and from dense NVFP4 is exact, and every tensor in
this upload was checked with a round trip. At load time vLLM rebuilds each layer's dense
packed weight and compresses it into the kernel's own layout.
The full MoESQ config for this run is in moe_sq_config.yaml.
gate_weight_exponent = 2).bibtex
@misc{lee2026hardwarenativejointsparsequantizationtrillionscale, title={Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts}, author={Kwanhee Lee and Namhoon Lee and Dan Alistarh}, year={2026}, eprint={2610.02241}, archivePrefix={arXiv}, primaryClass={cs.AR}, doi={10.48550/arXiv.2610.02241}, url={https://arxiv.org/abs/2610.02241},}For questions, open a discussion here or contact kwanhee.lee@postech.ac.kr.
| Format | compressed-tensors, nvfp4-pack-quantized, with a paired48_sparse storage marker |
weight_global_scale, input_global_scale | float32 | per-tensor global scales |