GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ISTA-DASLab
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ISTA-DASLab
Model Tree
Qwen/Qwen3.5-397B-A17B
Input Modalities
Output Modalities
Supported Functionality
This checkpoint does not load in upstream vLLM. The MoESQ repository installs a patched
vLLM v0.30.0 (the paired48_nvfp4 MoE backend and sparse-storage loader) together with the
kernels. It needs an NVIDIA Blackwell SM100 (B200, GB200) or SM120 (RTX 5090, RTX PRO 6000) GPU and a
CUDA toolkit >= 12.8; SM103 (B300) and SM121 (DGX Spark) are not supported.
bash
git clone --recurse-submodules https://github.com/IST-DASLab/MoESQ.git && cd MoESQbash integrations/vllm/install.sh && source .venv-vllm/bin/activate # 4x B200, tensor + expert parallelvllm serve ISTA-DASLab/Qwen3.5-397B-A17B-P48NVFP4-MoESQ \ --tensor-parallel-size 4 --enable-expert-parallelvLLM selects the backend automatically.
The six tasks are ARC-Challenge (25-shot), HellaSwag (10-shot), MMLU (5-shot), TruthfulQA-MC2 (0-shot), Winogrande (5-shot) and GSM8K (5-shot, greedy). Scores use lm-evaluation-harness with the full test sets, as mean ± std over 3 few-shot seeds (1234, 0, 1). Recovery is computed per seed against dense, then averaged.
| Model | Avg | Recovery vs dense |
|---|---|---|
| Qwen3.5-397B-A17B (dense) | 80.24 ± 0.14 | — |
| This model (MoESQ) | 75.07 ± 0.21 | 93.55 % |
| SparseGPT + GPTQ (same W4A4 paired-4:8 target) | 70.61 ± 0.25 | 88.00 % |
| OBR (same target) | 73.42 ± 0.36 | 91.50 % |
| JSQ (same target) | 62.96 ± 0.14 | 78.46 % |
Most of the gap to the baselines comes from GSM8K, which moves by about ±1.3 points across few-shot seeds. Reasoning benchmarks have not been run for this model yet.
| Field | Value |
|---|---|
| Weights | NVFP4 (E2M1), one FP8-E4M3 scale per 32 dense K elements, FP32 per-tensor global scale |
| Activations | NVFP4, dynamic per-32 group scales, FP32 per-tensor global scale stored per expert linear |
| Sparsity | Paired 4:8 along K: in every 8 consecutive elements (4 pairs), exactly 2 pairs are kept |
| Compressed layers | Routed MoE experts (gate_proj, up_proj, down_proj) in all 60 layers |
| Left in BF16 | lm_head, embeddings, full and linear attention, norms, router (mlp.gate), shared experts, vision tower / merger, the MTP head |
The scale group is 32 because the Blackwell sparse NVFP4 MMA (SM100 and SM120) requires one scale per 32 dense K elements, which is 16 surviving elements after the 4:8 prune.
paired48_sparse, pair-bitmask v1)Packed NVFP4 stores two adjacent K elements per byte, so one pair is one byte. A paired-4:8 weight therefore has exactly 2 nonzero bytes in every 4-byte chunk. Each routed-expert linear stores:
| Tensor | dtype / shape | Contents |
|---|---|---|
weight_sparse_packed | uint8 [out, K/4] | the 2 kept bytes of every 4, in K order |
weight_sparse_mask | uint8 [out, K/16] | 4 bits per 4-byte chunk (exactly 2 set; bit i means byte i is kept); the low nibble is the lower-K chunk |
weight_scale | float8_e4m3 [out, K/32] | block scales (linear layout) |
quantization_config carries "paired48_sparse": {"layout": "pair-bitmask", "version": 1}.
The encoding is lossless: converting to and from dense NVFP4 is exact, and every tensor in
this upload was checked with a round trip. At load time vLLM rebuilds each layer's dense
packed weight and compresses it into the kernel's own layout.
The full MoESQ config for this run is in moe_sq_config.yaml.
gate_weight_exponent = 2). Learning rates are 2e-4 for masks and 2.5e-5 for weights.bibtex
@article{moesq, title = {Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts}, author = {TODO}, journal = {arXiv preprint arXiv:TODO}, year = {TODO}}For questions, open a discussion here or contact kwanhee.lee@postech.ac.kr.
| Format | compressed-tensors, nvfp4-pack-quantized, with a paired48_sparse storage marker |
weight_global_scale, input_global_scale | float32 | per-tensor global scales |