nota-ai

GLM-5.3-Nota-NVFP4-Global-Pruned-17.75

README

License: glm-5.3

Highlights

NVFP4 (4-bit float, W4A4) — group_size=16, packed in the compressed-tensors format for direct serving in vLLM. Both weights and activations are quantized to 4-bit floating point.
- Requires NVIDIA Blackwell. NVFP4 relies on the FP4 tensor cores introduced in the Blackwell architecture (e.g. B200 / B300 / GB200), so inference must run on a Blackwell-class GPU. Earlier architectures (Hopper, Ada, Ampere) do not support NVFP4 execution.
- Only the routed experts are quantized. MLA attention, the DSA indexer, routers, shared experts, the leading dense MLPs, norms, embeddings and the MTP block stay in BF16 — the same split the reference NVFP4 release uses.
Global expert pruning. Nota AI's proprietary global importance-estimation method prunes experts non-uniformly across layers — keeping only the experts that matter most in each layer — so the quantized model runs on fewer GPUs with minimal accuracy loss. Across the 75 MoE layers each block keeps between 208 and 256 experts, 15,792 in total out of the original 19,200 (17.75% pruned). The per-layer counts ship in config.json as num_experts_per_layer.
Nota AI's proprietary MoE quantization method. The release is produced with Nota AI's quantization stack for Mixture-of-Experts models, developed to preserve model quality under aggressive low-bit quantization — including MoE-aware calibration and routing-preserving quantization of the routed experts.

Performance

Table with columns: Benchmark, GLM-5.3 (BF16), Ours (Global-Pruned NVFP4)
Benchmark	GLM-5.3 (BF16)	Ours (Global-Pruned NVFP4)
Terminal-Bench 2.1	88.2	83.1
DeepSWE (v1.1)	66.9	63.7

Both gaps sit inside the sampling noise of a single run. With 89 and 113 tasks at k=1 the standard error is 4.0 and 4.5 points, and the 95% intervals — 75.4–90.9 and 54.9–72.6 — both contain the reference score, so neither difference is statistically distinguishable from no change. Relative retention is 94.3% and 95.2% at 26% of the BF16 footprint.
The run-to-run spread is that large in practice: re-running eleven already-passing tasks under identical settings flipped three of them to failure, which is expected at temperature=1.0.

Evaluation setup

Identical to the original model card's footnotes.

Table with columns: Terminal-Bench 2.1, DeepSWE (v1.1)
	Terminal-Bench 2.1	DeepSWE (v1.1)
Agent	Claude Code 2.1.207	mini-swe-agent
temperature / top_p	1.0 / 1.0	0.95 / 1.0
max_new_tokens	65,536	65,536
Timeout	6 h	6 h
Context	1M	400K
reasoning_effort	max

Installation

Built against the vLLM version from the official GLM-5.3 recipe, installed as documented there (as of 2026-08-31):

markdown
vllm 0.28.0

Then put patch/deepseek_v2.py from this repository in place of vLLM's own deepseek_v2.py:

bash
hf download nota-ai/GLM-5.3-Nota-NVFP4-Global-Pruned-17.75 patch/deepseek_v2.py --local-dir .

TARGET="$(python3 -c 'import vllm, pathlib; print(pathlib.Path(vllm.__file__).parent / "model_executor/models/deepseek_v2.py")')"
cp "${TARGET}" "${TARGET}.orig"     # keep the stock file for restoring
cp patch/deepseek_v2.py "${TARGET}"

vLLM serves GLM-5.3 through its DeepSeek modeling file — GlmMoeDsaForCausalLM is registered to deepseek_v2 and defined there as a subclass of DeepseekV2ForCausalLM — so that is the file to replace, not a GLM one.

The patch swaps config.n_routed_experts for that layer's count only while a decoder block is being constructed — DeepseekV2MoE.__init__ reads the scalar for the gate width, the e_score_correction_bias size and the FusedMoE expert count, and never reads it again. A checkpoint without num_experts_per_layer passes through untouched, so the patched file also serves stock DeepSeek and GLM models. Only model.layers.* are overridden; the MTP head is not pruned and keeps reading the scalar.

On a different vLLM version, do not copy the file over. Port the _nu_per_layer_num_experts block and the one with line in DeepseekV2DecoderLayer.__init__ into that version's file.

Quick Start

B300 × 2

bash
VLLM_NO_USAGE_STATS=1 CUDA_VISIBLE_DEVICES=0,1 \
vllm serve nota-ai/GLM-5.3-Nota-NVFP4-Global-Pruned-17.75 \
  --served-model-name GLM-5.3-Nota-NVFP4-Global-Pruned-17.75 \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.96 \
  --kv-cache-dtype fp8_e4m3 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice --tool-call-parser glm47

--gpu-memory-utilization 0.96 matters: vLLM's CUDA-graph memory profiling makes the default 0.90 behave like 0.879, and at that budget the KV cache holds 345,920 tokens — not enough for a single full-length request. At 0.96 the cache holds 1,590,400 tokens, so the model serves its full 1,048,576-token context without --max-model-len.

--enable-expert-parallel is safe here: EP requires each layer's expert count to divide by the EP size, and every layer in this checkpoint is a multiple of 16.

Do not add --enable-eplb. vLLM reads the expert count of the first MoE block and assumes every layer matches, which a non-uniform checkpoint breaks.

GLM-5.3's reasoning_effort parameter (low / high / max) works unchanged.

Patch Files for vLLM

Table with columns: Path, Role
Path	Role
`patch/deepseek_v2.py`	vLLM modeling file that reads the per-layer expert counts
`config.json`	`num_experts_per_layer` holds the 79 per-layer counts
`kept_experts.json`	which original expert indices survived, per layer

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

nota-ai

Model Tree

Base

zai-org/GLM-5.3-BF16

Quantized

this model

Input Modalities

Text

Output Modalities