GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
cgg507
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
cgg507
Model Tree
cgg507/GLM-5.3-Flash-GM-CoT-LoRA
Input Modalities
Output Modalities
Supported Functionality
compressed-tensors nvfp4-pack-quantized. Tensor names, dtypes, shapes, file grouping and quantization_config
(including the 1,607-entry ignore list) are identical to
cgg507/GLM-5.3-Flash-GM-E32-NVFP4. All 147,634 tensors
were checked.
| Modules | Format |
|---|---|
routed experts of layers 3-44, gate_proj / up_proj / down_proj (36,288 matrices) | NVFP4: fp4 e2m1 weights, group 16, fp8-e4m3 group scales + fp32 global scale (gate/up fused); input_global_scale from calibration |
attention (KDA, MLA), DSA indexer, routers, shared experts, dense MLPs, mHC, norms, embeddings, lm_head | BF16 (mHC hc_*_base/scale, A_log, dt_bias, e_score_correction_bias fp32), same dtypes as the E32 repo |
| vision tower | BF16, unchanged |
MTP / NextN layer 45 (model_mtp.safetensors, 889 tensors) | BF16 (dequantised from FP8), not adapted, not quantized |
Total 205 GB (191.0 GiB): 46 layer/non-layer shards + model_mtp.safetensors.
nmoe-quantThe quant was produced natively by nmoe-quant (C++/HIP, part of nmoe, branch tools/nmoe-quant) on
2x Radeon AI PRO R9700 (ROCm 7.2). It reuses the nmoe trainer's GLM-5.3-Flash forward code. No Python and no
llm-compressor were used for the heavy work. The run goes layer by layer and never holds a full BF16 model:
W += (alpha/r) B A (fp32, stored
as bf16). The adapters are rank 64 / alpha 32 for attention / shared / dense (496 adapters) and rank 32 /
alpha 16 for routed experts (36,288 adapters). There are 36,784 targets (attention 361, shared expert 126,
dense MLP 9, routed expert 36,288). Each target was merged exactly once with a matching shape (asserted).
Adapter sha256[:16] 8d931db5d5fda2e2.H = X^T X;H_e = A_e^T A_e, with A_e = SiLU(gate_e x) * (up_e x) over all tokens;NVFP4PackedCompressor re-quantisation of bf16(Q) and packing (low nibble = even column).
The quantized experts then propagate to the next layer with normal top-8 routing.quantization_config, ignore list regenerated and required to equal
the E32 set), tokenizer/template files. --verify re-reads the output and checks:
Calibration: 512 x 2048 tokens with the GLM chat template (ultrachat_200k 267, roleplay training conversations 154, calibration_datav3 31, wikitext-2 60). 2 of the 512 sequences are held out from the Hessians for the metrics below.
Time: ~97 min of layer compute. A MoE layer took ~136 s: 13 s capture, 61 s Hessians, 38 s GPTQ, 21 s propagation. The ~68 s NFS read of the next layer runs in parallel. The whole run, including a 5-layer trial that was continued, checkpoints, finalize and verify, took 2 h 10 min.
tr(D H D^T) / tr(W H W^T) on the 4,096 held-out
tokens, over all 36,288 matrices: median per layer 0.0031-0.0083 (median 0.0062), p99 per layer <= 0.0089.
Relative weight error 0.099. No RTN fallbacks.refusal_bench/scripts/glm_nvfp4_local, glmq.py + compressed-tensors):
NVFP4PackedCompressor.decompress equals the expected dequant.quantize_weight + NVFP4PackedCompressor;nmoe serve --gpu --tp-size 2, thinking ON, GM system prompt, temperature 0.6):
<think> plan: set the scene, play the informant, don't act for the player, end
on a decision;quantization/nmoe_smoke.json.bash
hf download cgg507/GLM-5.3-Flash-GM-CoT-NVFP4 --local-dir GLM-5.3-Flash-GM-CoT-NVFP4nmoe serve GLM-5.3-Flash-GM-CoT-NVFP4 --gpu --port 1337The adapter is already merged, so no --lora-path is needed. Requests use chat_template_kwargs.enable_thinking
(default on for GLM); the reasoning is returned in reasoning_content.
This is standard compressed-tensors nvfp4-pack-quantized with per-expert names, the same as the E32 repo. Any
runtime that supports glm5_next and compressed-tensors NVFP4 MoE should load it, but none was tested here.
| Path | Content |
|---|---|
model-000NN-of-00046.safetensors | layers 0-44 (one file per layer) + embeddings / lm_head / vision tower (file 46) |
model_mtp.safetensors | MTP layer 45 (BF16) |
quantization/ | merge plan, run config, per-layer GPTQ reports (layers/NN.json), verify report, check reports, smoke test, run log |
hf download cgg507/GLM-5.3-Flash-GM-CoT-NVFP4 --local-dir GLM-5.3-Flash-GM-CoT-NVFP4nmoe serve GLM-5.3-Flash-GM-CoT-NVFP4 --gpu --port 1337