GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
festr2
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
festr2
Model Tree
QuantTrio/GLM-5.2-Int4-Int8Mix
Input Modalities
Output Modalities
Supported Functionality
Base model:
Quantized sources:
This is not a full re-quantization from BF16. It is a merged checkpoint:
mtp_block, fused_qkv_a_proj) used at runtime.The effective quantization_config uses compressed-tensors with
format: nvfp4-pack-quantized.
| Scope | Format |
|---|---|
model.layers.0 and ignored special paths | BF16 |
| Dense attention and ordinary linear weights in layers 1-77 | W8A16 INT8, symmetric group quantization, group size 128 |
| Shared experts in layers 1-77 | W8A16 INT8, symmetric group quantization, group size 128 |
| Non-shared routed MoE experts in layers 3-77 | NVFP4-style float 4-bit weights, tensor-group strategy, group size 16 |
| Layer 78 MTP block | W8A16 INT8, channel-wise |
mlp.gate, attention indexer, norms, embeddings, and special heads | BF16 / ignored |
Compared with the original QuantTrio checkpoint, the routed expert tensors are not INT4 group-size-128 weights anymore. They are replaced by Luke's NVFP4 expert tensors.
Compared with Luke's NVFP4 checkpoint, this checkpoint does not keep the dense and attention parts in the same BF16/NVFP4 ModelOpt layout. Those parts come from QuantTrio's compact W8A16 export.
Luke's NVFP4 checkpoint quantizes directly from the BF16 GLM-5.2 checkpoint using NVIDIA Model Optimizer. In that source checkpoint, only the non-shared MoE expert MLP projections are quantized to NVFP4; attention weights, early dense MLP layers, and shared experts are left unquantized. The calibration uses natural top-k routing rather than forcing all experts active, with broad sample coverage to better match the distributions experts see during inference.
That matters for this hybrid checkpoint because the routed MoE experts are the largest parameter component and the most routing-sensitive part of GLM-5.2. NVFP4 uses small 16-value floating-point blocks with FP8 scale metadata, while the original QuantTrio expert path uses integer 4-bit group quantization with group size 128. The finer scaling granularity is one reason the NVFP4 expert path can preserve the BF16 distribution better in local KLD tests.
KLD/JS is a local next-token distribution proxy, not a full model-quality benchmark. It is useful for detecting numerical regressions, but deployment quality should also be checked with long-context tasks, coding prompts, tool calling, repetition/CJK watchdogs, MTP acceptance, throughput, and VRAM.
Repeated local KLD measurements from the vLLM/B12X test stack showed:
| Checkpoint | Prefill KLD mean | Decode JS mean |
|---|---|---|
| Luke NVFP4 | 0.068257 | 0.00000236 |
| QuantTrio GLM-5.2 Int4-Int8Mix | 0.070448 | 0.00000286 |
| This hybrid, W8A16 + Luke NVFP4 experts | 0.071182 | 0.00000264 |
Interpretation:
This checkpoint was prepared for the local vLLM/B12X GLM-5.2 stack used by local-inference-lab/rtx6kpro. It is not claimed to be a generic drop-in model for every runtime.
Known working class of configuration:
--quantization compressed-tensors--kv-cache-dtype fp8--attention-backend B12X_MLA_SPARSE--moe-backend b12xExample shape used in local testing:
bash
vllm serve /path/to/GLM-5.2-NVFP4-Int8Mix \ --served-model-name GLM-5.2 \ --trust-remote-code \ --tensor-parallel-size 8 \ --decode-context-parallel-size 1 \ --quantization compressed-tensors \ --attention-backend B12X_MLA_SPARSE \ --moe-backend b12x \ --kv-cache-dtype fp8 \ --enable-auto-tool-choice \ --tool-call-parser glm47 \ --reasoning-parser glm45For exact Docker images and launch recipes used in local benchmarking, see the GLM-5.2 v12 notes in:
https://github.com/local-inference-lab/rtx6kpro/blob/master/models/glm5.2_v12.md
Approximate uploaded size: 409.33 GiB.
The model card inherits the MIT license metadata from the source GLM-5.2 release and source model cards. Check the upstream model cards for complete license and usage details.
vllm serve /path/to/GLM-5.2-NVFP4-Int8Mix \ --served-model-name GLM-5.2 \ --trust-remote-code \ --tensor-parallel-size 8 \ --decode-context-parallel-size 1 \ --quantization compressed-tensors \ --attention-backend B12X_MLA_SPARSE \ --moe-backend b12x \ --kv-cache-dtype fp8 \ --enable-auto-tool-choice \ --tool-call-parser glm47 \ --reasoning-parser glm45