GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
soyrsoyr
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
soyrsoyr
Model Tree
zai-org/GLM-5.3
Input Modalities
Output Modalities
Supported Functionality
Full quantization and export passed on four NVIDIA B200 GPUs in approximately 54 minutes. The exported checkpoint contains 118,607 tensors, including 1,568 MTP tensors. All 777 quantized MTP matrices passed numerical reconstruction checks, with a maximum normalized MSE of 0.0142504545. Dense exclusions matched the source exactly; scale and checkpoint-index checks passed. See validation.json.
The exported checkpoint also passed a vLLM 0.28.0 decoding smoke test on four B200s: 474 accepted draft tokens out of 544 drafted tokens (87.13%), using one MTP draft token per step. The test used eight prompts, greedy generation, and a 128-token output cap per prompt. Every output reached that cap while producing reasoning text, even though the test passed enable_thinking=False to the chat template. These results establish nonzero MTP acceptance; they are not a quality evaluation or a speedup benchmark. See mtp-acceptance.json.
The exact scripts used for the run are included:
The quantization call in that harness was:
python
import torchfrom transformers import AutoModelForCausalLM, FineGrainedFP8Configfrom compressed_tensors.offload import set_onload_devicefrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.utils import load_context # `source` is the local snapshot at the pinned source revision.# Run under the included harness: it initializes distributed execution and# installs the padded-block FP8 loader workaround described below.with load_context(): model = AutoModelForCausalLM.from_pretrained( source, quantization_config=FineGrainedFP8Config(dequantize=True), dtype=torch.bfloat16, device_map="auto_offload", max_memory={"cpu": "300GiB"}, offload_folder=str(RUN / "offload"), local_files_only=True, )set_onload_device(model, "cuda")if not model.generation_config.do_sample: model.generation_config.top_p = 1.0model.generation_config.validate(strict=True) recipe = QuantizationModifier( scheme="MXFP4", targets=["Linear", r"re:^mtp\.layers\."], ignore=[ "lm_head", r"re:.*\.eh_proj$", r"re:.*\.mlp\.gate$", r"re:.*\.indexer\.(?:wk|weights_proj)$", ],)oneshot(model=model, recipe=recipe, output_dir=str(destination))This recipe does not require a calibration dataset. It uses the MXFP4 weight observer and dynamic activation quantization. The MTP configuration, native FP8 scale restoration, and integration with the existing export path come from the pinned llm-compressor commit.
The quantization venv used Python 3.12, PyTorch 2.13.0 with CUDA 13.0, Transformers 5.17.0, and compressed-tensors 0.19.0. The full installed package inventory is in environment-quantization.json. The run used four B200s, a 300 GiB CPU offload limit, disk offload, and a host with about 1.9 TiB RAM. The harness's download step requires at least 4 TiB of free disk headroom; the completed run occupied about 2.8 TiB before cleanup.
Use a dedicated venv containing those dependencies, then install the tested llm-compressor checkout without changing them:
bash
# From a local copy of this repository, with the quantization venv activated:cd reproductiongit clone https://github.com/soyr-redhat/llm-compressor.gitgit -C llm-compressor checkout 7ebb01341195269a2236e24fe14336a74dcda17apython -m pip install --no-deps -e ./llm-compressor export HF_HUB_CACHE="$PWD/hf-cache"export HF_XET_CACHE="$PWD/xet-cache"export TMPDIR="$PWD/tmp"export OMP_NUM_THREADS=8 TOKENIZERS_PARALLELISM=false NCCL_DEBUG=WARNmkdir -p "$HF_HUB_CACHE" "$HF_XET_CACHE" "$TMPDIR"python ohio-glm53-download.pyexport HF_HUB_OFFLINE=1python -m torch.distributed.run --standalone --nproc-per-node=4 \ ohio-glm53-runtime.py fullThe Ohio run reserved four GPUs through chg before executing the torch.distributed.run command. Ensure four GPUs are assigned to your process on a shared host. The output is reproduction/GLM-5.3-MXFP4-MTP.
The source checkpoint uses FP8 block quantization. This run required three explicit harness adjustments, all present in the included runtime script:
FP8BlockDequantizer, preserving the source scales. The replacement was checked against a direct numerical reference.top_p=0.95 while sampling is disabled. The harness sets top_p=1.0 to preserve greedy behavior and satisfy Transformers' strict save validation.This is therefore not a stock-Transformers end-to-end quantization result. The snippet above depends on the included harness for those adjustments. The llm-compressor checkout was installed with --no-deps; its development metadata requests a newer compressed-tensors prerelease, but this run explicitly used 0.19.0.
The acceptance test used a separate activated venv with vLLM 0.28.0, PyTorch 2.13.0, Transformers 5.16.1, and compressed-tensors 0.17.0. See environment-serving.json. Activate your serving venv before launching so its ninja executable is available; CUDA 13.0's nvcc must also be on PATH for kernel compilation.
This server command uses the same engine settings as the successful Python LLM smoke test:
bash
source /path/to/serving-venv/bin/activateexport PATH="${CUDA_HOME:-/usr/local/cuda-13.0}/bin:$PATH"export OMP_NUM_THREADS=1 TOKENIZERS_PARALLELISM=false vllm serve soyrsoyr/GLM-5.3-MXFP4-MTP \ --tensor-parallel-size 4 \ --dtype bfloat16 \ --max-model-len 4096 \ --max-num-seqs 8 \ --max-num-batched-tokens 4096 \ --gpu-memory-utilization 0.85 \ --enforce-eager \ --no-enable-prefix-caching \ --seed 42 \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}'The four-GPU smoke test loaded approximately 96.13 GiB of model weights per GPU, before KV cache and other runtime allocations. Two 180 GiB B200s are insufficient for this fully GPU-resident checkpoint. The server command is an equivalent launch configuration; the recorded acceptance test used vLLM's Python LLM API, not the HTTP server.
The weights retain the original GLM-5.3 license, including its copyright notice and conditions. This is an experimental quantized derivative, not an official Z.AI release.
import torchfrom transformers import AutoModelForCausalLM, FineGrainedFP8Configfrom compressed_tensors.offload import set_onload_devicefrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.utils import load_context # `source` is the local snapshot at the pinned source revision.# Run under the included harness: it initializes distributed execution and# installs the padded-block FP8 loader workaround described below.with load_context(): model = AutoModelForCausalLM.from_pretrained( source, quantization_config=FineGrainedFP8Config(dequantize=True), dtype=torch.bfloat16, device_map="auto_offload", max_memory={"cpu": "300GiB"}, offload_folder=str(RUN / "offload"), local_files_only=True, )set_onload_device(model, "cuda")if not model.generation_config.do_sample: model.generation_config.top_p = 1.0model.generation_config.validate(strict=True) recipe = QuantizationModifier( scheme="MXFP4", targets=["Linear", r"re:^mtp\.layers\."], ignore=[ "lm_head", r"re:.*\.eh_proj$", r"re:.*\.mlp\.gate$", r"re:.*\.indexer\.(?:wk|weights_proj)$", ],)oneshot(model=model, recipe=recipe, output_dir=str(destination))# From a local copy of this repository, with the quantization venv activated:cd reproductiongit clone https://github.com/soyr-redhat/llm-compressor.gitgit -C llm-compressor checkout 7ebb01341195269a2236e24fe14336a74dcda17apython -m pip install --no-deps -e ./llm-compressor export HF_HUB_CACHE="$PWD/hf-cache"export HF_XET_CACHE="$PWD/xet-cache"export TMPDIR="$PWD/tmp"export OMP_NUM_THREADS=8 TOKENIZERS_PARALLELISM=false NCCL_DEBUG=WARNmkdir -p "$HF_HUB_CACHE" "$HF_XET_CACHE" "$TMPDIR"python ohio-glm53-download.pyexport HF_HUB_OFFLINE=1python -m torch.distributed.run --standalone --nproc-per-node=4 \ ohio-glm53-runtime.py fullsource /path/to/serving-venv/bin/activateexport PATH="${CUDA_HOME:-/usr/local/cuda-13.0}/bin:$PATH"export OMP_NUM_THREADS=1 TOKENIZERS_PARALLELISM=false vllm serve soyrsoyr/GLM-5.3-MXFP4-MTP \ --tensor-parallel-size 4 \ --dtype bfloat16 \ --max-model-len 4096 \ --max-num-seqs 8 \ --max-num-batched-tokens 4096 \ --gpu-memory-utilization 0.85 \ --enforce-eager \ --no-enable-prefix-caching \ --seed 42 \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}'