GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
soyrsoyr
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
soyrsoyr
Model Tree
zai-org/GLM-5.3
Input Modalities
Output Modalities
Supported Functionality
The backbone is calibrated and quantized inside oneshot. The mtp_scheme
argument asks the same call to compress the MTP weights during post-processing.
python
from datasets import Datasetfrom transformers import AutoModelForCausalLM from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.utils import load_context model_id = "/path/to/glm-5.3-tiny-mtp-source"dataset = Dataset.from_dict( { "text": [ "Explain why speculative decoding can reduce inference latency.", "Write a short Python function that adds two integers.", "Summarize the benefits of mixture-of-experts language models.", "What is the capital of France?", "List three considerations when deploying a language model.", "Describe the difference between weights and activations.", "Complete the sequence: one, two, three, four.", "Give a concise definition of quantization calibration.", ] }) recipe = QuantizationModifier( targets="Linear", scheme="NVFP4", ignore=[ "lm_head", r"re:.*mlp\.gate$", r"re:.*self_attn\.indexer\.(?:weights_proj|wk)$", ],) with load_context(AutoModelForCausalLM): model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16")model.generation_config.top_p = None oneshot( model=model, recipe=recipe, dataset=dataset, output_dir="GLM-5.3-MTP-NVFP4-Test-Fixture", max_seq_length=128, num_calibration_samples=len(dataset), moe_calibrate_all_experts=True, mtp_scheme="NVFP4",)Transformers does not construct the MTP layer, so it cannot observe MTP activations during calibration. The MTP group is therefore weight-only NVFP4; the backbone remains calibrated W4A4 NVFP4.
model_mtp.safetensors; no uncalibrated MTP activation scales are stored.bash
vllm serve soyrsoyr/GLM-5.3-MTP-NVFP4-Test-Fixture \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}'from datasets import Datasetfrom transformers import AutoModelForCausalLM from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.utils import load_context model_id = "/path/to/glm-5.3-tiny-mtp-source"dataset = Dataset.from_dict( { "text": [ "Explain why speculative decoding can reduce inference latency.", "Write a short Python function that adds two integers.", "Summarize the benefits of mixture-of-experts language models.", "What is the capital of France?", "List three considerations when deploying a language model.", "Describe the difference between weights and activations.", "Complete the sequence: one, two, three, four.", "Give a concise definition of quantization calibration.", ] }) recipe = QuantizationModifier( targets="Linear", scheme="NVFP4", ignore=[ "lm_head", r"re:.*mlp\.gate$", r"re:.*self_attn\.indexer\.(?:weights_proj|wk)$", ],) with load_context(AutoModelForCausalLM): model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16")model.generation_config.top_p = None oneshot( model=model, recipe=recipe, dataset=dataset, output_dir="GLM-5.3-MTP-NVFP4-Test-Fixture", max_seq_length=128, num_calibration_samples=len(dataset), moe_calibrate_all_experts=True, mtp_scheme="NVFP4",)