GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
flukethoughts
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
flukethoughts
Model Tree
ConicCat/GLM-4.5-Architect-106B-A12B
Input Modalities
Output Modalities
Supported Functionality
Blackwell or newer (compute capability ≥ 10.0). NVFP4 is W4A4: the activations are also 4-bit, and Hopper (sm 9.0) has no kernel that can execute it. This will not run on an H100/H200.
| count | precision | why | |
|---|---|---|---|
| attention + MLP Linears | 17,602 | NVFP4 | 46×4 attn + 1×3 dense + 45×(128×3 + 1×3) MoE |
lm_head | 1 | BF16 | 4-bit here buries control-token logits — see below |
model.embed_tokens | 1 | BF16 | never quantized |
MoE routers (mlp.gate) | 45 | BF16 | 4-bit noise changes which experts fire, not just how much |
MTP head (model.layers.46.*) | 401 | BF16 | driven by vLLM's speculative decoder |
Counts are derived from config.json (46 layers, first_k_dense_replace: 1,
n_routed_experts: 128, n_shared_experts: 1), not hardcoded.
Why
lm_headstays BF16. A previous NVFP4 build of a different model passed every static check — weight cosine 0.997, all scales finite, tokenizer faithful — and was functionally dead. Control-token embeddings are 5–10× smaller in magnitude than ordinary tokens, so round-to-nearest 4-bit error buries them. Raw text completion looked perfect while every chat request failed, because chat templates end on a control token. Weight cosine cannot detect this. This build holdslm_headat BF16 and was gated on a generation test that puts a control token at position 0 and at the end of the prompt.
bash
vllm serve <this-repo> \--max-model-len 65536 \--gpu-memory-utilization 0.92 \--kv-cache-dtype fp8 \--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
config.json retains num_nextn_predict_layers: 1 and the layer-46 weights are intact,
so vLLM can run the model's own MTP head as the draft model. This is a decode-speed win
on bandwidth-bound hardware, which is where a 12B-active MoE spends its time.
--gpu-memory-utilization 0.92, not 0.970.97 boots cleanly and then crash-loops under real load. Nothing about that is visible at startup. Learned on a sibling build; do not raise it without testing under load.
256 sequences at 4,096 tokens — 70% roleplay / 30% general text, matching the intended workload. Calibrating an RP model purely on wikitext moves the quantization grid toward text it will never see.
Built on 2×B300 SXM6. Scripts, gates and the failure log are in the build repo, including the measured RAM requirement (this MoE peaks at ~363 GB of anonymous memory during expert linearization — a 250 GB box dies at layer 11/45 and a 376 GB box dies at layer 39/45).