GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
petra312
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
petra312
Model Tree
ArliAI/GLM-4.6-Derestricted-v3
Input Modalities
Output Modalities
Supported Functionality
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig model_id = "petra312/GLM-4.6-Derestricted-v3-NF4"config = AutoConfig.from_pretrained(model_id, trust_remote_code=True) device_map = { "model.embed_tokens": 0, "model.norm": 0, "model.rotary_emb": 0, "lm_head": 0,}for i in range(config.num_hidden_layers): device_map[f"model.layers.{i}.self_attn"] = 0 device_map[f"model.layers.{i}.input_layernorm"] = 0 device_map[f"model.layers.{i}.post_attention_layernorm"] = 0 if i < config.first_k_dense_replace: device_map[f"model.layers.{i}.mlp"] = 0 else: device_map[f"model.layers.{i}.mlp.gate"] = 0 device_map[f"model.layers.{i}.mlp.shared_experts"] = 0 device_map[f"model.layers.{i}.mlp.experts"] = "cpu" tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained( model_id, device_map=device_map, trust_remote_code=True, torch_dtype=torch.bfloat16,)Requires bitsandbytes, accelerate, transformers==4.57.6.
Glm4MoeForCausalLM).modeling_glm4_moe.py in this repo returns router logits so REAP-style expert observers still work.import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig model_id = "petra312/GLM-4.6-Derestricted-v3-NF4"config = AutoConfig.from_pretrained(model_id, trust_remote_code=True) device_map = { "model.embed_tokens": 0, "model.norm": 0, "model.rotary_emb": 0, "lm_head": 0,}for i in range(config.num_hidden_layers): device_map[f"model.layers.{i}.self_attn"] = 0 device_map[f"model.layers.{i}.input_layernorm"] = 0 device_map[f"model.layers.{i}.post_attention_layernorm"] = 0 if i < config.first_k_dense_replace: device_map[f"model.layers.{i}.mlp"] = 0 else: device_map[f"model.layers.{i}.mlp.gate"] = 0 device_map[f"model.layers.{i}.mlp.shared_experts"] = 0 device_map[f"model.layers.{i}.mlp.experts"] = "cpu" tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained( model_id, device_map=device_map, trust_remote_code=True, torch_dtype=torch.bfloat16,)