GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
cloud19
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
cloud19
Model Tree
joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive
Input Modalities
Output Modalities
Supported Functionality
The following modules were excluded from quantization: lm_head, embeddings,
the vision tower (model.visual.*), the Gated DeltaNet linear-attention layers
(linear_attn.*), MoE routers (mlp.gate), the shared-expert gate
(shared_expert_gate) and the MTP head (mtp.*). The ignore list matches the
reference RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic quantization.
python
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGenerationfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifier MODEL_ID = "joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive"model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")processor = AutoProcessor.from_pretrained(MODEL_ID) recipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=[ "re:.*lm_head", "re:visual.*", "re:model.visual.*", "re:.*mlp.gate$", "re:.*embed_tokens$", "re:.*shared_expert_gate$", "re:.*linear_attn.*", "re:^mtp.*", ],)oneshot(model=model, recipe=recipe)model.save_pretrained("out")processor.save_pretrained("out")bash
vllm serve cloud19/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive-FP8-Dynamic \ --served-model-name joyfox \ --enable-chunked-prefill \ --max-model-len 16384 --kv-cache-dtype fp8 \ --limit-mm-per-prompt '{"image": 0, "audio": 0, "video": 0}' \ --default-chat-template-kwargs '{"enable_thinking": false}'Notes:
align; passing
--no-enable-chunked-prefill makes the engine refuse to start.model_mtp.safetensors, but speculative decoding
is intentionally not used here.<think>) is disabled at the chat-template level via
enable_thinking: false.from transformers import AutoProcessor, Qwen3_5MoeForConditionalGenerationfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifier MODEL_ID = "joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive"model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")processor = AutoProcessor.from_pretrained(MODEL_ID) recipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=[ "re:.*lm_head", "re:visual.*", "re:model.visual.*", "re:.*mlp.gate$", "re:.*embed_tokens$", "re:.*shared_expert_gate$", "re:.*linear_attn.*", "re:^mtp.*", ],)oneshot(model=model, recipe=recipe)model.save_pretrained("out")processor.save_pretrained("out")