GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
OptimizeLLM
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
OptimizeLLM
Model Tree
Qwen/Qwen3.5-122B-A10B
Input Modalities
Output Modalities
Supported Functionality
Tested on vLLM 0.19+.
bash
vllm serve OptimizeLLM/Qwen3.5-122B-A10B-heretic-MTP-NVFP4 \ --tensor-parallel-size 2 \ --reasoning-parser qwen3 \ --reasoning-config '{}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":6}'| MTP Speculative Tokens | tok/s |
|---|---|
| 0 (disabled) | ~105 |
| 1 | ~115 |
| 2 | ~145 |
| 3 | ~170 |
| 6 | ~190 |
Quantized with llm-compressor (compressed-tensors v0.14.1.dev28).
nvfp4-pack-quantizedfloat8_e4m3fnmoe_calibrate_all_experts=TrueThe upstream heretic quant quantized MTP heads to FP4, which breaks speculative decoding. All 785 mtp.* tensors were extracted from base Qwen/Qwen3.5-122B-A10B in BF16 and saved as a separate model_mtp.safetensors shard (4.8GB).
use_auth_token kwarg crashes on transformers 5.x — patch to token.offload_folder is required even without disk offload (transformers reformats fused expert tensors during loading).ctypes.CDLL("libc.so.6").mallopt(-1, 0) before loading prevents glibc malloc arena bloat.vllm serve OptimizeLLM/Qwen3.5-122B-A10B-heretic-MTP-NVFP4 \ --tensor-parallel-size 2 \ --reasoning-parser qwen3 \ --reasoning-config '{}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":6}'