GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
scottgl
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
scottgl
Model Tree
Qwen/Qwen3.5-122B-A10B
Input Modalities
Output Modalities
Supported Functionality
The base NVFP4 checkpoint leaves several layer types in BF16 to keep it hardware-agnostic. On GB10 (NVIDIA Grace Blackwell, SM12.1), those BF16 layers are a significant bottleneck:
These optimizations are implemented in scottgl9/sglang-spark-gb10-optimizations as runtime post-quantization hooks. This checkpoint bakes them in so:
| Layer | Original | Baked format | Why |
|---|---|---|---|
| MoE expert weights (gate/up/down) | NVFP4 (W4A4) | NVFP4 (W4A4) | From source checkpoint — preserved as-is |
GDN in_proj_qkv, in_proj_z | BF16 | NVFP4 (Marlin FP4) | Large projections; SM12.1 has native Marlin FP4 kernels. CUTLASS FP4 is broken on GB10. |
GDN in_proj_a, in_proj_b, out_proj | BF16 | FP8 (per-tensor, dynamic activations) | Smaller GDN projections; FP8 CUTLASS works correctly on SM12.1. cos_sim > 0.999, SNR = 31.5 dB vs BF16. |
| lm_head (vocabulary projection) | BF16 |
~46 tok/s decode on a single NVIDIA DGX Spark (GB10, 128 GB unified memory) using SGLang with NEXTN speculative decoding (steps=2, draft_tokens=2).
Baseline without any of these optimizations: ~28 tok/s. The baked quantizations account for approximately +18 tok/s of that gain.
Tested on: NVIDIA DGX Spark (ASUS Ascent GX10), GB10 Grace Blackwell, SM12.1, 128 GB unified memory.
Requirements:
Will NOT work on:
bash
# Load directly — no post-quant overhead at startupSGLANG_QUANTIZE_LM_HEAD_FP8=0 ./sglang.sh Qwen3.5-NVFP4 \--model-path scottgl9/Qwen3.5-122B-A10B-NVFP4-GB10
SGLANG_QUANTIZE_LM_HEAD_FP8=0 tells SGLang to skip the lm_head FP8 post-quant (already baked in). All other sglang.sh flags remain unchanged.
With NEXTN speculative decoding (recommended):
bash
SGLANG_QUANTIZE_LM_HEAD_FP8=0 ./sglang.sh Qwen3.5-NVFP4 \--model-path scottgl9/Qwen3.5-122B-A10B-NVFP4-GB10 \--speculative-algorithm NEXTN \--speculative-num-steps 2 \--speculative-num-draft-tokens 2
Apache 2.0 (same as Qwen3.5)
| FP8 (dynamic activations) |
| 3072 x 248,320 matmul at every decode step. FP8 halves DRAM reads vs BF16. Activations remain dynamic per-token for accuracy. |
| Attention Q/K/V/O | BF16 | BF16 | Standard attention; not the bandwidth bottleneck at batch=1. |