GLM-5.2-AWQ-INT4-15pct API & Inference Endpoint

Method (data-free)

GLM/DeepSeek routers carry a learned e_score_correction_bias per expert: a high bias means the router had to boost that expert to select it (least favored). awq_surgery.py drops the 38 highest-bias experts per layer, keeps the 218 lowest, re-indexes survivors, and row-slices the router. No calibration data, no forward passes. Both num_experts and n_routed_experts become 218. This is not REAP (REAP needs calibration data and was infeasible on this hardware).

Serving

A 256k 4-bit MoE for multi-node TP, not a single-GPU model. Needs sm_121 Triton sparse-MLA kernels (native _flashmla_C is Hopper-only). Stack and bootstrap: github.com/CosmicRaisins/glm-5.2-gb10. Runtime: TP=4,

markdown

--kv-cache-dtype fp8_ds_mla

markdown

--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice

, cudagraph FULL, gpu-memory-utilization 0.93.

For MTP speculative decode, pair it with the matching draft built for this target: CosmicRaisins/GLM-5.2-MTP-INT4-aligned.

License

MIT, inherited. Retain upstream notices on redistribution. GLM-5.2 © Z.ai (MIT); GLM-5.2-AWQ-INT4 © cyankiwi (MIT). The data-free prune is the only modification. Not affiliated with Z.ai or cyankiwi.

Method (data-free)

Serving

markdown

--kv-cache-dtype fp8_ds_mla

markdown

--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice

, cudagraph FULL, gpu-memory-utilization 0.93.

For MTP speculative decode, pair it with the matching draft built for this target: CosmicRaisins/GLM-5.2-MTP-INT4-aligned.

GLM-5.2-AWQ-INT4-15pct

Get help setting up a custom Dedicated Endpoints.

README

Method (data-free)

Serving

License

Explore FriendliAI today

README

Method (data-free)

Serving

License