GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
casperhansen
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
casperhansen
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Method | RTN (round-to-nearest), data-free (llmcompressor.model_free_ptq) |
| Precision | W4A16 (4-bit integer weights, 16-bit activations) |
| Strategy | group, group_size = 32 |
| Symmetric | true |
| Format | pack-quantized (compressed-tensors) |
| Quantized modules | routed MoE experts only (*.mlp.experts.*.{gate,up,down}_proj), incl. the MTP module |
| Left in BF16 | attention (self_attn + linear_attn/Gated DeltaNet), shared expert, router gates, vision tower, lm_head, embeddings, norms |
This mirrors Kimi's config: num_bits=4, type=int, strategy=group, group_size=32, symmetric=true, format=pack-quantized,
quantizing the MoE components while keeping attention and the shared expert at full precision.
Result: ~72 GB (BF16) → ~23 GB on disk.
bash
vllm serve casperhansen/Qwen3.6-35B-A3B-INT4-RTN \ --tensor-parallel-size 1 \ --max-model-len 8192 \ --reasoning-parser qwen3# add --language-model-only for text-only servingpython
from openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")resp = client.chat.completions.create( model="casperhansen/Qwen3.6-35B-A3B-INT4-RTN", messages=[{"role": "user", "content": "Give me a fun fact about the ocean."}], max_tokens=256, temperature=0.7, top_p=0.8,)print(resp.choices[0].message.content)Verified to load and generate on a single NVIDIA H200 with vLLM (INT4 WNA16 MoE Marlin path).
python
from compressed_tensors.config import CompressionFormatfrom compressed_tensors.quantization import ( QuantizationArgs, QuantizationScheme, QuantizationStrategy, QuantizationType,)from llmcompressor import model_free_ptq scheme = QuantizationScheme( targets=[r"re:.*mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"], weights=QuantizationArgs( num_bits=4, type=QuantizationType.INT, strategy=QuantizationStrategy.GROUP, group_size=32, symmetric=True, observer="minmax", dynamic=False, ), format=CompressionFormat.pack_quantized.value,) model_free_ptq( model_stub="Qwen/Qwen3.6-35B-A3B", save_directory="Qwen3.6-35B-A3B-INT4-RTN", scheme=scheme, max_workers=8, device="cuda:0",)Apache-2.0, inherited from the base model Qwen/Qwen3.6-35B-A3B.
vllm serve casperhansen/Qwen3.6-35B-A3B-INT4-RTN \ --tensor-parallel-size 1 \ --max-model-len 8192 \ --reasoning-parser qwen3# add --language-model-only for text-only servingfrom openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")resp = client.chat.completions.create( model="casperhansen/Qwen3.6-35B-A3B-INT4-RTN", messages=[{"role": "user", "content": "Give me a fun fact about the ocean."}], max_tokens=256, temperature=0.7, top_p=0.8,)print(resp.choices[0].message.content)from compressed_tensors.config import CompressionFormatfrom compressed_tensors.quantization import ( QuantizationArgs, QuantizationScheme, QuantizationStrategy, QuantizationType,)from llmcompressor import model_free_ptq scheme = QuantizationScheme( targets=[r"re:.*mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"], weights=QuantizationArgs( num_bits=4, type=QuantizationType.INT, strategy=QuantizationStrategy.GROUP, group_size=32, symmetric=True, observer="minmax", dynamic=False, ), format=CompressionFormat.pack_quantized.value,) model_free_ptq( model_stub="Qwen/Qwen3.6-35B-A3B", save_directory="Qwen3.6-35B-A3B-INT4-RTN", scheme=scheme, max_workers=8, device="cuda:0",)