GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ISTA-DASLab
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ISTA-DASLab
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
Qwen/Qwen3.6-35B-A3B{-2, -1, 0, +1} × scaleself_attn, linear_attn, shared experts, and lm_headgate_proj, up_proj, down_proj)self_attn, linear_attn), shared experts, and LM headgate, and other non-quantized componentsThe Hugging Face "Tensor types" widget reports the container dtype of each tensor stored in the safetensors checkpoint, rather than the effective precision of the original model weights.
The routed-expert MLPs use the Humming exact-width 2-bit layout. For every
2-bit expert-MLP Linear with original weight shape
[out_features, in_features], the following tensors are stored:
| Tensor | Dtype | Shape on disk | Meaning |
|---|---|---|---|
<layer>.weight | I32 | [out_features, in_features × 2 / 32] = [out_features, in_features / 16] | 2-bit values bit-packed along the input dimension, LSB-first: 16 weights per INT32 word |
<layer>.weight_scale | BF16 | [out_features, in_features / 128] | One symmetric scale per group of group_size = 128 weights along the input dimension |
For the 2-bit GSQ expert weights, the effective storage is:
text
2 bits (packed) + 16 bits / 128 (group scale) ≈ 2.13 bpp
The checkpoint therefore uses a mixed-precision quantization layout: 2-bit GSQ for routed-expert MLP weights, INT8 for attention, shared experts, and the LM head, and BF16 for the remaining unquantized components.
The quantization_config in config.json describes the Humming quantization
layout used by the checkpoint.
Note: GSQ training first writes shards in
compressed-tensorspack-quantizedformat, where the 2-bit codebook is padded into a 4-bit INT32 container. The published checkpoint has been re-packed viaconvert_to_humming.pyinto exact-width 2-bit Humming storage, hence the2 / 32shape factor for the 2-bit expert weights.
Serving this checkpoint requires:
Install the required vLLM version:
bash
pip install vllm==0.27.1
Then, from the directory containing the model files, run the included
patch_vllm.py script:
bash
python patch_vllm.py
The patch must be executed in the same Python environment in which vLLM 0.27.1 is installed. It patches the installed vLLM package with the changes required to load and serve this checkpoint.
After applying the patch, the model can be served normally:
bash
vllm serve ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ --reasonin-parser qwen3
Important: running the model with an unpatched vLLM installation is not supported. If vLLM is reinstalled or the environment is recreated, run
python patch_vllm.pyagain before serving the model.
Model size / text-only usage: the full checkpoint is approximately 12.6 GB, including the MTP and vision components. These components are optional for ordinary text-only generation: the vision weights are only required for multimodal inputs, while the MTP weights are only required when using MTP/speculative decoding. Removing both the vision and MTP weights reduces the model size to approximately 9.9 GB.
Actual VRAM usage during serving will be higher than the raw model size and depends on KV-cache allocation, context length, batch size, and vLLM runtime overhead.
bibtex
@article{gsq2026,title = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurti{\'c}, Eldar and Kleinegger, Max and Alistarh, Dan},journal= {arXiv preprint arXiv:2604.18556},year = {2026},url = {https://arxiv.org/abs/2604.18556}}
self_attn / linear_attn / shared-expert / lm_head weights | I8 | implementation-dependent | INT8-quantized weights |
Embeddings / norms / MoE routing gate / other unquantized tensors | BF16 | unchanged | Copied or retained at BF16 precision |