GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
0xSero
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
0xSero
Model Tree
Qwen/Qwen3.5-397B-A17B
Input Modalities
Output Modalities
Supported Functionality
| Base model | Qwen/Qwen3.5-397B-A17B |
| Format | W4A16 |
| Total params | 264B |
| Active / token | — |
| Experts / layer | — |
| Layers | — |
| Hidden size | — |
| Context | — |
| On-disk size | 282 GB |
| Variant | Format | Link |
|---|---|---|
Qwen3.5-264B | BF16 | link |
Qwen3.5-264B-FP8 | FP8 | link |
Qwen3.5-264B-W4A16 (this) | W4A16 | link |
0xSero/Qwen3.5-264B-W4A16Qwen/Qwen3.5-397B-A17Bquantized34%reapW4A16auto_round:auto_gptq0xSero/Qwen3.5-264B0xSeroSybil SolutionsREAP PR170xSeroHybrid MoE + Linear Attention (GDN/Mamba-style):
linear_attention and full_attention layer typesQwen3_5MoeForConditionalGeneration architectureThe vision encoder (visual-encoder.safetensors, 870 MB, 333 tensor keys) was transplanted from the 262B variant. The original 264B model was text-only; the vision weights are from the same Qwen3.5 architecture family and are fully compatible. Vision supports image understanding via the standard OpenAI image_url content format.
/home/ubuntu/qwen397-full/observer-calibv1/qwen397-pr17-calibv1-23k-16k-observer-state.raw.pt/home/ubuntu/qwen397-full/observer-calibv1/qwen397-pr17-calibv1-23k-16k-detail-state.raw.ptEvaluated on 8x RTX 3090 (24 GB each) with vLLM, TP=8, expert parallel, fp8 KV cache.
| Benchmark | Samples | Score |
|---|---|---|
| HumanEval (coding) | 50 | 100% |
| MATH-500 (competition math) | 54 | 89% |
| Reasoning & Logic | 2 | 100% |
| Terminal/CLI | 2 | 100% |
| SWE (bug fixing) | 2 | 100% |
| Cybersecurity | 2 |
Generation speed: ~62 tokens/s at batch_size=1.
text
Python 3.12CUDA 12.88x GPU with 24+ GB each (tested on RTX 3090)text
vllm==0.19.0torch==2.10.0+cu128transformers==4.57.6flashinfer-python==0.6.6flashinfer-cubin==0.6.6quack-kernels==0.3.10nvidia-cutlass-dsl==4.4.2nvidia-cutlass-dsl-libs-base==4.4.2triton==3.6.0xgrammar==0.1.33conch-triton-kernels==1.3bash
uv venv vllm-env --python 3.12uv pip install --python vllm-env/bin/python3 'vllm==0.19.0' conch-triton-kernelsThe tokenizer_config.json shipped with this model uses "tokenizer_class": "Qwen2Tokenizer". If you encounter tokenizer errors, verify this field is set correctly:
python
import jsonwith open("tokenizer_config.json") as f: cfg = json.load(f)cfg["tokenizer_class"] = "Qwen2Tokenizer"with open("tokenizer_config.json", "w") as f: json.dump(cfg, f, indent=2)bash
vllm serve 0xSero/Qwen3.5-264B-W4A16 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --enable-prefix-caching \ --max-model-len 262144 \ --max-num-seqs 4 \ --gpu-memory-utilization 0.9 \ --kv-cache-dtype fp8_e4m3 \ --dtype bfloat16 \ --trust-remote-code \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choice \ --served-model-name qwen35-264bsetpci -s <addr> ECAP_AER+0x08.l=0xFFFFFFFF on affected slots.--max-cudagraph-capture-size 256 or --enforce-eager.python
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")response = client.chat.completions.create( model="qwen35-264b", messages=[{"role": "user", "content": "Solve: what is the integral of x^2 * e^x dx?"}], max_tokens=8192,)print(response.choices[0].message.content)python
import base64response = client.chat.completions.create( model="qwen35-264b", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_image}"}}, {"type": "text", "text": "What's in this image?"} ] }], max_tokens=4096,)python
tools = [{ "type": "function", "function": { "name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}} }}]response = client.chat.completions.create( model="qwen35-264b", messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], tools=tools,)License inherited from the base model.
bibtex
@misc{lasby2025reap, title = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression}, author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa}, year = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}}Made possible by NVIDIA · TNG Technology · Lambda · Prime Intellect · Hot Aisle.
Qwen3.5-28B | BF16 | link |
Qwen3.5-35B-EXL3-4bpw | EXL3-4bpw | link |
Qwen3.5-76B | BF16 | link |
Qwen3.5-76B-GGUF | GGUF | link |
Qwen3.5-88B | BF16 | link |
Qwen3.5-99B | BF16 | link |
Qwen3.5-99B-GGUF | GGUF | link |
| 100% |
| Philosophy | 2 | 100% |
| MMLU (general knowledge) | 2 | 100% |