GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
JRamirez-UAB
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
JRamirez-UAB
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
| Revision | bpw | Weights | KLD internal · public | GPU | Context | Decode with MTP, 1 / 2 sessions (tok/s) | Prefill 62k → 237k (tok/s) | Needle | Vision | Tools A · B · C (of 30) | Reasoning xhigh (48 problems) | Really free at peak |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
main | 4.00 | 19.0 GB | 0.0066 · 0.0082 | RTX 3090 | 262,144 | 120.6 / 204.9 | 2,459 → 1,261 | 8/8 at 237k | 15/15 | 28 · 30 · 30 | 133,765 (39/48) | 1,065 MiB |
3.50bpw | 3.50 | 16.9 GB | 0.0113 · 0.0143 | RTX 3090 | 262,144 | 117.8 / 204.5 | 2,431 → 1,248 | 8/8 at 237k | 15/15 | 27 · 30 · 30 | 133,827 (40/48) | 3,069 MiB |
How to read it:
qbench on an internal sysadmin/agent evaluation
set (not published) and on turboderp's public trace (qbench_prompts_gen.json). Lower is better.xhigh at the model's default temperature, B = xhigh at T=0.3,
C = thinking off at T=0.3. Reasoning: total reasoning tokens over 6 checkable problems × 8 tries (T=0.6), and how many were correct.opencode run, 4 sysadmin tasks × 2 with real tool execution): 6/8 (4.00) · 7/8 (3.50).--max-model-len 262144, fp8 KV cache, MTP 1, 2 sequences, prefix caching, 1 image per request (≤1 MP). RTX 3090 rows: KV 3.5e9 B (287,575-token pool).Qwen3.6 reasons a lot. On our 48 short checkable problems it spends ≈130k reasoning tokens at any reasoning_effort and some answers run into the cap (39-41/48 correct). Both bitrates behave the same, so this comes from the model, not from the quantization. Cap it with thinking_token_budget: with "thinking_token_budget": 512 in the request the same set took 22,737 reasoning tokens and 47/48 correct in 241 s (4.00 bpw); 1024 → 41,880 (45/48); 2048 → 72,375 (47/48). It requires --reasoning-parser qwen3 (enforced by vLLM, also with MTP).
EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:
7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers and
INT8 embed_tokens / packed lm_head for qwen3_5 models.qwen38_exl3_vision 0.2.0 (ours, in plugin/qwen38_exl3_vision/): EXL3 MoE experts (moe.py: the per-expert trellis tensors are
stacked and run with exllamav3's fused MoE kernel, CUDA-graph friendly), the EXL3 vision tower, padding to multiples of 128 and dense fp16 layers converted once to the
model dtype.Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:
bash
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Qwen3.6-35B-A3B-EXL3-4.00bpw-24GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionWe checked these exact steps on a fresh machine (A40): plugin 0.2.0 loads our dense Swift-1.5 27B EXL3 checkpoint (downloaded from Hugging Face) and Qwen3.6-35B-A3B 4.00 (MoE), with the same results as our own build.
bash
export CUDA_HOME=/usr/local/cuda-13.2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_DISABLE_SHARED_EXPERTS_STREAM=1vllm serve JRamirez-UAB/Qwen3.6-35B-A3B-EXL3-4.00bpw-24GB \ --served-model-name qwen3.6-35b-a3b \ --limit-mm-per-prompt '{"image":1,"video":0}' \ --mm-processor-kwargs '{"max_pixels":1003520}' \ --max-model-len 262144 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 3500000000 \ --max-num-batched-tokens 1024 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \ --compilation-config '{"max_cudagraph_capture_size":16}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder3.50bpw branch: add --revision 3.50bpw (same flags; validated as in the table).On 48 GB GPUs raise --kv-cache-memory-bytes (10e9 B gives an 821k-token pool: room for 8 sessions; add --max-num-seqs 8).
VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 is required. The fused exllamav3 MoE kernel synchronises its blocks through global counters and needs the whole GPU;
vLLM 0.30 otherwise runs the shared expert on a parallel CUDA stream and decoding hangs after a few seconds. The plugin sets it, but export it yourself too.max_cudagraph_capture_size 16): the fused kernel handles up to 128-256 rows per expert; larger batches (prefill) take a
slower path that reconstructs the expert weights.nvidia-smi --query-gpu=memory.free. Size the KV cache
with --kv-cache-memory-bytes, not --gpu-memory-utilization. The KV cache of these models is small (≈13 KB/token in fp8: only 10 of the 40 layers are
full attention, with 2 KV heads), which is why 262k fits next to the 19 GB of weights.quantization_config.bits not being an integer is expected (EXL3 is an average).sc_measure --shaped, --max-sys 24) on a 196 × 2048-token calibration trace,
bit allocation with sc_optimize, conversion with the same trace. The trace was generated by Qwen3.8-27B (same tokenizer) on prompts from exllamav3's SC set
plus internal Linux sysadmin/agent conversations (not published); with a trace generated by the model itself the KLD was not better (Ornith 4.00: 0.0171 vs 0.0163).lm_head 6 bits, vision tower 6 bits, MTP head 4 bits, codebook mul1, out_scales: always.scripts/embed_int8.py from the converted checkpoint (≈−0.5 GB of VRAM, no measurable change).Qwen/Qwen3.6-35B-A3B revision 995ad96e. Chat template and tokenizer are the base model's.Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).
qbench evaluation, public trace from turboderp's repos);
OrcaSAQ2-kernel (EXL3 in vLLM); vLLM.bits--tool-call-parser qwen3_coder and --reasoning-parser qwen3. reasoning_effort accepts xhigh, medium and low; they are prompt instructions, not limits.