GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
JRamirez-UAB
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
JRamirez-UAB
Model Tree
Jackrong/Qwopus3.6-35B-A3B-Coder
Input Modalities
Output Modalities
Supported Functionality
| Branch | Size |
|---|---|
main | 18.9 GB of weights |
3.75bpw | 17.9 GB of weights |
Download a branch with hf download JRamirez-UAB/Qwopus3.6-35B-A3B-Coder-EXL3-4.00bpw-24GB --revision <branch> --local-dir <dir>.
| Revision | bpw | Weights | KLD internal · public | GPU | Context | Decode with MTP, 1 / 2 sessions (tok/s) | Prefill 62k → 237k (tok/s) | Needle | Vision | Tools A · B · C (of 30) | Reasoning xhigh (48 problems) | Really free at peak |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
main | 4.00 | 19.0 GB | 0.0042 · 0.0053 | RTX 3090 | 262,144 | 118.6 / 207.9 | 2,386 → 1,243 | 8/8 at 237k | 14/15 | 22 · 28 · 30 | 54,606 (47/48) | 1,061 MiB |
3.75bpw | 3.75 | 18.0 GB | 0.0057 · 0.0072 | RTX 3090 | 262,144 | 108.9 / 204.2 |
How to read it:
qbench on an internal sysadmin/agent evaluation
set (not published) and on turboderp's public trace (qbench_prompts_gen.json). Lower is better.xhigh at the model's default temperature, B = xhigh at T=0.3,
C = thinking off at T=0.3. Reasoning: total reasoning tokens over 6 checkable problems × 8 tries (T=0.6), and how many were correct.opencode run, 4 sysadmin tasks × 2 with real tool execution): 6/8 (both).--max-model-len 262144, fp8 KV cache, MTP 1, 2 sequences, prefix caching, 1 image per request (≤1 MP). RTX 3090 rows: KV 3.5e9 B (287,575-token pool).Measured on our EXL3 4.00 build of the same model (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens (a loop = more than 30 % repeated 8-grams).
| Temperature | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|---|---|---|---|---|---|---|---|
| Loops / 30 | 13 | 9 | 14 | 10 | 7 | 11 | 2 | 1 |
Loops are frequent up to 0.8 (7–14 of 30) and almost disappear at 0.9–1.0: keep the default 1.0. Other models on the same test:
| Model (same test, vLLM) | T 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|---|---|---|---|---|---|---|---|
| NeoHorse-1-9B | 2 | 3 | 3 | 1 | 2 | 0 | 0 | 0 |
| Ornith-1.5-9B | 5 | 7 |
EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:
7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers and
INT8 embed_tokens / packed lm_head for qwen3_5 models.qwen38_exl3_vision 0.2.1 (ours, in plugin/qwen38_exl3_vision/): EXL3 MoE experts (moe.py: the per-expert trellis tensors are
stacked and run with exllamav3's fused MoE kernel, CUDA-graph friendly), the EXL3 vision tower, padding to multiples of 128 and dense fp16 layers converted once to the
model dtype.Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:
bash
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Qwopus3.6-35B-A3B-Coder-EXL3-4.00bpw-24GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionWe checked these exact steps on a fresh machine (A40): plugin 0.2.0 (0.2.1 only changes one condition in moe.py, checked on vLLM 0.30 with this Quick start) loads our dense Swift-1.5 27B EXL3 checkpoint (downloaded from Hugging Face) and Qwen3.6-35B-A3B 4.00 (MoE), with the same results as our own build.
bash
export CUDA_HOME=/usr/local/cuda-13.2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_DISABLE_SHARED_EXPERTS_STREAM=1vllm serve JRamirez-UAB/Qwopus3.6-35B-A3B-Coder-EXL3-4.00bpw-24GB \ --served-model-name qwopus3.6-35b-a3b-coder \ --limit-mm-per-prompt '{"image":1,"video":0}' \ --mm-processor-kwargs '{"max_pixels":1003520}' \ --max-model-len 262144 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 3500000000 \ --max-num-batched-tokens 1024 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \ --compilation-config '{"max_cudagraph_capture_size":16}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder3.75bpw branch: add --revision 3.75bpw (same flags; validated as in the table).On 48 GB GPUs raise --kv-cache-memory-bytes (10e9 B gives an 821k-token pool: room for 8 sessions; add --max-num-seqs 8). For more sessions, raise max_cudagraph_capture_size with them
(see Important notes: ≥ max-num-seqs × 2 with MTP 1).
How we use it as a coding / sysadmin agent from opencode (opencode 1.18.x). limit.context must match --max-model-len of your
vllm serve command (262144 in the Quick start above).
json
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "qwopus3-6-35b-a3b-coder-exl3-4-00bpw-24g": { "name": "Qwopus3.6-35B-A3B-Coder-EXL3-4.00bpw-24GB (local)", "tool_call": true, "temperature": true, "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 262144, "output": 22000 } } } } }, "agent": { "build": { "model": "local-vllm/qwopus3-6-35b-a3b-coder-exl3-4-00bpw-24g", "temperature": 1.0 }, "plan": { "model": "local-vllm/qwopus3-6-35b-a3b-coder-exl3-4-00bpw-24g", "temperature": 1.0 } }}"temperature": true in the model, otherwise opencode sends no temperature to a custom model and vLLM falls back to its defaults.generation_config.json default.output: 22000 opencode compacts the history at ≈context − 22k tokens of input.opencode run from scripts, close stdin (</dev/null).Sampling (fixed 2026-10-09): the source model has no generation_config.json, so vLLM sampled the whole vocabulary (top_p 1, no top_k) when the client sent no sampling parameters; this repo now ships Qwen3.6's defaults (temperature 1.0, top_p 0.95, top_k 20). If you downloaded before, fetch generation_config.json again. Use it with thinking off, as intended by its author: "chat_template_kwargs": {"enable_thinking": false} (or opencode's none variant). Tool calling was 30/30 that way, against 22-28/30 at xhigh; with thinking on it reasons ≈55-65k tokens on our 48 problems.
VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 is required. The fused exllamav3 MoE kernel synchronises its blocks through global counters and needs the whole GPU;
vLLM 0.30 otherwise runs the shared expert on a parallel CUDA stream and decoding hangs after a few seconds. The plugin sets it, but export it yourself too.max_cudagraph_capture_size ≥ max-num-seqs × (1 + MTP tokens), so that every decode step is replayed from a graph (16 is enough for the
2 sessions of the Quick start). Plugin 0.2.1 (2026-10-13) runs every step of up to 256 tokens in exllamav3's fused MoE kernel; 0.2.0 only did so up to
32 tokens (8 experts per token × tokens ≤ 256) and fails at start with a larger capture size. RTX 3090, vLLM 0.30, 32 sequences, MTP 1, short prompts:
0.2.0 (graphs ≤ 32) and 0.2.1 (≤ 128) give the same 106 / 378 / 615 tok/s in total with 1 / 8 / 16 requests at once; with 32 at once, 499 → 587 tok/s.
Graphs plus MTP 1 matter more than the plugin: with MTP 3 and graphs ≤ 16, 8-32 requests ran 1.7-1.8× slower. KL divergence 0.2.1 vs 0.2.0: 0.0155,
the same as between two runs of 0.2.0 (0.0152). On 24 GB use at most 128 (256 does not fit while capturing).nvidia-smi --query-gpu=memory.free. Size the KV cache
with --kv-cache-memory-bytes, not . The KV cache of these models is small (≈13 KB/token in fp8: only 10 of the 40 layers are
full attention, with 2 KV heads), which is why 262k fits next to the 19 GB of weights.sc_measure --shaped, --max-sys 24) on a 196 × 2048-token calibration trace,
bit allocation with sc_optimize, conversion with the same trace. The trace was generated by Qwen3.8-27B (same tokenizer) on prompts from exllamav3's SC set
plus internal Linux sysadmin/agent conversations (not published); with a trace generated by the model itself the KLD was not better (Ornith 4.00: 0.0171 vs 0.0163).lm_head 6 bits, vision tower 6 bits, MTP head 4 bits, codebook mul1, out_scales: always.scripts/embed_int8.py from the converted checkpoint (≈−0.5 GB of VRAM, no measurable change).Jackrong/Qwopus3.6-35B-A3B-Coder revision 4ba785ca. Chat template and tokenizer are the base model's.Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).
qbench evaluation, public trace from turboderp's repos);
OrcaSAQ2-kernel (EXL3 in vLLM); vLLM.| 2,388 → 1,237 |
| 8/8 at 237k |
| 15/15 |
| 25 · 30 · 30 |
| 65,294 (43/48) |
| 2,065 MiB |
| 5 |
| 2 |
| 1 |
| 0 |
| 0 |
| 0 |
| Swift 1.5 27B (EXL3 3.70) | 4 | 5 | 4 | 2 | 2 | 2 | 1 | 1 |
| Qwen3.8-27B (EXL3 3.70) | 14 | 9 | 10 | 5 | 4 | 5 | 2 | 0 |
| Qwen3.8-27B-pi (EXL3 3.70) | 6 | 12 | 7 | 5 | 5 | 4 | 0 | 0 |
| Qwopus3.8-27B-Flash-V2 (EXL3 3.70) | 5 | 1 | 1 | 3 | 1 | 1 | 0 | 1 |
| Qwen3.6-35B-A3B (EXL3 4.00) | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 2 |
| Qwopus3.6-35B-A3B-Coder (EXL3 4.00) | 13 | 9 | 14 | 10 | 7 | 11 | 2 | 1 |
| Ornith-1.5-35B-A3B (EXL3 4.00) | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 |
| Swift 1.5 Qwen3.8-Flash-Next (EXL3 2.25) | — | — | 22 | 16 | 11 | — | — | 0 |
--gpu-memory-utilizationquantization_config.bits not being an integer is expected (EXL3 bits is an average).--tool-call-parser qwen3_coder and --reasoning-parser qwen3. reasoning_effort accepts xhigh, medium and low; they are prompt instructions, not limits.