GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
JRamirez-UAB
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
JRamirez-UAB
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
| Model | Size | Context in 24 GB | KLD mean / median | Perplexity (BF16: 1.5860 · pub. 1.3332) | Needle · tools at T=1.0 · T=0.3 · agent | Reasoning xhigh (tokens) | Prefill | Decode with MTP, 1 / 2 sessions | Tokens per step | Accepted drafts (of 3) |
|---|---|---|---|---|---|---|---|---|---|---|
W4A16 GPTQ + INT4 embeddings (mtp-int8, this repo) | 14.2 GiB | 222,000 (text) · 176,128 (vision) | 0.0400 / 0.0127 · pub. 0.0312 / 0.0018 | 1.6631 · pub. 1.3779 | 8/8 at 205,792 · 60/60 · 60/60 · 8/8 | 26,481 | 586 tok/s | 94.4 / 187 tok/s | 3.07–3.10 | 2.07–2.10 |
EXL3 3.85 bpw (main) | 13.64 GiB | 215,000 (with vision) | 0.0073 / 0.0018 · pub. 0.0067 / 0.0003 | 1.5976 · pub. 1.3430 | 8/8 at 200,069 · 59/60 · 60/60 · 8/8 | 23,221 | 539 tok/s | 75.9 / 135.8 tok/s | 3.03 | 2.03 |
EXL3 3.70 bpw (3.70bpw) | 13.22 GiB | 226,000 (with vision) | 0.0089 / 0.0023 · pub. 0.0085 / 0.0004 | 1.5999 · pub. 1.3466 | 8/8 at 210,602 · 60/60 · 60/60 · 8/8 | 21,912 | 526 tok/s | 81.4 / 139.2 tok/s | TBD† | TBD† |
| Reference, not ours: turboderp EXL3 SC 4.00 bpw H5 V6 | 15.26 GiB | not tested | 0.0087 / 0.0024 · pub. 0.0062 / 0.0003 | 1.6017 · pub. 1.3424 | not tested | not tested | — | — | — | — |
| Reference, not ours: turboderp EXL3 SC 3.00 bpw H4 V4 + our INT8 embeddings | 10.94 GiB (12.12 as published) | 262,144 (with vision) | 0.0343 / 0.0098 · pub. 0.0257 / 0.0013 | 1.6432 · pub. 1.3658 | 8/8 at 245,166 · 15/15 (5 tries per case, default sampling) · not measured | 18,526 | 479 tok/s | 75.4 / 131.2 tok/s | 2.84* | 1.84* |
How to read it:
qbench. The first
figure uses an internal sysadmin/agent evaluation set (not published; separate from the calibration data, ≈30 % of it with tool calls); pub.
uses turboderp's public qbench_prompts_gen.json trace (from turboderp/Qwen3.8-27B-exl3),
so anyone can reproduce it. Noise floor on the public trace (BF16 against itself): 0.0005. The W4A16 checkpoint was evaluated as a BF16 copy holding
the dequantized INT4 × scale weights (Transformers, full load); the EXL3 ones with exllamav3. Lower is better. Perplexity is on the same texts.--max-model-len validated with MTP 3, 2 sequences, fp8 KV cache and ≥100 MiB of VRAM really free at peak.xhigh, at T=1.0 (no temperature sent, so vLLM uses the model's generation_config.json) and at T=0.3 (what opencode sends with the
configuration below). The only misses (all at T=1.0) were answers taken from the history ("the folder is empty") instead of a new tool call.
: 4 sysadmin tasks × 2 with and real tool execution.In short: the EXL3 versions (3.85 and 3.70 bpw) are ≈5× closer to the BF16 model (KLD) and reason less on the problems where the base model runs long, at the cost of ≈20 % lower decode speed on Ampere; this W4A16 checkpoint is the fastest and needs only a small patch on stock vLLM.
| Revision | MTP head | Size | Notes |
|---|---|---|---|
main | BF16 (0.79 GiB) | 14.6 GiB | reference; use without speculation for the longest context |
mtp-int8 | INT8 g128 (0.40 GiB) | 14.2 GiB | recommended with MTP: frees ≈0.4 GiB for KV cache |
Sibling repositories of the same family: EXL3 3.85 bpw (+ branch 3.70bpw), see the table above.
| Component | Treatment |
|---|---|
Linear layers of the language model (attention, MLP, GatedDeltaNet in_proj_qkv / in_proj_z / out_proj) | GPTQ W4A16, symmetric INT4, group size 128, actorder=static, dampening 0.01 |
embed_tokens, lm_head | INT4 symmetric, group size 128, RTN (min-max) |
GatedDeltaNet in_proj_a, in_proj_b, norms, A_log, dt_bias, |
HuggingFaceH4/ultrachat_200k (the first 512 of the train_sft split, shuffled with seed 42; chat template applied). Only the GPTQ linear layers use it: embed_tokens / lm_head (INT4) and the INT8 MTP head of the mtp-int8 revision are round-to-nearest, with no data.1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0; quantization took 26.9 min on a single RTX PRO 6000 Blackwell (96 GB).compressed-tensors pack-quantized (Marlin kernels).Stock vLLM (checked on 0.29.0 and 0.30.0) does not pass quant_config to the embedding layer of qwen3_5 models, so a checkpoint with
quantized embeddings fails to load (no parameter named 'embed_tokens.weight_packed'). Apply the 4-line patch in
patches/qwen3_5-embed-quant.patch (from syv-ai/qwen38-27b-rtx3090, Apache-2.0):
bash
uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.29.0" --torch-backend cu130 \ --extra-index-url https://flashinfer.ai/whl/ --index-strategy unsafe-best-matchVLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patchWith MTP (mtp-int8 revision), text only, 222k context, 2 concurrent sequences — validated as a service on a GPU used only for the LLM:
bash
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwen3.8-27B-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \ --language-model-only \ --max-model-len 222000 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 8640000000 \ --max-num-batched-tokens 1024 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coderWith the two memory settings this leaves 363 MiB of VRAM really free at peak (see Important notes). If the GPU also drives a display or runs anything else, use the 200k variant below:
Same, 200k context (more VRAM headroom):
bash
vllm serve JRamirez-UAB/Qwen3.8-27B-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \ --language-model-only \ --max-model-len 204800 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 8160437862 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coderWith MTP and vision (mtp-int8 revision), 172k context, 2 concurrent sequences, 1 image per request (≤1 MP):
bash
vllm serve JRamirez-UAB/Qwen3.8-27B-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \ --limit-mm-per-prompt '{"image":1,"video":0}' \ --mm-processor-kwargs '{"max_pixels":1003520}' \ --max-model-len 176128 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 7140000000 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder8 concurrent sessions (text, shared 200k KV pool): the 222k text command with --max-model-len 200000 --kv-cache-memory-bytes 8150000000 --max-num-seqs 8
(validated, 751 MiB really free at peak; see Benchmarks). With 8 sequences vLLM 0.29 runs out of memory while capturing CUDA graphs at ≥222k.
Without MTP, the same layout fits the full 262k context with --max-model-len 262144 --kv-cache-memory-bytes 9101639680 --max-num-seqs 4
(measured on the fine-tune quantized with the same recipe; not re-measured on this checkpoint).
nvidia-smi --query-gpu=memory.used,memory.free --format=csv. The original 222k command left only ≈50–70 MiB really free, and the same
configuration crashed once with out-of-memory on the Swift 1.5 checkpoint (same recipe and size) during an agent request. The 222k command
above now adds --max-num-batched-tokens 1024 and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: measured on this checkpoint the peak
drops to 23,760 MiB (363 MiB really free) with the same prefill (586 tok/s at 205,792 tokens, 8/8 facts) and decode speed, no OOM.
(The "really free" values in the 8-session rows of Benchmarks were read directly with nvidia-smi.)--kv-cache-memory-bytes, not --gpu-memory-utilization. With this hybrid (GatedDeltaNet) model, vLLM 0.29's
memory profiling under-estimates the prefill activations (it measured 0.4–0.7 GiB; the real peak is ≈2 GiB). With a percentage it
over-allocates the KV cache and runs out of memory on the first long prompt. The values above leave ≈0.9 GB of VRAM free.--tool-call-parser qwen3_coder (Qwen3.8's XML tool-call format).This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). What was measured with opencode:
opencode run: multi-step tasks (list files, count lines, read a file, write a summary) with real tool execution
and streaming.Context, speed and memory figures were measured directly against the vLLM API, not through opencode.
Start vLLM with --served-model-name qwen3.8-27b (the model key in opencode must match it), then add a provider to ~/.config/opencode/config.json:
json
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "qwen3.8-27b": { "name": "Qwen3.8-27B (local)", "tool_call": true, "temperature": true, "limit": { "context": 222000, "output": 22000 }, "variants": { "none": { "chat_template_kwargs": { "enable_thinking": false } }, "low": { "chat_template_kwargs": { "reasoning_effort": "low" } }, "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } }, "xhigh": { "chat_template_kwargs": { "reasoning_effort": "xhigh" } } } } } } }, "agent": { "build": { "model": "local-vllm/qwen3.8-27b", "variant": "xhigh" }, "plan": { "model": "local-vllm/qwen3.8-27b", "variant": "xhigh" } }}Parameters that matter:
limit.context = vLLM's --max-model-len (222,000 in the text configuration above). opencode compacts the history at about
context − output, so output: 22000 makes it compact at ≈200k tokens of input and never exceed the server's limit."temperature": true in the model. Without it opencode does not send any temperature to a custom model, even if the agent sets one
(checked by capturing the requests).chat_template_kwargs; reasoning_effort accepts only xhigh, medium and low (high returns
HTTP 400). They are prompt instructions and do not limit the amount of reasoning (see the notes above).Setup: vLLM 0.29.0 + patch, CUDA 13.0 wheels, driver 595 (CUDA 13.2), fp8 (e4m3) KV cache without calibrated scales, FlashInfer,
--language-model-only, mtp-int8 revision, MTP 3 tokens, 2 sequences, prefix caching, 200k context (209,096-token KV pool).
Decode is measured with 800 forced output tokens (ignore_eos).
| Metric | Result |
|---|---|
| Decode, 1 request (t=0 / t=0.6) | 97.6 / 90.5 tok/s |
| Decode, 2 concurrent (aggregate) | 172.6 tok/s |
| MTP accepted tokens per step (official head) | 2.80–3.05 (mean ≈2.93); per position ≈80-85 % / 59-68 % / 40-52 % |
| 8 facts at 3–97 % depth, 57,874-token prompt | 8/8, 60.8 s (951 tok/s prefill) |
Tool calling, 10 sysadmin tasks (qwen3_coder) | 10/10 called, right tool and valid JSON |
| Tool calling with the real opencode agent request (system prompt + 11 tools; fresh, after a long story, and a full multi-turn session; 5 tries each) | 15/15 (2026-10-02, 20 tries each: 60/60 at T=1.0, 60/60 at T=0.3) |
| Peak VRAM |
Quality sanity checks (small in-house battery): arithmetic, a multi-step word problem, a Python function with asserts, multi-turn memory and a tool call were all correct. A free-form historical summary (Spanish) was largely accurate; a factual question in Catalan about the location of a local university campus was answered incorrectly.
Each configuration below was started as a real service and checked with a near-maximum-length prompt. The VRAM peak barely moves with prompt length (prefill runs in 2048-token chunks): with 222k, idle 24,050 MiB vs. 24,052 MiB at peak.
| Configuration | Context (KV pool) | Checks | Peak VRAM (free) |
|---|---|---|---|
Text, MTP 3 (INT8 head), prefix caching, 2 seqs, qwen3_coder (used daily from opencode) | 222,000 (223,441) | 8/8 facts at 205,792 tokens (348 s, 591 tok/s prefill); decode 94.4 / 88.5 tok/s (t=0 / 0.6), 187 tok/s with 2 concurrent; MTP ≈2.9 accepted tokens per step; real opencode request 5/5 tool calls | 24,052 MiB (524 MiB) |
Vision + MTP 3 (INT8 head), prefix caching, 2 seqs, qwen3_coder | 176,128 (178,923) | real opencode request 5/5 tool calls and a real ls executed from opencode; sysadmin tool calling 10/10; 3/3 test images (terminal screenshot, chart, dense nft ruleset); prefix caching with an image (2.1 s → 1.5 s); ≈1 MP image + 127k-token prompt correct | 23,710 MiB (866 MiB) |
(The 222k row is the configuration before the two memory settings; with them the peak is 23,760 MiB, 363 MiB really free, see Important notes.)
Note: with MTP enabled, KV capacity grows in block-sized steps. With 8.33–8.35 GB of KV cache vLLM accepted at most 212,800 tokens,
so that is the maximum --max-model-len for the text configuration (not 212,992).
Images with the new 4-image battery (2026-10-02, vision configuration: 176,128 context, KV 7.14e9 B, MTP 3, --limit-mm-per-prompt '{"image":1,"video":0}'):
4/4 tasks, 18/18 checks (failed-service screenshot 3/3, chart 3/3, dense firewall ruleset 3/3, port held by another process with a dead PID as a trap 6/6),
follow-up turn with prefix cache OK, 957 MiB really free at the end of the run (not a long-prompt peak), no engine errors.
Sizing tips. With MTP enabled vLLM needs the KV pool to exceed --max-model-len by ≈1.44k tokens (MTP look-ahead plus GatedDeltaNet
state), and capacity grows in block-sized steps, so after changing --kv-cache-memory-bytes always check the GPU KV cache size log line.
The KV pool is shared by all running requests and by the prefix cache: --max-model-len caps a single request, it does not reserve memory per session.
With agent clients that compact the history at context − max_output (e.g. opencode), set the output limit accordingly
(222,000 − 22,000 → compaction at ≈200k tokens of input).
Agent turn = the real opencode request (≈13.5k → 14.5k tokens over 10 turns) with prefix caching. 8 sessions = --max-num-seqs 8, the KV pool
shared by all (it does not multiply the context).
| Configuration | Turn: time to first token / total (median) | First turn (no cache) | 8 sessions: needle each, aggregate prefill | Decode with 8 sessions (t=0 / 0.6) | Really free (min) |
|---|---|---|---|---|---|
| 150,000, KV 6.25e9 | 2.7 / 3.7 s | 11.6 s | 8 × 16.5k, 8/8 in all, 1,129 tok/s | 459.9 / 466.9 tok/s | 2,569 MiB |
| 200,000, KV 8.15e9 | 2.7 / 3.6 s | 11.6 s | 8 × 22k, 8/8 in all, 1,106 tok/s | 472.2 / 463.9 tok/s | 751 MiB |
Measured with exllamav3's qbench (see Family comparison for the method). Internal evaluation set: KLD 0.0400 (median 0.0127), perplexity
1.6631 vs 1.5860 for BF16. Public trace: 0.0312 (median 0.0018), perplexity 1.3779 vs 1.3332. Most of the gap is attributed to the INT4 RTN
embeddings / lm_head (on the Swift sibling an INT8-embedding variant lowered the KLD by ≈12–15 %).
Problems with checkable answers (arithmetic, word problem, counting, logic, subnetting, systemd) × 8 tries, temperature 0.6, top_p 0.95,
reasoning tokens counted with /tokenize, totals over 48 requests (2026-09-30):
| Level | low | medium | xhigh |
|---|---|---|---|
| Reasoning tokens (mean / p90 / max at xhigh: 552 / 1,291 / 2,615) | 17,957 | 21,317 | 26,481 |
| Correct | 48/48 | 48/48 | 48/48 |
Repeat on 2026-10-02: 20,004 / 21,714 / 27,761, 48/48 at every level. Almost all of the xhigh excess comes from one problem (a systemd question:
mean 1,450 and max 2,615 reasoning tokens on 2026-09-30; on turboderp's 3.00 bpw EXL3 the same problem stayed below 1,000). Agent tasks with opencode (4 sysadmin tasks × 2, xhigh, temperature 0.3): 8/8, 4,817 reasoning tokens, 245 s.
Comparison with the fine-tunes: see the Swift 1.5 card.
Instead of the MTP head, this checkpoint also works with the DFlash 2 draft
z-lab/Qwen3.8-27B-DFlash2, quantized for 24 GB GPUs in
JRamirez-UAB/Qwen3.8-27B-DFlash2-W4A16 (INT4 in main, INT8 in w8a16).
Measured on one RTX 3090 (vLLM 0.29.0, 7 speculative tokens, fp8 KV cache, 2 sequences):
| Speculation | Context (KV pool) | Accepted tokens per step (decode · reasoning · tools) | Decode, 1 request (t=0 / 0.6) | 2 requests |
|---|---|---|---|---|
| MTP 3 (INT8 head), this card's text configuration | 222,000 (223,441) | ≈2.9 | 94 / 89 tok/s | 187 tok/s |
| DFlash 2 W4A16 draft, 6.3 GB KV cache | 69,842 | 3.79 · 4.62 · 4.70 | 115 / 109 tok/s | 173 / 175 tok/s |
| DFlash 2 W8A16 draft, 5.5 GB KV cache | 60,854 | 4.00 · 4.36 · 3.90 |
The draft accepts more tokens per step, but on one GPU it gives only +10–25 % with one request and nothing with two, and its KV cache (≈90 KB per token against ≈35 KB with MTP in vLLM 0.29) limits the context to ≈60–70k tokens. The draft is meant for a two-GPU setup, where there is room for a long context; we do not have two GPUs, so that setup has not been tested. Details and commands on the draft's card.
lm_head.scripts/quantize.py: exact quantization script (llm-compressor); recipe.yaml: recipe as serialized by llm-compressor.scripts/mtp_quant.py: script that builds the mtp-int8 revision from main.patches/: vLLM patch and its Apache-2.0 license. SHA256SUMS: checksums of the model files.I would be grateful for reports of installations of this checkpoint, both working and failed ones. Please open a discussion in the Community tab with your GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error message). Failed attempts are just as useful as successful ones: they help to document which setups work.
This work is directly inspired by Marcin116 and their checkpoint Qwen3.8-27B-W4A16-AutoRound-fast-embed-int4, which showed that Qwen3.8-27B can serve very long contexts on RTX 3090-class GPUs by quantizing the token embeddings to INT4. We served and measured that checkpoint on our RTX 3090 before building our own.
What we took from that line of work: the INT4 embed_tokens / lm_head layout (group 128, compressed-tensors pack-quantized),
keeping the GatedDeltaNet in_proj_a / in_proj_b projections in BF16, and the vLLM patch for quantized embeddings in qwen3_5.
Marcin116's checkpoint builds on born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast
and syvai/qwen3.8-27b-3090-fast-variant
(patch: syv-ai/qwen38-27b-rtx3090), developed with the Club3090 community.
Many thanks to all of them. Any mistakes in this checkpoint are ours.
What is different here: GPTQ (llm-compressor) instead of AutoRound for the linear layers, and the MTP head kept in BF16 or INT8 instead of INT4.
qbench from exllamav3 (turboderp) and its public evaluation trace;
reference rows in the family table: .Apache-2.0, same as the base model. The included patch is Apache-2.0 (syv-ai). This is a community quantization, not an official Qwen release.
opencode runxhigh (temperature 0.6); 48/48 correct in every measured row.
The W4A16 value is from 2026-09-30; a repeat on 2026-10-02 gave 27,761 (the model's tokens vary with one systemd problem, see Benchmarks).--max-num-seqs 8); the W4A16 range covers runs with a 150k and a 200k KV pool. *turboderp 3.00: measured on a different workload (needle, decode,
vision, tools), not comparable. †3.70 bpw: no 8-session run yet; on that same validation workload it gave 2.96 (3.85 bpw: 2.93). The value depends on the workload (a pure decode benchmark gave ≈2.9 for W4A16), so compare within the column only.conv1d| kept in BF16 |
MTP head (mtp.*) | BF16 in main; INT8 g128 RTN in mtp-int8 (0.8 % mean relative error; shares the INT4 embeddings / lm_head) |
| Vision tower | kept in BF16 (validated with MTP and prefix caching, see below; use --language-model-only for text-only serving) |
reasoning_effort (via chat_template_kwargs) accepts only xhigh (default), medium and low; enable_thinking: false disables
reasoning. The levels are prompt instructions, not token budgets, and in our measurements they do not regulate the amount of reasoning
(6 checkable problems × 3 tries, temperature 0.6: 7,304 / 8,592 / 8,605 reasoning tokens in total for low / medium / xhigh, 17–18/18 correct).
For a hard cap, send thinking_token_budget in the request: with --reasoning-parser qwen3, vLLM 0.29 enforces it (also with MTP)
by forcing </think> when the budget is reached (e.g. 569 → 199 reasoning tokens with a budget of 200, answer still correct).config.json generated by llm-compressor are adjusted for vLLM (already applied): the lm_head target is
re:.*lm_head$, and the MTP ignore rule is re:^mtp\.(fc|layers\..*)$, so that the MTP shares the quantized embeddings and lm_head.--tool-call-parser qwen3_codergeneration_config.jsonopencode run from a script, close stdin (opencode run "..." </dev/null); otherwise it may wait for input and hang at start-up.| 23,658 MiB of 24,576 (918 MiB free) |
| Text, MTP 3 (INT8 head), prefix caching, 2 seqs | 204,800 (209,096) | 8/8 facts at 58k; decode 97.6 tok/s | 23,658 MiB (918 MiB) |
| 113 / 137 tok/s |
| 170 / 177 tok/s |