GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
RukaRat
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
RukaRat
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
FP8 needs compute capability 8.9 or higher, so on Ampere cards like the 3090, A6000 and A100 it isn't available and INT8 W8A8 is the practical choice. Worth checking what you're on:
bash
nvidia-smi --query-gpu=compute_cap --format=csv
If you're on Ada or newer, an FP8 build will probably serve you better than this one.
I read this off the tensor dtypes rather than trusting the config:
| module group | dtypes | size |
|---|---|---|
| MLP | 192 INT8 + 192 BF16 | 15.94 GiB |
| linear_attn (48 layers) | 432 BF16 + 144 INT8 | 5.21 GiB |
| norms / misc | 130 BF16 | 2.37 GiB |
lm_head | BF16 | 2.37 GiB |
| full_attn (16 layers) | 96 BF16 + 64 INT8 | 1.56 GiB |
| vision tower | 333 BF16 | 0.86 GiB |
| MTP | 15 BF16 | 0.79 GiB |
The MLP and full-attention projections carry most of the quantization. The vision
tower stays BF16 throughout, and the Gated DeltaNet linear_attn layers are only
partly quantized, since in_proj_a, in_proj_b and norm are excluded. That's
where most of that 5.21 GiB sits. lm_head and MTP are excluded too.
It's a fairly conservative recipe. You could get this model smaller, though in my experience you tend to give up speculative decoding or vision along the way, and for a two card setup the tradeoff didn't seem worth it.
One thing dominates memory planning on this model, and it isn't the KV cache.
48 of the 64 layers are Gated DeltaNet linear attention. Each concurrent sequence needs its own recurrent state, and unlike a KV cache that state is a fixed cost that doesn't shrink for short prompts:
markdown
48 value heads x 128 key dim x 128 value dim x 4 bytes (fp32) = 3 MiB per layer3 MiB x 48 linear-attention layers = 144 MiB per sequenceat TP=2 = ~72 MiB per GPU per sequence
Both engines default to a concurrency well above what a single-user setup needs, and on 2x24 GB that reservation is the difference between starting and not:
mamba_full_memory_ratio (default 0.9), which
handed roughly half the cache arena to DeltaNet state on my box. Setting
--max-mamba-cache-size 8 freed most of it.--max-num-seqs defaults to 128, which is several GB per card gone
before a single KV token is allocated, on top of a ~15 GiB weight share. Set it
to 1 or 2.If you hit an OOM, or get a much smaller context than you expected, look here
first — before --gpu-memory-utilization.
bash
python -m sglang.launch_server \--model-path RukaRat/Qwen3.8-27B-INT8-W8A8-imatrix-MTP \--tp 2 \--speculative-algorithm NEXTN \--speculative-num-steps 3 \--speculative-eagle-topk 1 \--speculative-num-draft-tokens 4 \--context-length 245760 \--mem-fraction-static 0.95 \--max-running-requests 2 \--max-mamba-cache-size 8 \--mamba-ssm-dtype bfloat16 \--kv-cache-dtype fp8_e4m3 \--cuda-graph-max-bs 2 \--tool-call-parser qwen3_coder \--reasoning-parser qwen3 \--default-chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "low"}'
Out of the box I was capped at a much smaller context than the card advertises. Four flags got me to 245,760, tested one at a time:
--max-mamba-cache-size 8 with --kv-cache-dtype fp8_e4m3 — by far the
biggest win, and it's the per-sequence state described above. Speculative
decoding consumes roughly 4 state slots per request, so 8 is the floor. Fewer
slots also shrink the mamba radix prefix cache, which is what gives you warm
starts on multi-turn sessions, so don't go below it.--mem-fraction-static 0.95 — I stress-tested vision at this setting (1024px
image at deep context) and it held, with well under a gigabyte free. Don't go
higher.--mamba-ssm-dtype bfloat16 — worth a useful chunk of context. Needle recall
stayed perfect at 105k, 215k and 240k tokens, and generation stayed coherent.
Those are light probes rather than a parity proof, so if you see long-session
weirdness, revert this one first.Two things that didn't help: --speculative-num-draft-tokens 3 gained nothing,
and YaRN is moot here, since the pool sits below the model's native 262,144
window. The cap is memory-bound, not rope-bound.
The first request after boot takes the better part of a minute for CUDA graph capture, and the first request at any new depth also runs slow. Discard it when benchmarking.
I've moved to SGLang and don't run vLLM day to day any more, so treat this as last known-good rather than current — it was last measured on vLLM 0.22.0. What it buys you is context, 311,296 against SGLang's 245,760, and better prefill at short depth. SGLang was faster on decode at every depth I compared, and a quality A/B on identical weights came out a dead heat.
bash
vllm serve RukaRat/Qwen3.8-27B-INT8-W8A8-imatrix-MTP \--tensor-parallel-size 2 \--trust-remote-code \--max-num-seqs 1 \--disable-custom-all-reduce \--gpu-memory-utilization 0.92 \--hf-overrides '{"text_config":{"rope_parameters":{"rope_type":"yarn","factor":1.5,"original_max_position_embeddings":262144,"mrope_interleaved":true,"mrope_section":[11,11,10],"partial_rotary_factor":0.25,"rope_theta":10000000}}}' \--max-model-len 311296 \--kv-cache-dtype fp8_e4m3 \--enable-prefix-caching \--enable-chunked-prefill \--max-num-batched-tokens 4096 \--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \--limit-mm-per-prompt '{"image":4,"video":0}' \--mm-processor-kwargs '{"max_pixels":2000000,"min_pixels":65536}' \--enable-auto-tool-choice \--tool-call-parser qwen3_coder \--reasoning-parser qwen3 \--default-chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "low"}'
Three things trip people up:
--max-num-seqs 1. See the per-sequence state section above. This is the
flag that decides whether the server starts at all on 2x24 GB.--max-model-len exceeds max_position_embeddings. Keep
mrope_section, partial_rotary_factor and rope_theta in there or the model
won't load. Extending this way also seemed to help deep-context behaviour
rather than hurt it: at 1.5x, position 250k sits where roughly 167k would land
natively, comfortably inside the trained range.--disable-custom-all-reduce on PCIe. vLLM's custom all-reduce kernels
assume NVLink.That config reported around 345,620 tokens of KV cache on 2x24 GB, and I set
--max-model-len to 311,296 to leave room, since MTP spends cache on the drafter
too. Read the figure off your own startup output rather than trusting mine, and
set --max-model-len just under it. --max-num-batched-tokens 4096 beat 8192 and
16384 for me, both of which cost KV cache without buying prefill back — vLLM's own
startup warning telling you to raise it is wrong on this hardware, at least when
serving one request at a time.
The vision tower has 2,304 position embeddings and each merged token covers 1,024 px, which works out to roughly 2.36 MP per image. Above that — anything much over 1536x1536 — you get:
markdown
Mismatch in `image` token count between text and `input_ids`
On vLLM, --mm-processor-kwargs '{"max_pixels":2000000,"min_pixels":65536}' caps
it, and setting it also gives you back a chunk of KV cache, since vLLM otherwise
reserves room for a worst-case image.
The chat template defaults to xhigh, the maximum. I default to low instead and
raise it per request when a task calls for it. On a self-grading build task —
the model writes a unittest suite and the harness runs it — low came out close
to xhigh on tests passed while taking a fraction of the time and a small
fraction of the reasoning tokens.
Two things worth knowing:
medium is not a middle setting. The template injects no instruction for it,
whereas low explicitly says "move directly to the conclusion". So low gets a
clear directive and follows it, while medium drifts, and it scored worst of
the three for me.xhigh can talk itself out of the right answer. On that build task it rejected
the stdlib zoneinfo module every time and hand-rolled a datetime.tzinfo
subclass that then crashed; low and medium imported zoneinfo every time.
The xhigh instruction says to "consider plausible alternatives", which on a
problem the stdlib already solves is a liability. I also tried it as an agentic
default and reverted the same day — the think-to-output ratio made sessions
unworkable.Either way, give it plenty of max_tokens. This model thinks at length, and if it
runs out mid-thought you get reasoning back with empty content.
The chat template hides historical reasoning before the last real user query and keeps it after. That boundary moves every time you send a message, so assistant turns that rendered with their reasoning last turn render without it this turn. Prefix caching needs append-only prompts, so this invalidates the cache from the first rewritten block onward.
It only bites during tool-call loops, where several assistant messages sit between two user messages. Plain turn-taking never moves the boundary, which is why a chat-shaped test shows nothing and a tool-shaped one reproduces it.
preserve_thinking: true keeps reasoning always, which makes prompts append-only
and measurably improved cache hit rate, re-prefill and wall-clock time on a
multi-turn tool-calling workload. It costs context, since retained reasoning is
carried forward, so if you hit the ceiling on long sessions, drop this first.
The repo ships Qwen's official 3.8 chat_template.jinja, so you shouldn't need to
pass a template at all. Don't substitute the custom Qwen3.6 one if you have it
lying around.
markdown
temperature 0.6 · top_k 20 · top_p 0.86 · repetition_penalty 1.07presence_penalty 0.0 · frequency_penalty 0.0
I arrived at these by scoring 8-gram repetition at depth rather than by feel.
Worth flagging that Qwen's documented anti-repetition advice
(presence_penalty 1.5) was the worst setting I tested — it repeated more, and it
suppressed natural stopping so replies ran to the token cap.
Avoid temperature 0. Qwen notes that greedy decoding degrades thinking mode, and that matched what I saw: reasoning that loops with no output at the end.
3 draft tokens is my default on both engines, and MTP is a large decode win across the whole usable range. Acceptance did not decay with depth in my testing.
Acceptance is task-dependent, not context-dependent. Generating code drafts well. A needle test measured near zero, because no drafter can predict an arbitrary random string. A low acceptance reading means the workload is unpredictable, not that MTP has broken.
Qwen3.8 ships one trained MTP layer, so there's no adding capacity through configuration. Acceptance also seems bounded by the architecture: those 48 Gated DeltaNet layers carry a compressed recurrent state, so a rejected draft can't be rolled back the way you'd truncate a KV cache suffix. Qwen3.6-27B reached noticeably higher acceptance on the same harness, and I couldn't close the gap — I tried quantization strategy, draft sampling, N of 2/3/5, mamba state dtypes, KV dtypes, and a nightly meant to address this directly.
Before you benchmark any of this: accepted draft tokens arrive batched into a
single SSE chunk. A harness that does ntok += 1 per chunk undercounts
speculative decoding by roughly the acceptance factor, and will tell you MTP made
things slower when it didn't. Every "MTP is slower on my machine" result I had
turned out to be this. Count tokens server-side.
markdown
format compressed-tensors, W8A8weights 8-bit int · symmetric · per-channel · static · observer: imatrix-mseactivations 8-bit int · symmetric · per-token · dynamictargets Linearignore all model.visual.* blocks · linear_attn in_proj_a / in_proj_b / norm· lm_head · re:.*mtp.*
Calibration was 512 sequences of roughly 2,000 tokens each, half of them containing tool calls, built from public open source Python (the vLLM and ComfyUI trees). I'm not shipping the corpus itself, but the description above should be enough to rebuild something equivalent.
imatrix-mse weights the quantization error by how much each activation actually
matters, rather than minimizing average weight error uniformly. It needs
calibration data and pipeline="sequential". If either is missing, llmcompressor
falls back to a data-free pipeline, logs
markdown
imatrix_mse: no importance data available. Falling back to uniform MSE
I built the same base model five ways on one machine and harness and compared MTP
acceptance. imatrix-mse and minmax came out about even at the top, W8A16 and
SmoothQuant behind them, and plain mse last — it clips the activation outliers
the drafter depends on. So: don't "fix" this build to plain mse.
AutoModelForImageTextToText. With AutoModelForCausalLM you get
the language model on its own and no vision tower, and there's no error to
tell you.mtp.* tensors back from
the base checkpoint afterwards. They stay BF16 with no scales, which is
expected.re:.*mtp.* to quantization_config.ignore. Without it vLLM reads those
BF16 tensors as INT8 and speculative decoding sits at 0% acceptance without
complaining, or throws KeyError: 'weight_scale'. After fixing it, clear
~/.cache/vllm/torch_compile_cache, otherwise vLLM reuses the graph it
compiled under the broken config and the fix looks like it did nothing. That
one cost me an hour.Version drift is real in this area and flag names keep moving between releases. Everything above was on Ubuntu 22.04, driver 580 (CUDA 13), 2x RTX 3090 at TP=2 with no NVLink, one card on an x4 chipset slot:
| SGLang | vLLM | |
|---|---|---|
| engine | 0.5.18, sglang-kernel 0.4.6.post1 | 0.22.0 |
| torch | 2.13.0+cu130 | 2.11.0+cu130 |
| flashinfer | 0.6.17 | 0.6.11.post2 |
| transformers | 5.12.1 | 5.9.0 |
Both are hand-built venvs rather than container images. Check flag names against your own version rather than pasting mine.
One host-side note if you're near the RAM line: building the MTP drafter re-reads
the whole 29 GiB checkpoint, so a box with ~31 GB of RAM will spike hard enough
for systemd-oomd to kill the load. Running the server in its own scope with
ManagedOOMPreference=omit fixed that for me. The kernel OOM killer still
applies, which is the point.
lm_head, MTP and a good chunk of
linear_attn are still BF16.Same license as the base model, Qwen/Qwen3.8-27B, which is Apache 2.0. The base repo has the authoritative terms. All the credit goes to the Qwen team, this is just a quant of their work.