GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
twolven
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
twolven
Model Tree
JonathanColetti/Qwen3.8-27B-Uncensored
Input Modalities
Output Modalities
Supported Functionality
qwen3_5 is a hybrid architecture and a VLM:
| Layers | 64 — 48 linear_attention (Qwen3_5GatedDeltaNet) + 16 full_attention, every 4th |
| Vision | 27-layer tower (left in bf16) |
| Context | 262,144 |
| MTP | mtp_num_hidden_layers: 1, kept unquantized in model-mtp.safetensors |
Left in bf16 (not quantized): the whole vision tower (model.visual.*), lm_head,
mtp.*, and in_proj_a / in_proj_b — the 48-wide per-head scalars driving the delta rule,
where 4-bit is destructive and saves nothing. conv1d is not an nn.Linear so AWQ never
touches it.
v_proj → o_proj smoothing is deliberately skipped: GQA means v_proj emits 1024 while
o_proj takes 6144 (q_proj is doubled to 12288 by attn_output_gate), so the shapes do not
line up for a channel-wise scale.
All figures are single-stream (c=1), 600 output tokens, thinking off, measured over an HTTP round-trip with the same harness — so they are directly comparable to each other.
Note on absolute numbers. Throughput on these builds is workload-dependent by roughly 30%: MTP acceptance is far higher on predictable text than on varied prose. Measured 2026-08-21 on the sibling HauhauCS AWQ build under a pinned protocol — same box, same settings, same day — an expository prompt gives 95.9 t/s where creative prose gives 74.3 t/s. The figures in this section were all taken in one session with one harness, so they are internally comparable; but the prompt was not recorded, so do not quote the absolute values against numbers measured elsewhere. The relative comparisons (W4A16 vs FP8, MTP on vs off) are unaffected.
| This model (W4A16) | Official FP8 | |
|---|---|---|
| c=1 throughput | 84 t/s median (best 93) | 57 t/s |
| Weights / GPU | 9.37 GiB | 15.11 GiB |
| KV cache | 560,900 tokens | 266,537 |
| Max concurrency @ 262,144 | 2.14× | 1.02× |
| MTP acceptance length | 2.96 / 3 (~60%) | 2.80 |
Measured on this model, same harness, only --speculative-config changed:
| MTP state | median | vs no-MTP |
|---|---|---|
| Disabled | 57.9 t/s | 1.00× |
| Enabled (working) | 84 t/s | 1.45× |
| Enabled but weights not loading | 33 t/s | 0.57× |
That third row is the failure in trap 3 below, and it is worth internalising: a broken MTP is slower than no MTP at all, because you pay the full drafting cost (~98 t/s of drafted tokens) and accept none of it. It presents as a mysterious throughput regression, not as a loading error. Disabling MTP entirely also buys you a larger KV cache (677,958 tokens, 2.59× at 262,144) if context depth matters more to you than latency.
Verified after quantization: 250,060-token needle retrieval; tool calling (correct JSON arguments); the abliteration survived; and vision still works — the model correctly described shapes and colours and read embedded text from a test image. The vision tower is left in bf16, so image quality should be unchanged from the source.
Not measured: no perplexity or benchmark comparison against the bf16 source was run. W4 does cost some accuracy relative to bf16/FP8; this card does not quantify it. Behaviour was verified, quality regression was not.
Do not use greedy decoding (temperature: 0) with this model. It will emit the same
sentence over and over until it hits your token cap and never produce a stop token. This is
the classic Qwen3 + quantization degenerate-repetition mode, not a defect in the weights,
and it is fully reproducible: greedy is deterministic, so the same prompt loops the same way
every time.
generation_config.json in this repo now ships a repetition_penalty of 1.05, which is
enough to prevent it. If your stack ignores generation_config.json, set it yourself:
| parameter | thinking mode | notes |
|---|---|---|
repetition_penalty | 1.05 | the important one — do not set below 1.02 |
temperature | 1.0 (repo default) or 0.6 | never 0 |
top_p | 0.95 | |
top_k | 20 |
Measured on this build, greedy worst case, 6000-token cap:
repetition_penalty | finish reason | sentence uniqueness | max verbatim repeats |
|---|---|---|---|
| unset | length — never stopped | 0.85 | 5 |
| 1.02 | stop | 1.00 | 1 |
| 1.05 | stop | 1.00 |
The penalty is not free — it costs about 6% throughput. Interleaved A/B/A/B on the same box, thinking disabled, 600-token generations, c=1: 68.4 / 67.1 t/s median with the penalty off versus 64.3 / 63.6 t/s at 1.05. That is the price of not looping; we think it is worth paying by default, and you can lower it to 1.02 if you would rather have the throughput.
It does not otherwise cost you anything measurable. Checked at 1.05 against penalty-off on the tasks a repetition penalty is most likely to damage — all identical:
| check | penalty off | 1.05 |
|---|---|---|
| verbatim reproduction of repetition-heavy JSON | EXACT | EXACT |
| generated FizzBuzz, executed and asserted | PASS | PASS |
| 12 near-identical repeated assignment lines | PASS | PASS |
| needle retrieval @ ~9k tokens | PASS | PASS |
| tool calling, streaming and non-streaming | PASS | PASS |
Do not go much above 1.10 — that is where repeated code syntax and identifiers start being penalized.
presence_penalty will not work as a server-side default in vLLM. vLLM only carries a
fixed whitelist out of generation config into its default sampling params —
repetition_penalty, temperature, top_k, top_p, min_p, max_new_tokens
(ModelConfig.get_diff_sampling_param). A presence_penalty entry is silently ignored.presence_penalty and frequency_penalty cannot be defaulted at all over the OpenAI
API, because the OpenAI schema defaults them to 0.0 — clients always send them
explicitly, so your default never applies. repetition_penalty defaults to None on the
request and is filled from the server's defaults, which is why it is the only lever that
survives a client setting its own .Coding agents commonly pin temperature: 0 for determinism and send no penalty at all. Two
checked as of 2026-08-21: one sends no temperature when thinking is enabled but hardcodes
temperature: 0 on its sub-agent and skill paths; the other sends no temperature and no
penalty of any kind, ever. In both cases every request rides entirely on the server defaults.
chat_template_medium.jinjaThis model's stock chat template defaults reasoning_effort to xhigh, which spends the
whole budget thinking and returns an empty answer on a large fraction of requests. The
default chat_template.jinja in this repo is unmodified upstream — we did not silently
change the behaviour of a redistributed artifact.
For convenience an opt-in copy is included as chat_template_medium.jinja, byte-identical
except for one line, which defaults reasoning_effort to medium instead:
diff
-{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}+{%- set resolved_reasoning_effort = reasoning_effort|default('medium') %}
Use it with --chat-template chat_template_medium.jinja, or keep the stock template and pin
the value per-request with --default-chat-template-kwargs as shown below. Both work; the
flag merges per-key, so clients sending their own chat_template_kwargs stay covered.
Needs a vLLM recent enough to register Qwen3_5ForConditionalGeneration and the
Qwen3_5MTP proposer. This was validated on vllm/vllm-openai:nightly (v0.20.2rc1.dev129,
compressed-tensors 0.15.0.1 in-image — it reads the 0.18.0 config fine).
bash
vllm serve twolven/Qwen3.8-27B-abliterated-AWQ-MTP \--tensor-parallel-size 2 \--max-model-len 262144 \--kv-cache-dtype fp8 \--enable-prefix-caching \--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
This is what produced the numbers in the table above.
bash
vllm serve /path/to/Qwen3.8-27B-abliterated-AWQ-MTP \--served-model-name qwen3.8-27b-abliterated \--tensor-parallel-size 2 \--max-model-len 262144 \--gpu-memory-utilization 0.95 \--max-num-seqs 4 \--max-num-batched-tokens 4096 \--kv-cache-dtype fp8 \--performance-mode interactivity \--mm-encoder-tp-mode data \--disable-custom-all-reduce \--reasoning-parser qwen3 \--default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "medium"}' \--enable-auto-tool-choice \--tool-call-parser qwen3_coder \--enable-prefix-caching \--enable-chunked-prefill \--compilation-config '{"cudagraph_mode": "PIECEWISE"}' \--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
With NCCL_P2P_DISABLE=1 in the environment.
Why these particular flags — each one is load-bearing on this hardware:
| Flag | Reason |
|---|---|
--gpu-memory-utilization 0.95 | 0.98 overshoots if anything else holds VRAM (a desktop session is enough). At 0.95 the full 262K context still fits at 2.14× concurrency. |
--kv-cache-dtype fp8 | The hybrid layout is cheap on KV — 262K needs only ~4.6 GiB — but fp8 is what buys the 2.14×. |
--compilation-config PIECEWISE | FULL cudagraph replay segfaults with MTP (vllm#40756). PIECEWISE is the same c=1 speed and keeps MTP. |
NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce | Dual 3090 over PCIe without NVLink. |
reasoning_effort — the default will truncate your answersThe chat template accepts reasoning_effort of xhigh / medium / low, and defaults to
xhigh:
jinja
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
xhigh is pathological for general serving. Measured over 4 hard reasoning prompts with a
16,384-token budget:
reasoning_effort | mean thinking | mean answer | thinking share | requests that produced an answer |
|---|---|---|---|---|
xhigh (default) | 11,888 tok | 792 tok | 93.8% | 2 of 4 |
medium | 2,092 tok | 4,548 tok | 31.5% | 4 of 4 |
At xhigh, half the requests spent the entire budget thinking and returned an empty
answer (finish_reason: length). Note low is counterintuitively worse than medium —
it thinks ~58% more — so medium is the sweet spot, not merely a midpoint.
This is not a repetition loop and not a quantization artifact. A truncated xhigh trace
analysed at 675/676 unique sentences, with only 1.7% of positions inside any repeated 8-gram
and 0% repetition in the final 400 words — the model is generating novel reasoning right to
the cutoff. Re-running an identical truncated prompt with a 40,000-token budget terminated
cleanly (stop, 11,756 thinking + 2,913 answer). The failure mode is variance: xhigh's
thinking length straddles typical budgets, so the same prompt sometimes finishes at 14k and
sometimes blows past 16k and returns nothing.
Pin it server-side:
bash
--default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "medium"}'
Clients can still opt up per request with
chat_template_kwargs: {"reasoning_effort": "xhigh"} — just give them a budget of 40k+ if
they do. Thinking can be turned off entirely with {"enable_thinking": false}.
If you enable --tool-call-parser qwen3_coder, vLLM's parser has a streaming bug where the
<tool_call> tag is emitted as prose and subsequent SSE chunks go silent. We run a small
patch that buffers <tool_call> tokens until <function= appears within 64 characters.
Patch and details: club-3090 →
models/qwen3.6-27b/vllm/patches/local/. Non-streaming tool calls are unaffected.
bash
# should report ~560,900 tokens and 2.14x at 262,144grep -E "GPU KV cache size|Maximum concurrency" <container logs># MTP must actually load — if you see "not found in params_dict" the drafter is# running on random weights and acceptance will be 0% (see trap 3 below)grep "Detected MTP model" <container logs># after some traffic, acceptance length should be ~2.9 of 3grep "SpecDecoding metrics" <container logs>
The compose files, the quantization script, and the deployment notes this model came out of live in club-3090.
match_modules_set() accumulates matches
until every pattern fires, so a generic re:.*input_layernorm$ → q/k/v mapping piles up
four smooth layers across three consecutive linear_attention blocks (which have no
q_proj) and aborts with "AWQ needs to match a single smoothlayer". Emit one mapping per
layer index — 192 over 64 layers.Qwen3_5GatedDeltaNet.forward's signature. Its
force_accelerate_hooks decorator is def wrapped(self, *args, **kwargs) with no
functools.wraps, so inspect.signature() reports (*args, **kwargs); AWQ's arg cache
then stores {'args': …} and replay dies with "missing 1 required positional argument:
'hidden_states'". Recover the inner function from the wrapper closure and re-attach
__signature__.Quantized by Todd Wolven - Lead AI Software Developer and open-source GenAI engineer.
Other projects and writeups | GitHub | Hugging Face
Apache-2.0, inherited from Qwen/Qwen3.8-27B. The abliteration is
JonathanColetti's; this repo contributes only the quantization.
This model is uncensored. Refusal behaviour has been removed by the upstream abliteration and that property survives quantization. You are responsible for how you use it.
| 1 |
| 1.10 | stop | 1.00 | 1 |
temperature--performance-mode interactivity | Favours single-stream latency. Costs aggregate throughput at high concurrency — drop it if you serve many users. |
--tool-call-parser qwen3_coder | See the caveat below. |
low | 3,314 tok | 3,251 tok | 50.5% | 4 of 4 |
save_pretrained writes a single
model.safetensors with no index, so a copied-in MTP file is never opened, the drafter
loads random weights, and you get 0% draft acceptance — which presents as a speed bug
(~33 t/s) rather than a load bug. Write a model.safetensors.index.json covering both
files, and add re:.*mtp.* to quantization_config.ignore or vLLM builds MTP as quantized
and demands weight_packed. After changing that, clear the torch.compile cache or a stale
AOT graph throws KeyError: 'weight_packed'.offload_device="cpu" on
AWQModifier and drop to 128×1024.