GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
writetoasik
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
writetoasik
Model Tree
orcarouter/Qwen3.8-27B-Uncensored
Input Modalities
Output Modalities
Supported Functionality
The parent weights had safety alignment substantially removed by abliteration (refusal-direction orthogonalization). This quant inherits that:
Qwen3.8-27B would refuse.Qwen/Qwen3.8-27B still applies. Abliteration and this quant do not change the license.By downloading you accept the above.
-hb 6)-mb 4) — TabbyAPI draft_mode: mtp worksConverted and served on a Razer Blade 16 (2025) RTX 5090 Laptop, 24 GB GDDR7, ExLlamaV3 1.4.3, TabbyAPI, AC power.
| TabbyAPI settings | GPU memory |
|---|---|
| Weights only (approx.) | ~15.7 GB |
max_seq_len / cache_size 49152, cache_mode: "8,8", chunk_size: 512, max_batch_size: 1 | ~16.2 GB / 24 GB |
max_seq_len / cache_size 262144, cache_mode: Q4, chunk_size: 4096 | ~21.4 GB / 24 GB |
cache_mode is ExLlamaV3 k_bits,v_bits. "8,8" first; "6,6" then "4,4" (legacy Q4) only if you need a longer window on 24 GB. cache_size must be ≥ max_seq_len and a multiple of 256. max_seq_len is prompt + response.
eval/perf.py -cs 16384 -max_length 16384, chunk 4096:
| Prefill length | tok/s |
|---|---|
| 256 | 844 |
| 1024 | 1307 |
| 4096 | 1450 |
| 16384 | 1351 |
| Decode context | tok/s |
|---|---|
| 0 | 42.1 |
| 4096 | 41.3 |
| 8192 | 40.5 |
| 16128 | 39.2 |
bash
hf download writetoasik/Qwen3.8-27B-Uncensored-exl3-4.0bpw --local-dir ./Qwen3.8-27B-Uncensored-exl3-4.0bpw
In config.yml (24 GB starting point):
yaml
model:model_dir: /path/to/modelsmodel_name: Qwen3.8-27B-Uncensored-exl3-4.0bpwbackend: exllamav3max_seq_len: 49152cache_size: 49152cache_mode: "8,8"chunk_size: 512max_batch_size: 1vision: falsereasoning: truereasoning_start_token: "<think>"reasoning_end_token: "</think>"tool_format: qwen3_5template_vars_default:enable_thinking: truedraft_model:draft_mode: mtp
Then:
markdown
http://127.0.0.1:5000/v1/chat/completions
If a long prompt appears stuck, drop chunk_size to 256 and keep max_batch_size: 1. Do not set RoPE scale unless you know you need it; this model already trains to 262K.
Need the full native window on 24 GB: max_seq_len / cache_size 262144 and cache_mode: "4,4" (or Q4). Quality of the KV cache is worse than "8,8".
python
from exllamav3 import Config, Model, Tokenizer, Generator, Jobmodel_dir = "Qwen3.8-27B-Uncensored-exl3-4.0bpw"cfg = Config.from_directory(model_dir)model = Model.from_config(cfg)model.load()tok = Tokenizer.from_config(cfg)gen = Generator(model, tok)prompt = tok.encode("Hello.")job = Job(input_ids=prompt, max_new_tokens=128)gen.enqueue(job)print(tok.decode(gen.iterate()[0]["token_ids"]))
See ExLlamaV3 examples.
text
python convert.py \-i orcarouter/Qwen3.8-27B-Uncensored \-w ./_exl3_work \-o ./Qwen3.8-27B-Uncensored-exl3-4.0bpw \-b 4.0 -hb 6 -mb 4
| Flag | Value | Meaning |
|---|---|---|
-b | 4.0 | body bits per weight |
-hb | 6 | lm_head bits |
-mb | 4 | MTP bits |
quantization_config.json in this repo is the converter's record (method exl3, version 1.4.3).
All refusal / capability numbers below are OrcaRouter's measurements on the BF16 (and its FP8), not re-run on this EXL3. Treat them as properties of the source, not a claim that 4.0 bpw is bit-identical in quality.
Refusal, thinking off (lower = less refusal): AdvBench 0.0%, JailbreakBench 0.0%, StrongREJECT 2.0%, HarmBench 2.7%, MaliciousInstruct 0.0% vs 94–99% on stock Qwen3.8-27B.
Capability vs stock (same scripts): MMLU 84.7% (+0.4), MMLU-Pro 76.8% (−0.8), GSM8K 88.7% (−1.3), CMMLU 80.8% (−0.6). WikiText-2-raw PPL 6.96 on the BF16.
Full write-up: orcarouter/Qwen3.8-27B-Uncensored.
Apache 2.0, inherited from Qwen/Qwen3.8-27B.
Redistribution must keep the Apache 2.0 license text (this repo includes LICENSE).