antonyMox

Qwen3.6-27B-abliterated-AutoRound-INT4-MTP

README

License: apache-2.0

Highlights

Benchmarks — decensoring cost nothing, and where it sits

Same base Qwen/Qwen3.6-27B, one harness. Decensoring left hard unchanged (70 = 70 vs the reference narrow quant) and tool-use went up (83 → 87):

Table with columns: Quant of Qwen3.6-27B (our harness), Recipe, hard, tools, Size
Quant of Qwen3.6-27B (our harness)	Recipe	hard	tools	Size
Lorbus narrow INT4 (reference)	narrow	70	83	18.5 GB
➡️ This — abliterated	narrow + Heretic	70	87	19 GB
wide sister	wide	78	83	26 GB
35B-A3B (coming soon)	MoE narrow	48	83	22 GB

[!NOTE] Numbers are % of tasks passed (40 hard coding + 30 tool tasks; raw pass/total in benchmark-results/). Conditions: temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 · seed 42 · non-thinking · vLLM 0.19 · RTX 4090 (48 GB) · MTP n=4. Solutions run in a restricted sandbox (no network) — safe, identical scoring. The Lorbus row is our own measurement of their public quant on the same tasks, not their claim. Reproduce it, don't trust it.

Surgically gentle abliteration

Abliteration strips the refusal direction — done carelessly it also scrambles the model's brains. Heretic co-minimizes refusals and KL divergence from the original. Lower KL = closer to the untouched model = intelligence kept. We tuned for minimum drift:

Table with columns: Approach, KL divergence ↓ (gentler), Refusals
Approach	KL divergence ↓ (gentler)	Refusals
Typical manual abliteration	0.45 – 1.04	low
Heretic — flagship showcase (Gemma-3-12B)	0.16	3/100
➡️ This model (Qwen3.6-27B)	`0.0089`	11/100

~18× below Heretic's own showcase, 50–100× below manual. The model barely moved from the base, so <think> reasoning and coding stayed intact. (KL is model/harness-dependent, so cross-model figures aren't a strict race — the order-of-magnitude gap is the point.) We kept 11/100 refusals as the price of rock-bottom drift — brains first.

Kept in BF16 (not quantized) — surgical precision

Table with columns: Component, Tensors, Why
Component	Tensors	Why
🧬 MTP head	15× `mtp.*`	makes speculative decoding work
👁️ Vision tower	333× `visual.*`	full image/video, untouched by quant
🌊 Mamba/GDN control	`A_log`, `conv1d`, `dt_bias`, `in_proj_a/b`, `norm`

Everything heavy is 4-bit; only the fragile bits that carry reasoning, speed and vision stay full-precision.

Speed — MTP speculative decoding

Honest 768-token × 3-run measurement on a single RTX 4090 (48 GB), vLLM 0.19:

Table with columns: MTP n, short tok/s, 28k-ctx tok/s, acceptance @28k
MTP `n`	short tok/s	28k-ctx tok/s	acceptance @28k
1	72	39	78%
2	91	46	66%
4 ⭐	115	49	46%
5	119	45

Use num_speculative_tokens: 4. Speculative decoding is lossless — n changes speed only, never output. All presets use n=4.

Multimodal — sees images & video

The full vision-language model, not a text-only trim. The entire vision tower (333 tensors) is kept in full BF16 — quantization never touched it — so image/video understanding is identical to the pristine base. Serve with --limit-mm-per-prompt '{"image":4,"video":1}'. Published benchmarks cover text/coding/tool-use; vision quality is inherited, not separately re-scored.

Quickstart — pick your GPU

Table with columns: GPU, VRAM, Script, max-model-len, gpu-mem-util
GPU	VRAM	Script	`max-model-len`	`gpu-mem-util`
RTX 3090 / 4090	24 GB	`run_24gb.sh`	40 960	0.95
RTX 5090	32 GB

bash
vllm serve antonyMox/Qwen3.6-27B-abliterated-AutoRound-INT4-MTP \
  --quantization auto-round \
  --max-model-len 40960 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
  --trust-remote-code

Long context is cheap here: Qwen3.6 is a hybrid (Mamba/GDN + attention), so KV costs only ~37 MB/1k tokens at fp8. The model's native maximum is 256k (262,144) tokens — how much of it fits depends on how much VRAM you give vLLM:

Context vs mem-util (48 GB card, measured)

Table with columns: gpu-mem-util, context that fits, ~free VRAM, notes
`gpu-mem-util`	context that fits	~free VRAM	notes
0.48	~48k	~25 GB	leaves lots of room for a 2nd model
0.60 ⭐ (default preset)	~200k	~19 GB	balanced — room for another model on the same card
0.65	~256k	~17 GB	full native context
0.68	256k · conc 1.30x

Our default preset is a deliberate 0.60 / 200k — it keeps ~19 GB free so you can load a second model on the same card. Want the full 256k? Just raise --gpu-memory-utilization to ~0.65–0.68.

⚠️ Pitfall — never set `--max-model-len` at exact KV capacity

[!WARNING] On vLLM 0.19 hybrid models: if max-model-len equals KV capacity (startup log shows Maximum concurrency ... : 1.00x), requests in the last ~3 % of the window pass validation then hang forever — no response, no error, nothing logged, GPU idle. The scheduler can't allocate the final KV block (draft tokens need headroom the validator ignores). Above the window → instant HTTP 400 (fine); below the danger zone → fine. Only the top sliver dead-locks.

[!TIP] Keep --max-model-len ≥ 2 attention blocks (2 × 1632) below capacity — our presets already do. Check the log: Maximum concurrency for N tokens: 1.13x must be ≥ ~1.10x, not 1.00x. Also: the log line GPU KV cache size: X tokens understates real capacity ~3× on hybrids (it divides by KV-cache groups) — trust only the Maximum concurrency line.

Long-context recall (measured)

Needle test on a 150k-token prompt (48 GB preset), exact-quote recall by depth:

Table with columns: Depth, 1%, 25%, 40%, 46%, 50%, 56%, 75%, 100%
Depth	1%	25%	40%	46%	50%	56%	75%	100%
Recall	✅	✅	✅	✅	⚠️*	✅	✅	✅

7/8 exact. The miss (*) at 50% returned the neighbouring line — mild "lost-in-the-middle", not a hallucination. Prefix caching works: a repeat 150k query answered in 5.5 s vs 131 s cold.

🧬 The family

Clean quants of the same base — pick by hardware:

wide · 26 GB · hard 78 🏆 — maximum reasoning, for 48 GB+ cards.
abliterated · 19 GB · hard 70 (this one) — uncensored, fits 24 GB cards.
35B-A3B · 22 GB · hard 48 (coming soon) — fast MoE (~188 tok/s).

All ours: AutoRound INT4, MTP preserved, multimodal, published benchmarks. Measured against the community reference Lorbus narrow INT4 (hard 70) — see the comparison table above.

Recipe

Base — pristine Qwen/Qwen3.6-27B BF16.
Abliteration — Heretic, gentle: KL 0.0089, refusals 11/100, <think> intact.
Quant — AutoRound W4A16, group 128, 200 iters, dense. BF16-kept: mtp.*, visual.*, Mamba control.
MTP — 15 mtp.* tensors grafted from the pristine BF16.
Sampler — ships temp 1.0 / top_p 0.95 / top_k 20 (generation_config.json).

Why HuggingFace shows "~6.7B params"

Not a small model — it's how the widget reads a packed INT4 quant. The 800 quantized tensors pack 8× int4 into each int32, so HF counts 3.06 B int32 slots, not the ~24 B real int4 weights. With the BF16 parts, the true model is Qwen3.6-27B, 19 GB. Every AutoRound/GPTQ INT4 looks like this. Verified: language_model 4.55 B · other 1.27 B · vision 0.46 B · mtp 0.42 B.

⚠️ Usage & responsibility

[!CAUTION] This is an uncensored research artifact — read before use.

Reduced refusals by design — can produce content a safety-aligned model would decline.
No safety guarantees — outputs are not filtered; review before use, especially in production.
You are responsible — for lawful, ethical use in your jurisdiction; all consequences rest with you.
No liability — provided "as is" as a community artifact, no warranty of any kind.
Not for harm — for research, evaluation, and lawful applications only.

License

Apache-2.0, same as the base. Quantized & abliterated from Qwen/Qwen3.6-27B. Community build; benchmark numbers from our own harness — JSON included, methodology open.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

antonyMox

Model Tree

Base

Qwen/Qwen3.6-27B

Quantized

this model

Input Modalities

Text

Image

Video

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Highlights

Benchmarks — decensoring cost nothing, and where it sits

Same base Qwen/Qwen3.6-27B, one harness. Decensoring left hard unchanged (70 = 70 vs the reference narrow quant) and tool-use went up (83 → 87):

Table with columns: Quant of Qwen3.6-27B (our harness), Recipe, hard, tools, Size
Quant of Qwen3.6-27B (our harness)	Recipe	hard	tools	Size
Lorbus narrow INT4 (reference)	narrow	70	83	18.5 GB
➡️ This — abliterated	narrow + Heretic	70	87	19 GB
wide sister	wide	78	83	26 GB
35B-A3B (coming soon)	MoE narrow	48	83	22 GB

[!NOTE] Numbers are % of tasks passed (40 hard coding + 30 tool tasks; raw pass/total in benchmark-results/). Conditions: temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 · seed 42 · non-thinking · vLLM 0.19 · RTX 4090 (48 GB) · MTP n=4. Solutions run in a restricted sandbox (no network) — safe, identical scoring. The Lorbus row is our own measurement of their public quant on the same tasks, not their claim. Reproduce it, don't trust it.

Surgically gentle abliteration

Table with columns: Approach, KL divergence ↓ (gentler), Refusals
Approach	KL divergence ↓ (gentler)	Refusals
Typical manual abliteration	0.45 – 1.04	low
Heretic — flagship showcase (Gemma-3-12B)	0.16	3/100
➡️ This model (Qwen3.6-27B)	`0.0089`	11/100

Kept in BF16 (not quantized) — surgical precision

Table with columns: Component, Tensors, Why
Component	Tensors	Why
🧬 MTP head	15× `mtp.*`	makes speculative decoding work
👁️ Vision tower	333× `visual.*`	full image/video, untouched by quant
🌊 Mamba/GDN control	`A_log`, `conv1d`, `dt_bias`, `in_proj_a/b`, `norm`

Everything heavy is 4-bit; only the fragile bits that carry reasoning, speed and vision stay full-precision.

Speed — MTP speculative decoding

Honest 768-token × 3-run measurement on a single RTX 4090 (48 GB), vLLM 0.19:

Table with columns: MTP n, short tok/s, 28k-ctx tok/s, acceptance @28k
MTP `n`	short tok/s	28k-ctx tok/s	acceptance @28k
1	72	39	78%
2	91	46	66%
4 ⭐	115	49	46%
5	119	45

Use num_speculative_tokens: 4. Speculative decoding is lossless — n changes speed only, never output. All presets use n=4.

Multimodal — sees images & video

Quickstart — pick your GPU

Table with columns: GPU, VRAM, Script, max-model-len, gpu-mem-util
GPU	VRAM	Script	`max-model-len`	`gpu-mem-util`
RTX 3090 / 4090	24 GB	`run_24gb.sh`	40 960	0.95
RTX 5090	32 GB

bash
vllm serve antonyMox/Qwen3.6-27B-abliterated-AutoRound-INT4-MTP \
  --quantization auto-round \
  --max-model-len 40960 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
  --trust-remote-code

Context vs mem-util (48 GB card, measured)

Table with columns: gpu-mem-util, context that fits, ~free VRAM, notes
`gpu-mem-util`	context that fits	~free VRAM	notes
0.48	~48k	~25 GB	leaves lots of room for a 2nd model
0.60 ⭐ (default preset)	~200k	~19 GB	balanced — room for another model on the same card
0.65	~256k	~17 GB	full native context
0.68	256k · conc 1.30x

Our default preset is a deliberate 0.60 / 200k — it keeps ~19 GB free so you can load a second model on the same card. Want the full 256k? Just raise --gpu-memory-utilization to ~0.65–0.68.

⚠️ Pitfall — never set `--max-model-len` at exact KV capacity

[!WARNING] On vLLM 0.19 hybrid models: if max-model-len equals KV capacity (startup log shows Maximum concurrency ... : 1.00x), requests in the last ~3 % of the window pass validation then hang forever — no response, no error, nothing logged, GPU idle. The scheduler can't allocate the final KV block (draft tokens need headroom the validator ignores). Above the window → instant HTTP 400 (fine); below the danger zone → fine. Only the top sliver dead-locks.

[!TIP] Keep --max-model-len ≥ 2 attention blocks (2 × 1632) below capacity — our presets already do. Check the log: Maximum concurrency for N tokens: 1.13x must be ≥ ~1.10x, not 1.00x. Also: the log line GPU KV cache size: X tokens understates real capacity ~3× on hybrids (it divides by KV-cache groups) — trust only the Maximum concurrency line.

Long-context recall (measured)

Needle test on a 150k-token prompt (48 GB preset), exact-quote recall by depth:

Table with columns: Depth, 1%, 25%, 40%, 46%, 50%, 56%, 75%, 100%
Depth	1%	25%	40%	46%	50%	56%	75%	100%
Recall	✅	✅	✅	✅	⚠️*	✅	✅	✅

🧬 The family

Clean quants of the same base — pick by hardware:

wide · 26 GB · hard 78 🏆 — maximum reasoning, for 48 GB+ cards.
abliterated · 19 GB · hard 70 (this one) — uncensored, fits 24 GB cards.
35B-A3B · 22 GB · hard 48 (coming soon) — fast MoE (~188 tok/s).

All ours: AutoRound INT4, MTP preserved, multimodal, published benchmarks. Measured against the community reference Lorbus narrow INT4 (hard 70) — see the comparison table above.

Recipe

Base — pristine Qwen/Qwen3.6-27B BF16.
Abliteration — Heretic, gentle: KL 0.0089, refusals 11/100, <think> intact.
Quant — AutoRound W4A16, group 128, 200 iters, dense. BF16-kept: mtp.*, visual.*, Mamba control.
MTP — 15 mtp.* tensors grafted from the pristine BF16.
Sampler — ships temp 1.0 / top_p 0.95 / top_k 20 (generation_config.json).

Why HuggingFace shows "~6.7B params"

⚠️ Usage & responsibility

[!CAUTION] This is an uncensored research artifact — read before use.

Reduced refusals by design — can produce content a safety-aligned model would decline.
No safety guarantees — outputs are not filtered; review before use, especially in production.
You are responsible — for lawful, ethical use in your jurisdiction; all consequences rest with you.
No liability — provided "as is" as a community artifact, no warranty of any kind.
Not for harm — for research, evaluation, and lawful applications only.

License

Apache-2.0, same as the base. Quantized & abliterated from Qwen/Qwen3.6-27B. Community build; benchmark numbers from our own harness — JSON included, methodology open.

Qwen3.6-27B-abliterated-AutoRound-INT4-MTP

README

Highlights

Benchmarks — decensoring cost nothing, and where it sits

Surgically gentle abliteration

Kept in BF16 (not quantized) — surgical precision

Speed — MTP speculative decoding

Multimodal — sees images & video

Quickstart — pick your GPU

Context vs mem-util (48 GB card, measured)

⚠️ Pitfall — never set --max-model-len at exact KV capacity

Long-context recall (measured)

🧬 The family

Recipe

Why HuggingFace shows "~6.7B params"

⚠️ Usage & responsibility

License

Explore FriendliAI today

README

Highlights

Benchmarks — decensoring cost nothing, and where it sits

Surgically gentle abliteration

Kept in BF16 (not quantized) — surgical precision

Speed — MTP speculative decoding

Multimodal — sees images & video

Quickstart — pick your GPU

Context vs mem-util (48 GB card, measured)

⚠️ Pitfall — never set --max-model-len at exact KV capacity

Long-context recall (measured)

🧬 The family

Recipe

Why HuggingFace shows "~6.7B params"

⚠️ Usage & responsibility

License

⚠️ Pitfall — never set `--max-model-len` at exact KV capacity

⚠️ Pitfall — never set `--max-model-len` at exact KV capacity