abcdsystems

Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw

README

License: apache-2.0

Model Details

Table with columns: Property, Value
Property	Value
Base model	Qwen/Qwen3.5-9B
Abliteration	huihui-ai via remove-refusals-with-transformers
Parameters	9B dense (all active per token)
Architecture	`Qwen3_5ForConditionalGeneration` — GatedDeltaNet + Attention hybrid
Quantization	EXL3 6.0bpw, head bits 6, codebook mul1
Calibration	250 rows × 2048 cols
VRAM loaded	~5.6GB model + KV cache
Total size on disk	~8.5GB
License	Apache 2.0 (Alibaba Cloud)

Architecture — Why This Model Is Different

Qwen3.5 is not a standard transformer. It uses a hybrid attention design that alternates between two fundamentally different attention mechanisms:

GatedDeltaNet Linear Attention (layers 0, 1, 2, 4, 5, 6, 8, 9, 10, ...)

O(n) complexity instead of O(n²) — scales linearly with sequence length
Recurrent state via conv1d kernel (kernel dim 4)
16 key heads × 128 dim, 32 value heads × 128 dim
Gated with learned input/output projections + beta decay

Full Quadratic Attention (layers 3, 7, 11, 15, 19, 23, 27, 31)

Standard grouped-query attention (16 heads, 4 KV heads, head dim 256)
RoPE with partial rotary factor 0.25, theta 10M
Provides global context every 4th layer

Layer Pattern

markdown
[linear, linear, linear, FULL, linear, linear, linear, FULL, ...] × 4 = 32 layers

24 GatedDeltaNet layers + 8 full attention layers. The linear layers handle local/sequential patterns efficiently while the full attention layers provide global context anchoring.

Multi-Token Prediction (MTP)

The model includes a 1-layer MTP head for predicting multiple next tokens simultaneously, enabling faster inference throughput than standard single-token autoregressive decoding.

Text-Only Weights

Despite the architecture string Qwen3_5ForConditionalGeneration (which Alibaba uses for all Qwen3.5 variants including VL), this model contains zero vision weights. No ViT encoder, no image projector. The architecture class name reflects the MTP head and hybrid design, not multimodal capability.

Quantization Details

Quantized with exllamav3 v1.1.0.

Format: EXL3 with trellis-based weight encoding
Bits per weight: 6.0 (all linear layers uniform)
Head bits: 6
Codebook: mul1
Why EXL3 over GGUF: Native tensor core utilization on Ampere GPUs (RTX 3090/4090). GGUF dequantizes to FP16 at runtime, wasting tensor cores. EXL3 operates natively, yielding ~1.5x faster inference at equivalent quality.

Recommended Serving

TabbyAPI (recommended)

yaml
# config.yml
network:
  host: 0.0.0.0
  port: 11435

model:
  model_dir: /path/to/models
  model_name: Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw
  max_seq_len: 4096
  gpu_split: [24]
  cache_mode: FP16

bash
CUDA_VISIBLE_DEVICES=0 python3 main.py --config config.yml

Important: Disable Thinking Mode

Qwen3.5 has a built-in chain-of-thought reasoning mode that is on by default. For real-time applications (voice, chat, API), you must disable it or the model will consume your entire token budget on internal reasoning before producing visible output.

Add to your system prompt:

markdown
/no_think

Or set enable_thinking: false in your chat template parameters.

Note on exllamav3 Raw API

The raw exllamav3 Generator Python API may fail to initialize GatedDeltaNet recurrent state (conv_state on meta device error). TabbyAPI handles this initialization correctly. Use TabbyAPI as the serving layer rather than calling exllamav3 directly.

Hardware Requirements

Table with columns: Config, VRAM, Notes
Config	VRAM	Notes
Minimum	~8GB	Model only, minimal KV cache
Recommended	16GB+	Model + FP16 KV cache for 4096 context
Tested on	RTX 3090 24GB	5.6GB model, 18.4GB free for cache

Performance (tested on RTX 3090)

Response latency: 300–500ms per completion
Suitable for real-time voice pipelines with STT overhead
First request ~14ms (cache warmup, empty response)

Use Cases

This quantization was built for low-latency, single-user dedicated inference — specifically real-time voice AI over SIP/phone calls. The abliteration ensures zero self-censorship in commercial conversation contexts.

Good for:

Real-time voice assistants and phone agents
Single-GPU dedicated inference servers
Applications requiring uncensored, natural conversational output
Latency-sensitive pipelines (sub-600ms budget)

Not ideal for:

Multi-user batched serving (use AWQ + vLLM instead)
Vision/multimodal tasks (no vision weights)
Applications requiring safety guardrails (abliterated model)

Credits

Base model: Qwen/Qwen3.5-9B by Alibaba Cloud (Apache 2.0)
Abliteration: huihui-ai via remove-refusals-with-transformers
EXL3 quantization: ABC&D System Inc using exllamav3 by turboderp
Serving: TabbyAPI by theroyallab

Usage Warnings

This model has had its safety filtering surgically removed (abliterated). It will comply with any instruction without refusal. Users are solely responsible for ensuring their use complies with applicable laws and ethical standards. See huihui-ai's original warnings for full details.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

abcdsystems

Model Tree

Base

huihui-ai/Huihui-Qwen3.5-9B-abliterated

Quantized

this model

Input Modalities

Text

Image

Video

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table with columns: Property, Value
Property	Value
Base model	Qwen/Qwen3.5-9B
Abliteration	huihui-ai via remove-refusals-with-transformers
Parameters	9B dense (all active per token)
Architecture	`Qwen3_5ForConditionalGeneration` — GatedDeltaNet + Attention hybrid
Quantization	EXL3 6.0bpw, head bits 6, codebook mul1
Calibration	250 rows × 2048 cols
VRAM loaded	~5.6GB model + KV cache
Total size on disk	~8.5GB
License	Apache 2.0 (Alibaba Cloud)

Architecture — Why This Model Is Different

Qwen3.5 is not a standard transformer. It uses a hybrid attention design that alternates between two fundamentally different attention mechanisms:

GatedDeltaNet Linear Attention (layers 0, 1, 2, 4, 5, 6, 8, 9, 10, ...)

O(n) complexity instead of O(n²) — scales linearly with sequence length
Recurrent state via conv1d kernel (kernel dim 4)
16 key heads × 128 dim, 32 value heads × 128 dim
Gated with learned input/output projections + beta decay

Full Quadratic Attention (layers 3, 7, 11, 15, 19, 23, 27, 31)

Standard grouped-query attention (16 heads, 4 KV heads, head dim 256)
RoPE with partial rotary factor 0.25, theta 10M
Provides global context every 4th layer

Layer Pattern

markdown
[linear, linear, linear, FULL, linear, linear, linear, FULL, ...] × 4 = 32 layers

24 GatedDeltaNet layers + 8 full attention layers. The linear layers handle local/sequential patterns efficiently while the full attention layers provide global context anchoring.

Multi-Token Prediction (MTP)

The model includes a 1-layer MTP head for predicting multiple next tokens simultaneously, enabling faster inference throughput than standard single-token autoregressive decoding.

Text-Only Weights

Quantization Details

Quantized with exllamav3 v1.1.0.

Format: EXL3 with trellis-based weight encoding
Bits per weight: 6.0 (all linear layers uniform)
Head bits: 6
Codebook: mul1
Why EXL3 over GGUF: Native tensor core utilization on Ampere GPUs (RTX 3090/4090). GGUF dequantizes to FP16 at runtime, wasting tensor cores. EXL3 operates natively, yielding ~1.5x faster inference at equivalent quality.

Recommended Serving

TabbyAPI (recommended)

yaml
# config.yml
network:
  host: 0.0.0.0
  port: 11435

model:
  model_dir: /path/to/models
  model_name: Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw
  max_seq_len: 4096
  gpu_split: [24]
  cache_mode: FP16

bash
CUDA_VISIBLE_DEVICES=0 python3 main.py --config config.yml

Important: Disable Thinking Mode

Add to your system prompt:

markdown
/no_think

Or set enable_thinking: false in your chat template parameters.

Note on exllamav3 Raw API

Hardware Requirements

Table with columns: Config, VRAM, Notes
Config	VRAM	Notes
Minimum	~8GB	Model only, minimal KV cache
Recommended	16GB+	Model + FP16 KV cache for 4096 context
Tested on	RTX 3090 24GB	5.6GB model, 18.4GB free for cache

Performance (tested on RTX 3090)

Response latency: 300–500ms per completion
Suitable for real-time voice pipelines with STT overhead
First request ~14ms (cache warmup, empty response)

Use Cases

Good for:

Real-time voice assistants and phone agents
Single-GPU dedicated inference servers
Applications requiring uncensored, natural conversational output
Latency-sensitive pipelines (sub-600ms budget)

Not ideal for:

Multi-user batched serving (use AWQ + vLLM instead)
Vision/multimodal tasks (no vision weights)
Applications requiring safety guardrails (abliterated model)

Credits

Base model: Qwen/Qwen3.5-9B by Alibaba Cloud (Apache 2.0)
Abliteration: huihui-ai via remove-refusals-with-transformers
EXL3 quantization: ABC&D System Inc using exllamav3 by turboderp
Serving: TabbyAPI by theroyallab