THe-Plague

Qwen3.6-35B-A3B-abliterated-NVFP4-MTP

README

License: apache-2.0

What Makes This Different

No other model on HuggingFace combines all three:

Table with columns: Feature, This Model, sakamakismile NVFP4, RedHatAI NVFP4
Feature	This Model	sakamakismile NVFP4	RedHatAI NVFP4
NVFP4 weights	✅	✅	✅
MTP layers	✅ BF16	❌	✅
Abliterated	✅	✅	❌
Speculative decode	✅ 95 tok/s	❌ ~70 tok/s	✅ but censored

Without MTP layers, you're limited to ~70 tok/s on GB10. MTP enables speculative decoding which gives a +35% speedup.

Model Details

Architecture: Qwen3 MoE (Mixture of Experts), 35B total / 3B active parameters
Main weights: NVFP4 quantized (21.9 GB) — optimized for NVIDIA Blackwell CUTLASS kernels
MTP layers: BF16 (1.6 GB, 19 tensors in model-mtp.safetensors)
Total size: ~23.5 GB
Context length: 131,072 tokens
Abliteration: Refusal direction removed via representation engineering (from huihui-ai)

How It Was Made

Started with sakamakismile/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4 (NVFP4 main weights, no MTP)
Extracted BF16 MTP layers from huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated (full BF16 model)
Injected MTP tensors into a separate model-mtp.safetensors file
Updated model.safetensors.index.json with MTP weight mappings
Verified with vLLM speculative decoding (MTP=1 through MTP=4)

Benchmark Results

Hardware: NVIDIA DGX Spark (GB10 Blackwell, sm_121, ~128GB unified memory) Engine: vLLM v0.23.1 with FlashInfer + CUTLASS NVFP4 MoE kernels Benchmark: 256 generated tokens at various context lengths

Table with columns: Config, 1K ctx, 4K ctx, 16K ctx, 32K ctx, 64K ctx
Config	1K ctx	4K ctx	16K ctx	32K ctx	64K ctx
MTP=3, seqs=1 (best)	92.2	95.2	90.7	89.7	84.4
MTP=2, seqs=4	87.2	75.5	82.8	81.6	74.4
MTP=4, seqs=1	75.6

MTP=3 is the sweet spot. MTP=4 regresses due to speculation overhead exceeding acceptance rate.

Recommended vLLM Launch Config

bash
docker run -d --gpus all --network host --ipc host --shm-size=16g \
  --name vllm-qwen36 \
  -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
  -e TORCH_MATMUL_PRECISION=high \
  -v /path/to/models:/models \
  ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest \
  vllm serve /models/Qwen3.6-35B-A3B-abliterated-NVFP4-MTP \
    --host 0.0.0.0 --port 8000 --max-model-len 131072 \
    --max-num-batched-tokens 32768 --max-num-seqs 1 \
    --trust-remote-code --gpu-memory-utilization 0.7 \
    --reasoning-parser qwen3 --kv-cache-dtype fp8 \
    --load-format instanttensor --attention-backend flashinfer \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
    --enable-prefix-caching -tp 1

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

THe-Plague

Model Tree

Base

huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What Makes This Different

No other model on HuggingFace combines all three:

Table with columns: Feature, This Model, sakamakismile NVFP4, RedHatAI NVFP4
Feature	This Model	sakamakismile NVFP4	RedHatAI NVFP4
NVFP4 weights	✅	✅	✅
MTP layers	✅ BF16	❌	✅
Abliterated	✅	✅	❌
Speculative decode	✅ 95 tok/s	❌ ~70 tok/s	✅ but censored

Without MTP layers, you're limited to ~70 tok/s on GB10. MTP enables speculative decoding which gives a +35% speedup.

Model Details

Architecture: Qwen3 MoE (Mixture of Experts), 35B total / 3B active parameters
Main weights: NVFP4 quantized (21.9 GB) — optimized for NVIDIA Blackwell CUTLASS kernels
MTP layers: BF16 (1.6 GB, 19 tensors in model-mtp.safetensors)
Total size: ~23.5 GB
Context length: 131,072 tokens
Abliteration: Refusal direction removed via representation engineering (from huihui-ai)

How It Was Made

Started with sakamakismile/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4 (NVFP4 main weights, no MTP)
Extracted BF16 MTP layers from huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated (full BF16 model)
Injected MTP tensors into a separate model-mtp.safetensors file
Updated model.safetensors.index.json with MTP weight mappings
Verified with vLLM speculative decoding (MTP=1 through MTP=4)

Benchmark Results

Table with columns: Config, 1K ctx, 4K ctx, 16K ctx, 32K ctx, 64K ctx
Config	1K ctx	4K ctx	16K ctx	32K ctx	64K ctx
MTP=3, seqs=1 (best)	92.2	95.2	90.7	89.7	84.4
MTP=2, seqs=4	87.2	75.5	82.8	81.6	74.4
MTP=4, seqs=1	75.6

MTP=3 is the sweet spot. MTP=4 regresses due to speculation overhead exceeding acceptance rate.

Recommended vLLM Launch Config

bash
docker run -d --gpus all --network host --ipc host --shm-size=16g \
  --name vllm-qwen36 \
  -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
  -e TORCH_MATMUL_PRECISION=high \
  -v /path/to/models:/models \
  ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest \
  vllm serve /models/Qwen3.6-35B-A3B-abliterated-NVFP4-MTP \
    --host 0.0.0.0 --port 8000 --max-model-len 131072 \
    --max-num-batched-tokens 32768 --max-num-seqs 1 \
    --trust-remote-code --gpu-memory-utilization 0.7 \
    --reasoning-parser qwen3 --kv-cache-dtype fp8 \
    --load-format instanttensor --attention-backend flashinfer \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
    --enable-prefix-caching -tp 1