soppyleon

Qwen3.8-27B-Uncensored-UD-V3-NVFP4

README

License: apache-2.0

Abliteration lineage

Inherited unchanged from the parent — this repo only re-quantizes it: Heretic directional ablation on attn.o_proj + mlp.down_proj (200 trials; published point: 12/100 refusals on mlabonne/harmful_behaviors test vs 98/100 stock, first-token KL 0.1191, lm-eval 0-shot mean −0.5 vs stock — see the parent card). The same parent underlies the popular outsourc-e/Qwen3.8-27B-Unleashed-GGUF. In this build's own eval (20 harmful_behaviors test prompts, greedy, thinking off): 0/20 refusals — identical to the parent's UD-Q5_K_M GGUF under the same harness.

How it was made

llm-compressor 0.13.0 / compressed-tensors 0.18.0, GPTQModifier with the two config_groups above, kv_cache_scheme FP8 tensor-static, and ignore = [visual, linear_attn.{norm,in_proj_a,in_proj_b}, mtp.*, embed_tokens]. Calibration: 512 samples × 2048 tokens — ⅓ UltraChat-200k, ⅓ nvidia/OpenCodeInstruct, ⅓ glaive-function-calling-v2, rendered through the model's chat template. ~100 min on one RTX 5090 (GPU ≈11 GB; the bf16 parent needs ≥64 GB system RAM).

Differences from the official Unsloth checkpoint: their private calibration mix and GPTQ hyperparameters are not public, so this build inherits Unsloth's layer→precision decisions but not their calibration data. The quantization script (quantize.py) ships in the repo files.

Two gotchas the script handles (relevant if you reproduce this on another finetune):

transformers never instantiates mtp.*, so llm-compressor silently drops re:^mtp.* from ignore — it must be re-added to config.json afterwards or vLLM quantizes the MTP drafter and generates garbage.
the BF16 MTP tensors must be re-grafted as model_mtp.safetensors and added to model.safetensors.index.json by hand.

Serving (vLLM ≥ 0.28.0, SM120)

bash
vllm serve <this-repo> \
  --max-model-len 131072 --kv-cache-dtype fp8 --attention-backend FLASHINFER \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --gpu-memory-utilization 0.96 --max-num-seqs 2 --max-cudagraph-capture-size 64 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml

On a 32 GB RTX 5090 this fits 128K context fully GPU-resident (139K-token FP8 KV pool, ~30.3 GiB total). Startup log should show Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM — if you see a Marlin warning instead, you are not getting native FP4 execution. Do not pass --calculate-kv-scales (broken on hybrid GDN models; the checkpoint ships calibrated scales).

Measured (RTX 5090, single user, greedy)

Table with columns: this repo (vLLM, MTP-3, FP8 KV, 128K), parent UD-Q5_K_M (llama.cpp, MTP-2, q8_0 KV)
	this repo (vLLM, MTP-3, FP8 KV, 128K)	parent UD-Q5_K_M (llama.cpp, MTP-2, q8_0 KV)
generation tok/s (≈0/6K/25K-token prompts)	108 / 110 / 108	110 / 87 / 82
prefill tok/s	≈8,400	≈2,000
TTFT @6K / @25K prompt	0.75 s / 3.6 s	3.3 s / 12 s
VRAM	30.3 GiB	24.7 GiB (64K)
MTP acceptance	0.54/draft-tok (N=3), 0.92 on code	0.89/draft-tok (N=2)

¹ one formatting miss (dropped a typing import), one real miss, one harness quirk shared by both stacks. Vision verified (image Q&A), tool calling verified (qwen3_xml parser), reasoning split into reasoning_content works. SGLang is currently not recommended for this checkpoint: it drops the FP8 lm_head scale (sgl-project/sglang#34895), producing repetition loops.

Limitations

Refusal reduction, benchmarks, and all parent-model caveats are inherited from the parent card — read its "Caveats that matter". Not intended for deployment to third parties without your own safety layer.
Needs Blackwell (SM120+) for native W4A4; on older GPUs vLLM falls back to weight-only Marlin (slower).
Quality evals here are small (n=40 per section), designed to detect quantization damage, not to rank models.

Apache 2.0, inherited from Qwen3.8-27B via the parent. Credit for the abliteration goes to JonathanColetti; credit for the Dynamic V3 NVFP4 recipe design goes to Unsloth.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

soppyleon

Model Tree

Base

JonathanColetti/Qwen3.8-27B-Uncensored

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality