com-kotobalabs

Qwen3.8-27B-Uncensored-whitehat-NVFP4

README

License: apache-2.0

What was changed

Qwen3.8-27B is a dense multimodal model (Qwen3_5ForConditionalGeneration, qwen3_5): 64 decoder layers with hybrid attention (linear + periodic full), dense MLP, hidden_size 5120, plus a vision tower.

Method: refusal-direction ablation (Arditi et al., 2024). Estimate the residual-stream direction that most separates refusal-eliciting from benign prompts, then orthogonalize every residual-stream writer against it — self_attn.o_proj (full-attn layers), linear_attn.out_proj (linear-attn layers), mlp.down_proj, and embed_tokens: W' = W − r·(rᵀW).
Untouched: the vision tower (visual.*), and lm_head. Vision behavior is unchanged.
Quantization: NVFP4 (Blackwell-native FP4) via llm-compressor → compressed-tensors. Vision, embeddings, lm_head, and the linear-attention conv1d/in_proj_a/in_proj_b tensors stay higher-precision.

Serving — one RTX PRO 6000

NVFP4 weights are ~18 GB, so they fit a single 96 GB RTX PRO 6000 with ~76 GB free for the KV cache (the base supports 262k context). (bf16, ~56 GB, also fits one card — NVFP4 buys Blackwell FP4 throughput + long-context headroom.)

Weights are compressed-tensors NVFP4 (format: nvfp4-pack-quantized, W4A4), auto-detected from config.json — no special flag needed on recent vLLM/SGLang.

bash

# vLLM (single RTX PRO 6000)vllm serve com-kotobalabs/Qwen3.8-27B-Uncensored-whitehat-NVFP4 --trust-remote-code

bash

# SGLangpython -m sglang.launch_server \  --model-path com-kotobalabs/Qwen3.8-27B-Uncensored-whitehat-NVFP4 \  --quantization compressed-tensors --tp 1 --trust-remote-code

No GGUF (yet)

llama.cpp can't convert this architecture: qwen3_5 hybrid linear-attention tensors (ssm_conv1d kernel dim + in_proj_a/in_proj_b expansion) are unhandled — ggml-org/llama.cpp#27019. NVFP4 for SGLang/vLLM is the supported path until that lands.

Validation (measured)

Ablation coverage: 129/129 residual-stream writers orthogonalized (64 attention output projections + 64 mlp.down_proj + embed_tokens).
Refusal rate (do_sample=False, on refusal-eliciting prompts base Qwen refuses ~universally): 0% in bf16 and 0% after NVFP4 — i.e. the abliteration survives quantization. This measures refusal removal, not general capability; run your own task-level evals before relying on it.
NVFP4 calibration: 256 general-instruction samples, W4A4.

Throughput / concurrency — RTX PRO 6000 (measured)

Measured 2026-09-18 on a single NVIDIA RTX PRO 6000 Blackwell (96 GB) via a Hugging Face Inference Endpoint, vLLM serving the NVFP4 weights (maxNumSeqs=64, max-model-len=32768, greedy, 256-token completions, streaming). Reproduce with bench/benchmark.py; raw data in bench/results.json.

Table with columns: concurrency, output tok/s, req/s, E2E p50 (256-tok request), errors
concurrency	output tok/s	req/s	E2E p50 (256-tok request)	errors
1	47	0.18	5.4 s	0
8	329	1.3	6.0 s	0
16	641	2.5	6.3 s	0
32

Peak throughput ≈ 1.9k output tok/s (concurrency 128). The batch fills at maxNumSeqs=64 (~1.6k tok/s, E2E 8.3 s); beyond that, requests queue and per-request latency roughly doubles (17 s at 128).
Max concurrency tested: 128, with zero errors — this is a sweep bound, not a failure ceiling; more is servable by raising maxNumSeqs at the cost of latency.
Interactive range: ≤ 64 concurrent keeps a full 256-token completion under ~8.3 s.
Note: fine-grained TTFT/ITL were not reliably captured this run (the --reasoning-parser qwen3 stream splits reasoning vs. content); wall-clock E2E latency is reported instead. Throughput/req-counts are from usage.

Reproduce

modal_pipeline.py (download → abliterate → NVFP4 → publish; smoke for the refusal check) and bench/ (endpoint deploy + concurrency sweep). Calibration prompt sets in data/harmful.txt and data/harmless.txt.

License

Apache-2.0, inherited from the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

com-kotobalabs

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What was changed

Method: refusal-direction ablation (Arditi et al., 2024). Estimate the residual-stream direction that most separates refusal-eliciting from benign prompts, then orthogonalize every residual-stream writer against it — self_attn.o_proj (full-attn layers), linear_attn.out_proj (linear-attn layers), mlp.down_proj, and embed_tokens: W' = W − r·(rᵀW).
Untouched: the vision tower (visual.*), and lm_head. Vision behavior is unchanged.
Quantization: NVFP4 (Blackwell-native FP4) via llm-compressor → compressed-tensors. Vision, embeddings, lm_head, and the linear-attention conv1d/in_proj_a/in_proj_b tensors stay higher-precision.

Serving — one RTX PRO 6000

Weights are compressed-tensors NVFP4 (format: nvfp4-pack-quantized, W4A4), auto-detected from config.json — no special flag needed on recent vLLM/SGLang.

bash

# vLLM (single RTX PRO 6000)vllm serve com-kotobalabs/Qwen3.8-27B-Uncensored-whitehat-NVFP4 --trust-remote-code

# vLLM (single RTX PRO 6000)vllm serve com-kotobalabs/Qwen3.8-27B-Uncensored-whitehat-NVFP4 --trust-remote-code

bash

# SGLangpython -m sglang.launch_server \  --model-path com-kotobalabs/Qwen3.8-27B-Uncensored-whitehat-NVFP4 \  --quantization compressed-tensors --tp 1 --trust-remote-code

# SGLangpython -m sglang.launch_server \  --model-path com-kotobalabs/Qwen3.8-27B-Uncensored-whitehat-NVFP4 \  --quantization compressed-tensors --tp 1 --trust-remote-code

No GGUF (yet)

Validation (measured)

Ablation coverage: 129/129 residual-stream writers orthogonalized (64 attention output projections + 64 mlp.down_proj + embed_tokens).
Refusal rate (do_sample=False, on refusal-eliciting prompts base Qwen refuses ~universally): 0% in bf16 and 0% after NVFP4 — i.e. the abliteration survives quantization. This measures refusal removal, not general capability; run your own task-level evals before relying on it.
NVFP4 calibration: 256 general-instruction samples, W4A4.

Throughput / concurrency — RTX PRO 6000 (measured)

Table with columns: concurrency, output tok/s, req/s, E2E p50 (256-tok request), errors
concurrency	output tok/s	req/s	E2E p50 (256-tok request)	errors
1	47	0.18	5.4 s	0
8	329	1.3	6.0 s	0
16	641	2.5	6.3 s	0
32

Peak throughput ≈ 1.9k output tok/s (concurrency 128). The batch fills at maxNumSeqs=64 (~1.6k tok/s, E2E 8.3 s); beyond that, requests queue and per-request latency roughly doubles (17 s at 128).
Max concurrency tested: 128, with zero errors — this is a sweep bound, not a failure ceiling; more is servable by raising maxNumSeqs at the cost of latency.
Interactive range: ≤ 64 concurrent keeps a full 256-token completion under ~8.3 s.
Note: fine-grained TTFT/ITL were not reliably captured this run (the --reasoning-parser qwen3 stream splits reasoning vs. content); wall-clock E2E latency is reported instead. Throughput/req-counts are from usage.

Reproduce

License

Apache-2.0, inherited from the base model.