orcarouter

Qwen3.8-27B-Uncensored-NVFP4

README

License: apache-2.0

⚠️ Disclaimer — read before use

This model has had its safety alignment substantially removed via abliteration (orthogonalizing the refusal direction out of the residual stream). As a direct consequence:

It will comply with harmful, unethical, offensive, or illegal requests that the original Qwen3.8-27B would refuse. It has no meaningful built-in guardrails.
It is released strictly for legitimate research — interpretability, AI-safety and refusal-mechanism study, red-teaming, robustness evaluation, and controlled experiments.
You assume full responsibility and liability for how you use it and for everything it generates. Do not deploy it to end users or in production without adding your own safety, moderation, and abuse-prevention layers.
Use must comply with the Apache 2.0 License inherited from the base model, and all laws and regulations that apply to you.
The authors and uploaders accept no liability for any misuse or harm arising from this model. Its outputs do not reflect the views of the uploaders or of Qwen / Alibaba.

By downloading or using this model you acknowledge and accept the above.

Model details

Table

Base model	`Qwen/Qwen3.8-27B`
Architecture	`Qwen3_5ForConditionalGeneration` — 64 layers, hidden 5120, hybrid Gated DeltaNet (48 linear-attention + 16 full-attention, interval 4), native VL tower + MTP head
Modification	Abliteration (refusal-direction removal) then dynamic mixed-precision NVFP4 + FP8 quantization
Quantization	Mixed-precision (compressed-tensors) — NVFP4 (W4A4) on the FFN of layers 0–55, FP8 (W8A8 dynamic) on attention / GDN projections / the last 8 layers' FFN / `lm_head`, and a static FP8 KV cache

Abliteration

Refusal-direction removal following Arditi et al. (2024), Refusal in Language Models Is Mediated by a Single Direction. A single refusal direction r (k = 1) is estimated as the massive-activation–masked mean-difference of harmful − harmless last-token residuals at layer 38 (round(0.6 × 64)), on AdvBench (harmful) vs Alpaca (harmless). r is then orthogonalized out of every residual-writing matrix — W' = W − r(rᵀW) — computed in float32:

Table with columns: Component, matrices edited
Component	matrices edited
`self_attn.o_proj` (16 full-attention layers + MTP)	17
`linear_attn.out_proj` (48 linear-attention / GDN layers)	48
`mlp.down_proj` (64 layers + MTP)	65
`embed_tokens` (row space)	1
Total	131

The vision tower is untouched and the MTP head is abliterated consistently with the main model, so speculative decoding keeps working. This is the same abliterated BF16 base as the FP8, GGUF and BF16 releases — only the quantization differs.

Dynamic mixed-precision (NVFP4 + FP8) scheme

Rather than quantizing every linear uniformly, precision-sensitive layers are kept at FP8 while only the bulk feed-forward layers go to 4-bit:

Table with columns: Component, Precision, # linears
Component	Precision	# linears
MLP `gate/up/down`, layers 0–55	NVFP4 — FP4 E2M1, group 16, FP8-E4M3 block scale + FP32 global scale (W4A4)	168
`self_attn.{q,k,v,o}_proj`, GDN `in_proj_qkv/in_proj_z/out_proj`	FP8 — W8A8, per-channel weight, dynamic per-token activation	208
MLP `gate/up/down` of the last 8 layers (56–63), `lm_head`	FP8 (W8A8 dynamic)	25

Weights: round-to-nearest. NVFP4 packs FP4 (E2M1) in groups of 16 with an FP8-E4M3 block scale and an FP32 per-tensor global scale; FP8 uses per-output-channel scales.
Activations: NVFP4 layers use dynamic per-token FP4 with a calibrated global scale; FP8 layers use dynamic per-token FP8 (no static activation scale).
KV cache: static per-tensor FP8, calibrated.
Calibration: 512 samples — 75% tatsu-lab/alpaca
- 25% compliant harmful completions, enable_thinking=False, sequence length 2048 — used only for the NVFP4 activation global scales and the static FP8 KV-cache scales, keeping calibration on the activation distribution the abliterated model actually produces.
Built with llmcompressor (QuantizationModifier, two config groups + kv_cache_scheme), format: mixed-precision. Split: NVFP4 / FP8 / BF16 linears.

vLLM serves this through the compressed-tensors path: the FP4 layers use FP4 tensor cores on Blackwell, while the FP8 layers run on Hopper-class and newer.

Intended use

Research into refusal mechanisms, alignment, and interpretability.
Red-teaming and safety / robustness evaluation in controlled environments.
Uncensored generation for authorized, lawful research settings.

Out of scope

Any use that violates the base model's Apache 2.0 license or applicable law.
Deployment to the public or to end users without additional safety and moderation layers.
Generating content intended to harm, harass, defraud, or endanger people.

Evaluation

Abliteration is a weight edit shared across all releases of this model, so the refusal behavior of this checkpoint tracks the BF16 / FP8 builds. On the byte-identical-scheme FP8 build, harmful-prompt refusal collapses from 64–99% (base) to 0–6% (thinking off) and ≤ 1.7% (thinking on), while benign over-refusal drops (XSTest-safe 5.6% → 0.4%) and capability stays within ±1.3 pts of the base (MMLU 84.3 → 84.7, MMLU-Pro 77.6 → 76.8, GSM8K 90.0 → 88.7, CMMLU 81.4 → 80.8). See that model card for the full tables.

Quant-specific numbers pending. Capability-retention and perplexity for this NVFP4 + FP8 mixed-precision checkpoint have not yet been measured — the FP4 layers require Blackwell FP4 tensor cores to run natively, and this build is released for evaluation on that hardware. Numbers will be added here once benchmarked. As a mixed 4-bit/8-bit checkpoint it is expected to trade a little accuracy for size versus the FP8 build; the dynamic split (only the less-sensitive FFN layers at FP4) is designed to keep that loss small.

Multimodal (vision)

The vision tower is preserved — all 167 visual.* weight tensors are kept in BF16 and the merger / image + video preprocessor configs are intact, so this stays a full vision-language model (Qwen3_5ForConditionalGeneration), a drop-in for the base. Abliteration only edits the language-model residual writers, so image understanding is architecturally unaffected (and image-conditioned refusals are reduced along with text ones). Serve without --language-model-only to use vision.

Usage

Self-host with vLLM (OpenAI-compatible)

Requires a recent vLLM (≥ 0.27, with compressed-tensors). The FP4 layers need a Blackwell GPU (B200 / GB200 / RTX 50-series) for native FP4 tensor cores; the FP8 layers run on Hopper-class and newer.

bash
docker run -d --name qwen38-uncensored-nvfp4 --gpus all --ipc=host --shm-size=8g \
  -v /path/to/Qwen3.8-27B-Uncensored-NVFP4:/model:ro \
  -p 8000:8000 vllm/vllm-openai:v0.27.1 \
  --model /model --served-model-name Qwen3.8-27B-Uncensored \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --gpu-memory-utilization 0.9 \
  --max-model-len 262144 --max-num-seqs 96 \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

The mixed-precision quantization_config (including the FP8 kv_cache_scheme) is read from config.json — do not pass --quantization or --kv-cache-dtype. --speculative-config mtp enables the preserved MTP draft head.

Reasoning (thinking) toggle

Thinking is on by default (Qwen3.8). Toggle it per request via chat_template_kwargs; the reasoning trace is returned in the reasoning field (--reasoning-parser qwen3).

python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="Qwen3.8-27B-Uncensored",
    messages=[{"role": "user", "content": "Prove that sqrt(2) is irrational."}],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(resp.choices[0].message.reasoning)  # thinking trace
print(resp.choices[0].message.content)    # final answer

Tool calling

Standard OpenAI tools + assistant tool_calls + role: tool result messages are supported, including multi-turn (feed the tool result back for a follow-up answer). Parsed by --tool-call-parser qwen3_coder.

Via OrcaRouter (hosted API — no setup)

Served on OrcaRouter through the OpenAI-compatible gateway (262K context, tools + reasoning). Grab an API key at orcarouter.ai (sk-orca-...).

python
from openai import OpenAI

client = OpenAI(base_url="https://api.orcarouter.ai/v1", api_key="sk-orca-...")
resp = client.chat.completions.create(
    model="qwen/qwen3.8-27b",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

Hardware requirements & performance

Software

vLLM ≥ 0.27 with compressed-tensors (Qwen3.5/3.8 support) — e.g. vllm/vllm-openai:v0.27.1.

Compute

The NVFP4 (FP4) layers require a Blackwell GPU (B200 / GB200 / RTX 50-series) for native FP4 tensor cores. The FP8 layers run on Hopper (H100 / H200) and newer.

Memory

Weights: ~23 GB (mixed 4-bit / 8-bit), vs the ~56 GB BF16 checkpoint.
Minimum ~32 GB VRAM for weights + a small KV cache; the full 262K context needs substantial extra KV cache (the checkpoint already stores the KV cache in FP8).
Recommended: a single Blackwell B200 (or larger) for the full FP4 path.

Throughput / concurrency

Continuous batching; concurrency bounded by --max-num-seqs and the KV cache that fits after weights are loaded. The MTP draft head gives a large decode speedup on real workloads.

Bias, risks, and limitations

Safety guardrails removed — the model will produce harmful, biased, or offensive content on request. See the disclaimer above.
It inherits any biases and limitations of the base Qwen3.8-27B.
Mixed NVFP4 + FP8 is not lossless versus BF16; quant-specific capability impact for this build is not yet measured (see Evaluation).
The FP4 layers require Blackwell hardware to run natively; on pre-Blackwell GPUs the FP4 path is unavailable.

License

Apache 2.0, inherited from the base model Qwen/Qwen3.8-27B. Abliteration and quantization do not change the underlying license obligations.

Changelog

2026-08-21 — Fixed vLLM loading error (`lm_head.weight_scale`)

Earlier revisions failed to load in vLLM with:

markdown
ValueError: There is no module or parameter named 'lm_head.weight_scale' in Qwen3_5ForCausalLM.
The available parameters belonging to lm_head (ParallelLMHead) are: {'lm_head.weight'}

Cause: the output head (lm_head) had been quantized to FP8, so the checkpoint shipped a lm_head.weight_scale tensor. vLLM's Qwen3_5ForCausalLM always builds lm_head as an unquantized ParallelLMHead (only a weight parameter), leaving the extra scale with no destination and aborting the load.

Fix: lm_head is now kept unquantized — restored to the original BF16 weight, lm_head.weight_scale removed, and lm_head moved to the quantization ignore list (matching the INT8 build). Only config.json, model.safetensors.index.json, and model-00005-of-00005.safetensors changed; all other tensors (FP4 body, FP8 layers, FP8 KV scales, MTP head) are byte-for-byte identical.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

orcarouter

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

⚠️ Disclaimer — read before use

This model has had its safety alignment substantially removed via abliteration (orthogonalizing the refusal direction out of the residual stream). As a direct consequence:

It will comply with harmful, unethical, offensive, or illegal requests that the original Qwen3.8-27B would refuse. It has no meaningful built-in guardrails.
It is released strictly for legitimate research — interpretability, AI-safety and refusal-mechanism study, red-teaming, robustness evaluation, and controlled experiments.
You assume full responsibility and liability for how you use it and for everything it generates. Do not deploy it to end users or in production without adding your own safety, moderation, and abuse-prevention layers.
Use must comply with the Apache 2.0 License inherited from the base model, and all laws and regulations that apply to you.
The authors and uploaders accept no liability for any misuse or harm arising from this model. Its outputs do not reflect the views of the uploaders or of Qwen / Alibaba.

By downloading or using this model you acknowledge and accept the above.

Model details

Table

Base model	`Qwen/Qwen3.8-27B`
Architecture	`Qwen3_5ForConditionalGeneration` — 64 layers, hidden 5120, hybrid Gated DeltaNet (48 linear-attention + 16 full-attention, interval 4), native VL tower + MTP head
Modification	Abliteration (refusal-direction removal) then dynamic mixed-precision NVFP4 + FP8 quantization
Quantization	Mixed-precision (compressed-tensors) — NVFP4 (W4A4) on the FFN of layers 0–55, FP8 (W8A8 dynamic) on attention / GDN projections / the last 8 layers' FFN / `lm_head`, and a static FP8 KV cache

Abliteration

Table with columns: Component, matrices edited
Component	matrices edited
`self_attn.o_proj` (16 full-attention layers + MTP)	17
`linear_attn.out_proj` (48 linear-attention / GDN layers)	48
`mlp.down_proj` (64 layers + MTP)	65
`embed_tokens` (row space)	1
Total	131

Dynamic mixed-precision (NVFP4 + FP8) scheme

Rather than quantizing every linear uniformly, precision-sensitive layers are kept at FP8 while only the bulk feed-forward layers go to 4-bit:

Table with columns: Component, Precision, # linears
Component	Precision	# linears
MLP `gate/up/down`, layers 0–55	NVFP4 — FP4 E2M1, group 16, FP8-E4M3 block scale + FP32 global scale (W4A4)	168
`self_attn.{q,k,v,o}_proj`, GDN `in_proj_qkv/in_proj_z/out_proj`	FP8 — W8A8, per-channel weight, dynamic per-token activation	208
MLP `gate/up/down` of the last 8 layers (56–63), `lm_head`	FP8 (W8A8 dynamic)	25

Weights: round-to-nearest. NVFP4 packs FP4 (E2M1) in groups of 16 with an FP8-E4M3 block scale and an FP32 per-tensor global scale; FP8 uses per-output-channel scales.
Activations: NVFP4 layers use dynamic per-token FP4 with a calibrated global scale; FP8 layers use dynamic per-token FP8 (no static activation scale).
KV cache: static per-tensor FP8, calibrated.
Calibration: 512 samples — 75% tatsu-lab/alpaca
- 25% compliant harmful completions, enable_thinking=False, sequence length 2048 — used only for the NVFP4 activation global scales and the static FP8 KV-cache scales, keeping calibration on the activation distribution the abliterated model actually produces.
Built with llmcompressor (QuantizationModifier, two config groups + kv_cache_scheme), format: mixed-precision. Split: NVFP4 / FP8 / BF16 linears.

vLLM serves this through the compressed-tensors path: the FP4 layers use FP4 tensor cores on Blackwell, while the FP8 layers run on Hopper-class and newer.

Intended use

Research into refusal mechanisms, alignment, and interpretability.
Red-teaming and safety / robustness evaluation in controlled environments.
Uncensored generation for authorized, lawful research settings.

Out of scope

Any use that violates the base model's Apache 2.0 license or applicable law.
Deployment to the public or to end users without additional safety and moderation layers.
Generating content intended to harm, harass, defraud, or endanger people.

Evaluation

Quant-specific numbers pending. Capability-retention and perplexity for this NVFP4 + FP8 mixed-precision checkpoint have not yet been measured — the FP4 layers require Blackwell FP4 tensor cores to run natively, and this build is released for evaluation on that hardware. Numbers will be added here once benchmarked. As a mixed 4-bit/8-bit checkpoint it is expected to trade a little accuracy for size versus the FP8 build; the dynamic split (only the less-sensitive FFN layers at FP4) is designed to keep that loss small.

Multimodal (vision)

Usage

Self-host with vLLM (OpenAI-compatible)

bash
docker run -d --name qwen38-uncensored-nvfp4 --gpus all --ipc=host --shm-size=8g \
  -v /path/to/Qwen3.8-27B-Uncensored-NVFP4:/model:ro \
  -p 8000:8000 vllm/vllm-openai:v0.27.1 \
  --model /model --served-model-name Qwen3.8-27B-Uncensored \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --gpu-memory-utilization 0.9 \
  --max-model-len 262144 --max-num-seqs 96 \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

The mixed-precision quantization_config (including the FP8 kv_cache_scheme) is read from config.json — do not pass --quantization or --kv-cache-dtype. --speculative-config mtp enables the preserved MTP draft head.

Reasoning (thinking) toggle

Thinking is on by default (Qwen3.8). Toggle it per request via chat_template_kwargs; the reasoning trace is returned in the reasoning field (--reasoning-parser qwen3).

python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="Qwen3.8-27B-Uncensored",
    messages=[{"role": "user", "content": "Prove that sqrt(2) is irrational."}],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(resp.choices[0].message.reasoning)  # thinking trace
print(resp.choices[0].message.content)    # final answer

Tool calling

Via OrcaRouter (hosted API — no setup)

Served on OrcaRouter through the OpenAI-compatible gateway (262K context, tools + reasoning). Grab an API key at orcarouter.ai (sk-orca-...).

python
from openai import OpenAI

client = OpenAI(base_url="https://api.orcarouter.ai/v1", api_key="sk-orca-...")
resp = client.chat.completions.create(
    model="qwen/qwen3.8-27b",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

Hardware requirements & performance

Software

vLLM ≥ 0.27 with compressed-tensors (Qwen3.5/3.8 support) — e.g. vllm/vllm-openai:v0.27.1.

Compute

The NVFP4 (FP4) layers require a Blackwell GPU (B200 / GB200 / RTX 50-series) for native FP4 tensor cores. The FP8 layers run on Hopper (H100 / H200) and newer.

Memory

Weights: ~23 GB (mixed 4-bit / 8-bit), vs the ~56 GB BF16 checkpoint.
Minimum ~32 GB VRAM for weights + a small KV cache; the full 262K context needs substantial extra KV cache (the checkpoint already stores the KV cache in FP8).
Recommended: a single Blackwell B200 (or larger) for the full FP4 path.

Throughput / concurrency

Continuous batching; concurrency bounded by --max-num-seqs and the KV cache that fits after weights are loaded. The MTP draft head gives a large decode speedup on real workloads.

Bias, risks, and limitations

Safety guardrails removed — the model will produce harmful, biased, or offensive content on request. See the disclaimer above.
It inherits any biases and limitations of the base Qwen3.8-27B.
Mixed NVFP4 + FP8 is not lossless versus BF16; quant-specific capability impact for this build is not yet measured (see Evaluation).
The FP4 layers require Blackwell hardware to run natively; on pre-Blackwell GPUs the FP4 path is unavailable.

License

Apache 2.0, inherited from the base model Qwen/Qwen3.8-27B. Abliteration and quantization do not change the underlying license obligations.

Changelog

2026-08-21 — Fixed vLLM loading error (`lm_head.weight_scale`)

Earlier revisions failed to load in vLLM with:

markdown
ValueError: There is no module or parameter named 'lm_head.weight_scale' in Qwen3_5ForCausalLM.
The available parameters belonging to lm_head (ParallelLMHead) are: {'lm_head.weight'}

Qwen3.8-27B-Uncensored-NVFP4

README

⚠️ Disclaimer — read before use

Model details

Abliteration

Dynamic mixed-precision (NVFP4 + FP8) scheme

Intended use

Out of scope

Evaluation

Multimodal (vision)

Usage

Self-host with vLLM (OpenAI-compatible)

Reasoning (thinking) toggle

Tool calling

Via OrcaRouter (hosted API — no setup)

Hardware requirements & performance

Bias, risks, and limitations

License

Changelog

2026-08-21 — Fixed vLLM loading error (lm_head.weight_scale)

Explore FriendliAI today

README

⚠️ Disclaimer — read before use

Model details

Abliteration

Dynamic mixed-precision (NVFP4 + FP8) scheme

Intended use

Out of scope

Evaluation

Multimodal (vision)

Usage

Self-host with vLLM (OpenAI-compatible)

Reasoning (thinking) toggle

Tool calling

Via OrcaRouter (hosted API — no setup)

Hardware requirements & performance

Bias, risks, and limitations

License

Changelog

2026-08-21 — Fixed vLLM loading error (lm_head.weight_scale)

2026-08-21 — Fixed vLLM loading error (`lm_head.weight_scale`)

2026-08-21 — Fixed vLLM loading error (`lm_head.weight_scale`)