trithemius

Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-PrismaAURA-4.75bit

README

License: apache-2.0

About PrismaAURA quantization

PrismaAURA is a mixed-precision post-training quantization method. Rather than applying a uniform bit-width to every linear layer, it profiles each Linear module's individual contribution to end-to-end output drift and assigns bits accordingly:

Per-layer sensitivity analysis — each linear layer is quantized in isolation and the resulting shift in the model's output distribution (KL divergence vs. the BF16 reference, measured on real generations rather than a proxy loss) is recorded.
Bit budget allocation — layers that are highly sensitive to quantization (typically attention projections and early/late MoE layers) are kept at higher precision (FP8 or BF16), while more tolerant layers (the bulk of the MoE expert weights) are pushed down to NVFP4. The result is an average effective rate of 4.75 bits/parameter across quantizable weights, not a flat 4-bit or 5-bit scheme.
Standard artifact format — the output is packaged as a normal compressed-tensors checkpoint (NVFP4 + FP8 + BF16 mixed per-layer), which vLLM already understands natively. There is no custom runtime, no patched attention kernel, and no bespoke inference server required — any vLLM build with CUTLASS NVFP4 W4A4 support on Blackwell-class GPUs (or an equivalent compatible backend) can serve it directly.

Because the bit allocation is chosen against measured KL rather than a fixed recipe, PrismaAURA generally holds tighter to the source model's output distribution at a given size than a uniform-precision quant of the same average bitwidth.

Vision-tower Linears are excluded from the bit search and assigned uniformly to BF16 — PrismaAURA's per-layer sensitivity sweep in this build targets the text/MoE stack only.

Allocator search (this build)

Target rate: 4.75 bits/parameter. The allocator swept a range of target rates against measured Δloss (end-to-end KL vs. BF16) before settling on the closest achievable allocation:

Table with columns: target, achieved, Δloss (body), NVFP4, FP8_E4M3, BF16
target	achieved	Δloss (body)	NVFP4	FP8_E4M3	BF16
4.500	4.500	1.1607e-01	229	1	0
4.600	4.600	1.0588e-01	91	132	7
4.700	4.700	1.0384e-01

Final allocation at the shipped 4.75 target (achieved_bits=4.749, Δloss=1.027e-01):

FP8_E4M3: 232 layers
BF16: 188 layers
NVFP4: 80 layers
Vision: 110 visual Linears, uniform BF16 (--visual-format=BF16)

Inherited behavior

The base checkpoint this quant was built from (HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive) is an "Aggressive" uncensoring variant of Qwen3.6-35B-A3B: safety refusals have been suppressed at the weight level, and the model is described upstream as generating full content without refusal across its evaluation set. This quantization step does not add, remove, or otherwise modify that behavior — PrismaAURA preserves the calibration distribution of whatever checkpoint it's pointed at, refusal-suppressed or not.

Practical implications:

Standard safety guardrails and refusal behavior present in the stock Qwen3.6 release are not present in this model.
Deployers are responsible for adding their own input/output moderation, rate limiting, and use-case-appropriate guardrails at the application layer if this model is exposed to end users.
This model is intended for research, red-teaming, local/offline use, and other contexts where the deployer has already accepted responsibility for the base model's uncensored behavior. It is not appropriate for unmoderated public-facing deployment.

MTP note

This checkpoint does not include a working Multi-Token-Prediction (MTP) head. The base HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive checkpoint's config declares an MTP layer, but its safetensors index ships no mtp.layers.* weights for it — a common side effect of fine-tunes that strip MTP during training/export. That gap was carried through the quantization step unchanged.

If you serve this model with --speculative-config '{"method":"mtp",...}' anyway, vLLM will still detect an MTP-shaped module (since the config says one exists) and will share the target model's embedding/lm_head weights with it, but the draft layer's own attention/MLP weights will be randomly initialized rather than loaded — since there's nothing in the checkpoint to load. That produces a non-functional draft head: expect 0% draft acceptance at every position, not degraded performance. Don't enable --speculative-config for this model; it will only add drafting overhead with no upside.

A future release may graft a working MTP head from the base Qwen/Qwen3.6-35B-A3B checkpoint (as done in rdtand/Ornith-1.0-35B-PrismaAURA-4.75bit-vllm-MTP for a different base model) — this repo will be updated if/when that happens.

Serving

This is the command used to serve this checkpoint:

bash
vllm serve trithemius/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-PrismaAURA-4.75bit \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name qwen3.6-35b \
  --max-model-len 262144 \
  --max-num-batched-tokens 32768 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.8 \
  --dtype auto \
  --load-format fastsafetensors \
  --attention-backend flashinfer \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --trust-remote-code \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --optimization-level 3 \
  --performance-mode throughput \
  --default-chat-template-kwargs '{"preserve_thinking":true}'

Do not add --speculative-config for this model — see MTP note above for why.

Notes:

Runs at full native context (--max-model-len 262144); drop this to something smaller if you're VRAM-constrained.
--kv-cache-dtype fp8 trades a small amount of long-context quality for a much smaller KV cache — drop it (or set auto) if you want BF16 KV cache instead.
--reasoning-parser qwen3 and --default-chat-template-kwargs '{"preserve_thinking":true}' keep the model's thinking traces intact in the API response; drop preserve_thinking if you only want final answers.
--tool-call-parser qwen3_coder + --enable-auto-tool-choice enable OpenAI-style tool/function calling.
--load-format fastsafetensors speeds up cold-start weight loading; falls back safely if unavailable.
Requires a Blackwell-class (or NVFP4-W4A4-capable) GPU for the native CUTLASS kernels; on older hardware vLLM will fall back to a dequantize-and-run path, which works but loses the throughput benefit of native NVFP4.

Also works with

Since the checkpoint is a standard compressed-tensors mixed-precision artifact, it should also load in any other engine with NVFP4/FP8 compressed-tensors support (e.g. SGLang with an equivalent backend flag set). vLLM is the tested/recommended path.

Specs (inherited from base model)

35B total parameters, ~3B active per forward pass (MoE)
256 experts, 8 routed per token
Hybrid architecture: linear attention + full softmax attention (3:1 ratio)
40 layers
262K native context
Natively multimodal (text, image, video) — vision tower kept at BF16 in this quant, see allocator search
Based on Qwen/Qwen3.6-35B-A3B via HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive

Recommended sampling settings

From the base model card, inherited from the upstream Qwen authors:

Thinking mode (default):

General: temperature=1.0, top_p=0.95, top_k=20, min_p=0, presence_penalty=1.5
Coding / precise tasks: temperature=0.6, top_p=0.95, top_k=20, min_p=0, presence_penalty=0

Non-thinking mode:

General: temperature=0.7, top_p=0.8, top_k=20, min_p=0, presence_penalty=1.5
Reasoning tasks: temperature=1.0, top_p=1.0, top_k=40, min_p=0, presence_penalty=2.0

Keep at least 128K context if you want to preserve thinking-mode capability.

License

Inherits the base model's license — see HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive (apache-2.0) and, upstream of that, Qwen/Qwen3.6-35B-A3B.

Contact

Open an issue in the Community tab of this repo.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

trithemius

Model Tree

Base

HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality