What is quantized
- Targets:
Linear layers (Qwen3_5ForConditionalGeneration)
- Quantized: full-attention projections and MLPs
- Left high-precision:
lm_head, vision tower, GDN linear_attn internals, norms, embeddings
~51 GB BF16 → ~26 GB NVFP4A16.
Serve with vLLM
vllm serve logicauro/Qwen3.8-27B-Dominatrix-NVFP4A16 \
--served-model-name Qwen3.8-27B-Dominatrix \
--language-model-only \
--quantization compressed-tensors \
--kv-cache-dtype fp8 \
--max-model-len 98304 \
--max-num-seqs 1 \
--reasoning-parser qwen3 \
--chat-template chat_template.jinja \
--trust-remote-code
Sampling (from the original card)
Temperature 1.0–1.25 with min_p 0.1 or top_p 0.95. Chat-template defaults: preserve_thinking off, reasoning_effort medium.
Credits
Original finetune: allura-org/Qwen3.8-27B-Dominatrix (Fizz / Allura), base Qwen/Qwen3.8-27B.