What is quantized
- Targets:
Linear layers (Qwen3_5ForConditionalGeneration)
- Quantized: full-attention projections and MLPs
- Left high-precision:
lm_head, vision tower, GDN linear_attn internals, norms, embeddings
~51 GB BF16 → ~26 GB NVFP4A16.
Serve with vLLM
vllm serve logicauro/Qwen3.8-27B-Dominatrix-NVFP4A16 \ --served-model-name Qwen3.8-27B-Dominatrix \ --language-model-only \ --quantization compressed-tensors \ --kv-cache-dtype fp8 \ --max-model-len 98304 \ --max-num-seqs 1 \ --reasoning-parser qwen3 \ --chat-template chat_template.jinja \ --trust-remote-code
Sampling (from the original card)
Temperature 1.0–1.25 with min_p 0.1 or top_p 0.95. Chat-template defaults: preserve_thinking off, reasoning_effort medium.
Credits
Original finetune: allura-org/Qwen3.8-27B-Dominatrix (Fizz / Allura), base Qwen/Qwen3.8-27B.