Quantization method
Table with columns: Layer, Format, Group| Layer | Format | Group |
|---|
| MLP, attention (q/k/v/o, in_proj_qkv/z, out_proj) | W4A16_ASYM (int4) | 128 |
| in_proj_a/b | BF16 | — |
| lm_head, embed_tokens | BF16 | — |
| visual, mtp, norms | BF16 | — |
- Tooling: llmcompressor
oneshot with CPU offloading + AWQModifier (hybrid-attention mappings, linear_attn rules dropped).
- Run command:
CUDA_VISIBLE_DEVICES=1,2 python3 quantize-awq-hybrid.py \ --model_path ./Qwen3.8-27B-Ultimate-Heretic-Uncensored \ --quant_path ./Qwen3.8-27B-Ultimate-Heretic-Uncensored-W4A16-AWQ
Recommended parameters
From the base model: temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 0, repetition_penalty 1.0.
Usage
Tested with 1Cat-vLLM (compressed-tensors, SM70 kernels):
from vllm import LLM, SamplingParams llm = LLM( "TheUnderscore/Qwen3.8-27B-Ultimate-Heretic-Uncensored-W4A16-AWQ", tensor_parallel_size=2, max_model_len=8192,)output = llm.generate("Hello, who are you?", SamplingParams(temperature=1.0, top_p=0.95, max_tokens=256))print(output[0].text)
License
Apache-2.0, the same as the finetune.
Files
quantize-awq-hybrid.py — the script used to produce this quantization (CPU-offloaded, DDP/torchrun, produces properly numbered -of-N shards).