krisclarkdev

Qwen3.8-27B-TURBO-DF735-882-NM-DAU-GPTQ-Int4-g64

README

License: apache-2.0

Why this repository exists

The upstream finetune was only distributed as BF16 (~55.6 GB) or GGUF. The few community Int4 quants of it do not load under vLLM:

two quantize lm_head to int8, and the XPU mixed-precision kernels implement only uint4/uint4b8, so engine init dies with Failed to find a kernel that can implement the WNA16 linear layer ... uint8b128
one uses auto-round, which is not a registered quant_method in vLLM

This checkpoint is plain GPTQ, so it loads on stock vLLM (CUDA and Intel XPU) with no custom patches.

Quantization recipe

Produced with gptqmodel 7.3.2, mirroring the recipe of the already-deployed Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16:

Table with columns: setting, value
setting	value
bits	4
group_size	64
sym	`true`
desc_act	`false`
true_sequential	`true`
lm_head	`false` (left unquantized)
dynamic	`{"-:.mtp.": {}}` (MTP excluded)
pack_dtype

Two choices are load-bearing on Intel XPU:

lm_head is left in BF16. There is no int8 weight-only kernel for XPU (XPUwNa16LinearKernel supports only uint4/uint4b8), so a quantized head makes the model unloadable.
Symmetric only. Asymmetric zero-points are likewise unsupported on that path.

lm_head, embed_tokens, linear_attn.in_proj_a, linear_attn.in_proj_b and the vision tower are all left unquantized; only the 10 Linear projections per decoder layer are quantized.

Speculative decoding

The MTP draft head is preserved unquantized (15 mtp.* tensors), so multi-token prediction still works. This is lost in most community quants.

bash

--speculative-config '{"method":"mtp","num_speculative_tokens":4}'

Note this requires the MTP layers to build unquantized. On the patched XPU image that means leaving lm_head unquantized as above; naive --quantization overrides can break the draft path.

Evaluation

Perplexity on held-out splits (calibration used TRAIN only; these are TEST), computed in-process against a live vLLM server so the method is identical across every model compared:

Table with columns: set, perplexity
set	perplexity
wikitext2 test	5.272932434355194
gsm8k test	1.9123032367884933

Serving

bash

vllm serve /models/Qwen3.8-27B-TURBO-DF735-882-NM-DAU-GPTQ-Int4-g64-mse \  --quantization gptq \  --dtype float16 \  --max-model-len 100000 \  --kv-cache-dtype fp8 \  --gpu-memory-utilization 0.84 \  --language-model-only \  --enable-auto-tool-choice --tool-call-parser qwen3_xml \  --speculative-config '{"method":"mtp","num_speculative_tokens":4}'

The model natively supports 262144 context; --max-model-len is lowered to fit a 32 GB card alongside the KV cache.

Licenses

base model Qwen/Qwen3.8-27B: Apache-2.0
finetune DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU: Apache-2.0
this quantization: Apache-2.0, attribution retained as required

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

krisclarkdev

Model Tree

Base

DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Why this repository exists

The upstream finetune was only distributed as BF16 (~55.6 GB) or GGUF. The few community Int4 quants of it do not load under vLLM:

two quantize lm_head to int8, and the XPU mixed-precision kernels implement only uint4/uint4b8, so engine init dies with Failed to find a kernel that can implement the WNA16 linear layer ... uint8b128
one uses auto-round, which is not a registered quant_method in vLLM

This checkpoint is plain GPTQ, so it loads on stock vLLM (CUDA and Intel XPU) with no custom patches.

Quantization recipe

Produced with gptqmodel 7.3.2, mirroring the recipe of the already-deployed Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16:

Table with columns: setting, value
setting	value
bits	4
group_size	64
sym	`true`
desc_act	`false`
true_sequential	`true`
lm_head	`false` (left unquantized)
dynamic	`{"-:.mtp.": {}}` (MTP excluded)
pack_dtype

Two choices are load-bearing on Intel XPU:

lm_head is left in BF16. There is no int8 weight-only kernel for XPU (XPUwNa16LinearKernel supports only uint4/uint4b8), so a quantized head makes the model unloadable.
Symmetric only. Asymmetric zero-points are likewise unsupported on that path.

lm_head, embed_tokens, linear_attn.in_proj_a, linear_attn.in_proj_b and the vision tower are all left unquantized; only the 10 Linear projections per decoder layer are quantized.

Speculative decoding

The MTP draft head is preserved unquantized (15 mtp.* tensors), so multi-token prediction still works. This is lost in most community quants.

bash

--speculative-config '{"method":"mtp","num_speculative_tokens":4}'

Note this requires the MTP layers to build unquantized. On the patched XPU image that means leaving lm_head unquantized as above; naive --quantization overrides can break the draft path.

Evaluation

Perplexity on held-out splits (calibration used TRAIN only; these are TEST), computed in-process against a live vLLM server so the method is identical across every model compared:

Table with columns: set, perplexity
set	perplexity
wikitext2 test	5.272932434355194
gsm8k test	1.9123032367884933

Serving

bash

vllm serve /models/Qwen3.8-27B-TURBO-DF735-882-NM-DAU-GPTQ-Int4-g64-mse \  --quantization gptq \  --dtype float16 \  --max-model-len 100000 \  --kv-cache-dtype fp8 \  --gpu-memory-utilization 0.84 \  --language-model-only \  --enable-auto-tool-choice --tool-call-parser qwen3_xml \  --speculative-config '{"method":"mtp","num_speculative_tokens":4}'

The model natively supports 262144 context; --max-model-len is lowered to fit a 32 GB card alongside the KV cache.

Licenses

base model Qwen/Qwen3.8-27B: Apache-2.0
finetune DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU: Apache-2.0
this quantization: Apache-2.0, attribution retained as required