TelperionAI

Qwen3.8-27B-NVFP4-AWQ-GPTQ

README

License: apache-2.0

Recipe

Table with columns: component, precision
component	precision
`mlp.{gate,up,down}_proj`, layers 0–55	NVFP4 (4-bit, group-16, FP8-e4m3 scales → 4.5 effective bits)
`mlp.{gate,up,down}_proj`, layers 56–63	FP8 e4m3 (dynamic)
`self_attn.{q,k,v,o}_proj`	FP8 e4m3 (dynamic)
`linear_attn.{in_proj_qkv,in_proj_z,out_proj}` (GDN)	FP8 e4m3 (dynamic)
`lm_head`, `embed_tokens`, all norms, GDN state params, vision tower	BF16

Two passes, in order:

AWQ — per-input-channel scaling on post_attention_layernorm → {gate_proj, up_proj} and up_proj → down_proj. Gate and up share one input, so the reciprocal scale folds into the norm weights: the accuracy gain costs zero bytes and zero throughput. The scales merge into weights entirely, so unlike rotation-based methods (QuIP/SpinQuant) this checkpoint still runs under tensor parallelism.
GPTQ on every quantized module (actorder="static", dampening_frac=0.01).

Calibration: 1024 sequences × 1024 tokens of a balanced Nemotron-v2 blend (25% code, 25% math, 20% STEM, 20% chat, 10% multilingual).

lm_head and embed_tokens are left in BF16 — matching Qwen's own official FP8 release, which does the same.

Benchmarks

Measured against the BF16 base model on 142,727 tokens of self-distilled thinking-mode output, plus 200 free greedy generations. vLLM 0.27.1, TP=2, 2×B300.

Table with columns: checkpoint, size ↓, top-1 ↑, near-tie ↓, moderate ↓, confident ↓, certain ↓, divmed ↑, tok/s ↑
checkpoint	size ↓	top-1 ↑	near-tie ↓	moderate ↓	confident ↓	certain ↓	divmed ↑	tok/s ↑
`Qwen/Qwen3.8-27B-FP8` (8-bit ref)	30.9 GB	96.15%	22.70%	3.48%	1.45%	0.08%	47	8711

Bold marks the best value in each column among the FP4 checkpoints; the FP8 row is a reference at a different precision and size class, so it is excluded from the comparison. All sizes are on-disk tensor bytes and include the ~0.85 GB BF16 MTP head, which every checkpoint in this table ships. Subtract ~0.85 GB for a no-MTP comparison.

Columns. top-1 is raw argmax agreement with BF16. The four bucket columns are disagreement rates, split by how confident the base model was at that position (top1−top2 logprob margin): near-tie <0.5, moderate 0.5–2, confident 2–5, certain >5. Only confident and certain represent real damage — a flip where the base model itself was nearly tied is numerical noise, not a quality loss. divmed is the median token index at which free greedy generation first diverges from BF16 (higher is better).

Perplexity is deliberately excluded. On this comparison it is anti-correlated with quality — the checkpoint with the best perplexity (RadixArk, −1.75%) has the worst certain-bucket damage of any arm measured (0.70%, 3.7× this model's). Do not rank FP4 checkpoints of this model by perplexity.

In an internal ablation, removing the AWQ pass and keeping everything else identical raises confident damage from 2.69% to 3.97% — so AWQ closes about half of the gap to FP8, at no size or speed cost.

Usage

python
from vllm import LLM
llm = LLM("selimaktas/Qwen3.8-27B-NVFP4-AWQ-GPTQ", tensor_parallel_size=2)

Requires a Blackwell-class GPU for native NVFP4, and vLLM with compressed-tensors.

Speculative decoding (MTP)

The model's MTP (multi-token prediction) head is included, in BF16, and works with vLLM's mtp speculative decoding:

python
from vllm import LLM
llm = LLM("TelperionAI/Qwen3.8-27B-NVFP4-AWQ-GPTQ", tensor_parallel_size=2,
          speculative_config={"method": "mtp", "num_speculative_tokens": 2})

Qwen3_5ForConditionalGeneration does not carry mtp.* in its state dict, so llm-compressor never sees it and it is silently dropped, even though config.json still declares mtp_num_hidden_layers: 1. It is grafted back in here from the base checkpoint and excluded from quantization (re:.*mtp.* in quantization_config.ignore; without that exclusion the quantization target regexes also match mtp.layers.0.mlp.* and vLLM fails to load). Draft quality drives acceptance rate, so it is kept at full precision rather than quantized.

Acceptance rate has not been measured; the head is verified to load and generate.

Limitations

Single evaluation corpus. All numbers come from one self-distilled corpus. The margins over the public NVFP4 checkpoints are large and statistically solid, but the comparison has not been replicated on a second distribution.
Vision tower is untouched (BF16); this was evaluated as a text model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

TelperionAI

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality