curvedinf

Qwen3.8-27B-GPTQ-INT8-W8A8-GS128

README

License: apache-2.0

Quantization details

Table with columns: Parameter, Value
Parameter	Value
Format	GPTQ
Bits	8
Group size	128
Symmetric	true
desc_act	false
true_sequential	true
lm_head	false (kept fp16 — tied weights feed the logits GEMM)
Vision encoder / MTP module	untouched BF16 (carried in checkpoint)
Quantizer	GPTQModel 7.3.4, RTN-free (Hessian-compensated)
Calibration	512 samples, mixed evol-codealpaca-v1 (code) + C4 (general text), binned uniformly 256–2048 tokens

Why group_size 128: the accompanying runtime dispatches AITER true int8×int8 compute (W8A8) for every decode and prefill GEMM shape, which requires one weight scale per 128-wide K block. Quality was verified by a teacher-forced KL-divergence sweep over 6 precision configurations before baking — this config measured KLD 0.011 vs the gs32 reference (gate 0.02) and showed activation quantization adds only +0.0032 KLD on top.

Serving (vLLM, gfx908 fork)

Use with the companion fork stack:

vLLM: curvedinf/vllm-gfx908 — branch mi100-optimized-sync (AITER W8A8 INT8 GEMMs everywhere, int8 embedding gather, int8 per-token-head KV, int8 mamba/GDN, AITER custom all-reduce)
AITER: curvedinf/aiter-gfx908 — branch mi100-optimized-sync (int8 unified-attention kernels + gfx908 tuning)
flash-attention: curvedinf/flash-attention — branch gfx908-sync (python-only 2.8.4, Triton AMD backend via the AITER fork)

bash
VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_CUSTOM_AR=1 \
VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 \
VLLM_DISABLED_KERNELS=TritonW8A16LinearKernel \
vllm serve <this-model-dir> \
  --tensor-parallel-size 4 \
  --max-num-seqs 8 \
  --dtype half \
  --max-model-len 65536 \
  --kv-cache-dtype int8_per_token_head \
  --mamba-ssm-cache-dtype int8 \
  --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","pass_config":{"fuse_allreduce_rms":true}}' \
  --speculative-config '{
    "method": "dflash",
    "model": "curvedinf/Qwen3.8-27B-DFlash2-GPTQ-INT8-W8A8-GS128",
    "num_speculative_tokens": 7,
    "kv_cache_dtype": "int8_per_token_head"
  }'

This is one fixed TP4/C8 contract: both GS128 models, AITER W8A8 INT8 GEMMs at every M, AITER unified attention in INT8, AITER custom all-reduce in INT8, fused AR+RMSNorm+per-group INT8 quant-out, INT8 Mamba, and INT8 target/draft KV. --dtype half is the model's residual/native dtype; GEMM inputs are dynamically quantized to INT8 by the AITER W8A8 path.

The top-level KV flag configures this target; the nested field configures the DFlash2 draft. Both use INT8 per-token-head KV. Do not substitute TRITON_ATTN, W8A16, RCCL all-reduce, fp16 KV, no speculation, another TP size, or another concurrency in the intended recipe.

Model architecture

Qwen3.8-27B is a hybrid dense multimodal model (model_type qwen3_5): 64 layers — 48 GDN linear-attention + 16 full-attention (repeating 3:1), hidden 5120, 27B parameters, vocab 248,320, context 262,144. This checkpoint serves the language model (--language-model-only); vision weights are carried for completeness.

Reproduction

Quantization script and full experiment ledger (KLD sweep, kernel microbenchmarks, A/B measurements) live in the vLLM fork: ~/models/quantize_qwen38_27b_gptq8.py and logs/c8_optimization/experiments.md.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

curvedinf

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality