JC1DA

Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-INT4-Autoround

README

License: apache-2.0

Quantization details

Table

Scheme	W4A16 (4-bit weights, 16-bit activations)
Group size	128
Symmetric	yes
Format	`auto_round:auto_gptq` (GPTQ-compatible packing)
Calibration data	NeelNanda/pile-10k
Samples / seq len	256 × 4096
Iters / batch size	512 / 4
Seed	42
AutoRound version	0.12.3

Layers kept in 16-bit (not quantized): lm_head, embed_tokens, the vision tower (model.visual.*), the MTP module (mtp.*), and the linear_attn.in_proj_a / in_proj_b projections in every language-model layer.

Model size

Total (all shards): ~18.7 GB (17.4 GiB)
Shards: 10 × model-*.safetensors + model_extra_tensors.safetensors

Usage

The weights are GPTQ-compatible, so any loader supporting that format (vLLM, llama.cpp, transformers with optimum, etc.) should work:

python
from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_id = "DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-W4A16"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Hello!"}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_tensors="pt"
).to(model.device)

out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0], skip_special_tokens=True))

Usage with vLLM

Verified with vLLM 0.27.1 (loads the AutoRound/GPTQ-packed weights natively, fuses norm_quant/act_quant ops; first run compiles flashinfer/Triton kernels, which are then cached):

python
from vllm import LLM, SamplingParams

llm = LLM(
    model="DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-W4A16",
    dtype="bfloat16",
    max_model_len=16384,
    enforce_eager=True,  # optional: skip cuda-graph capture for faster startup
)

out = llm.generate(
    ["Prove that the square root of 2 is irrational."],
    SamplingParams(temperature=1.0, top_p=0.95, top_k=20, max_tokens=512),
)
print(out[0].outputs[0].text)

Or serve it as an OpenAI-compatible API:

bash
vllm serve DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-W4A16 \
    --dtype bfloat16 --max-model-len 16384 --port 8000

The model fits comfortably on a single 80 GB GPU (~21 GB weights + KV cache).

Quantization command

bash
auto-round \
     --model /path/to/DavidAU_Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 \
     --scheme W4A16 \
     --format auto_round \
     --dataset NeelNanda/pile-10k \
     --nsamples 256 \
     --seqlen 4096 \
     --batch_size 4 \
     --iters 512 \
     --device_map 0 \
     --seed 42 \
     --ignore_layers "model.language_model.layers.*.linear_attn.in_proj_a,model.language_model.layers.*.linear_attn.in_proj_b,lm_head,model.language_model.embed_tokens,model.visual.*,mtp.*" \
     --output_dir /path/to/DavidAU_Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-W4A16

Benchmarks

Evaluated against the bf16 base model on 2× A100 80 GB (one model per GPU, in parallel) using lm-evaluation-harness on vLLM 0.27.1 (dtype bfloat16). Sampling for generative tasks follows the model's generation_config.json (temperature 1.0, top_p 0.95, top_k 20); single run.

Table with columns: Benchmark, Setup, Base (bf16), This model (W4A16), Δ
Benchmark	Setup	Base (bf16)	This model (W4A16)	Δ
WikiText2 PPL ↓	full test set	8.4621	8.6318	+0.17
MMLU	full test set (14,042 questions), 5-shot log-likelihood	83.78%	83.44%	−0.34 pt
HumanEval	full test set (164 problems), 0-shot, pass@1, sampled	80.49%	78.66%	−1.83 pt

Observations:

HumanEval: the −1.8 pt loss is within the ±3.1 pt standard error — flat.
WikiText2 PPL: +0.17 absolute, the expected small increase for 4-bit weights.
MMLU: the −0.34 pt loss is within the standard error (~±0.38 pt at n=14,042) — effectively flat.
ARC-Easy: the W4A16 loss is within the margin of error.
BBH: the loss is concentrated in hard multi-step reasoning — logical deduction seven objects drops the most (65.2% → 49.2%); most other subjects are within ±1 pt, and several (e.g. disambiguation QA, salient translation error detection) are flat or slightly better.

Notes

Benchmark results above are close to the bf16 base model (PPL +0.17, MMLU −0.34 pt, HumanEval −1.8 pt, ARC −0.13 pt, BBH −0.72 pt macro); please verify on your own workloads for your use case.
Inference is the intended use; the checkpoint is not fine-tunable in the usual way (4-bit packed weights).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JC1DA

Model Tree

Base

DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality