hyssra

Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NVFP4-W4A4

README

License: apache-2.0

Why W4A4, and what stays in BF16

The language-model backbone (MLP everywhere + full-attention projections) is 4-bit in both weights and activations — the highest-compression NVFP4 configuration, targeting Blackwell-class GPUs (it also runs weight-only on older hardware).

Three components are deliberately kept at full precision:

MTP head. A naive llm-compressor run does not save the MTP module (the transformers class never instantiates it), so most conversions of this architecture either drop it or quantify it. Here the 15 mtp.* tensors are restored from the BF16 source shards, shipped in model-towers.safetensors, and pinned in quantization_config.ignore (re:.*mtp.*) so vLLM loads them as BF16 rather than as quantized layers. The head sits on the acceptance-rate-critical path of speculative decoding; quantizing it moves the drafts away from what the main model predicts.
Vision tower. Kept BF16 so image input is unaffected by 4-bit.
Gated DeltaNet non-Linear parameters. The GDN recurrence state parameters (A_log, dt_bias, the conv1d kernel, norms) are not Linear modules, so the recipe's targets: [Linear] leaves them in BF16 automatically. The GDN projection Linear layers (in_proj_*, out_proj) are quantized, like the rest of the backbone.

Reproducibility

Built with llm-compressor oneshot + GPTQModifier (recipe.yaml included: block size 128, dampening 0.01, actorder static, MSE observer, group size 16) with a domain-matched calibration blend (long reasoning + general chat + code), on a GB10 (sm_121, CUDA 13.0, torch 2.11.0+cu130, transformers 5.14.1, llm-compressor 0.13.0), with the KV-cache calibration pass enabled (kv_cache_scheme: fp8, symmetric, per-tensor, memoryless_minmax) riding on the same one-shot pass. ~2 h 45 min of calibration, unified-memory peak ~100 GB.

What the KV calibration adds. 32 scalar FP32 tensors — k_scale/v_scale (one per full-attention layer's k_proj/v_proj; this model has 16 full-attention layers, the rest being Gated DeltaNet linear attention, which keeps a recurrent state and has no paged KV cache) — plus quantization_config.kv_cache_scheme in config.json. vLLM's compressed-tensors path reads these into the attention layers and multiplies the loaded K/V entries by the per-layer scale when quantizing them into the fp8 paged cache. Measured scale values span ~0.012–0.23 (layer-dependent), i.e. real activations-derived scales, not the 1.0 default. The quantized weights themselves are bit- identical to the W4A4 build (weight_packed/weight_scale key set unchanged, 992 keys); only the 32 scale tensors and the config block are new (index: 2,719 vs 2,687 entries; +32).

The quantization pipeline and calibration recipe derive from the NVFP4 work of maci0 (see e.g. his Qwen3.6-40B NVFP4 card).

Post-quantization structural fixes (MTP restoration, index.json rebuild, ignore guarantee in config.json) are idempotent and re-derivable from the BF16 source.

Verification

Structural: index carries 2,687 entries (2,672 main + 15 towers); all 15 mtp.* tensors present, unquantized, in ignore; all 333 model.visual.* tensors present, unquantized; no mtp tensors quantized anywhere.
MTP (served, measured): vLLM --speculative-config '{"method":"mtp","num_speculative_tokens":1}' — the MTP head loads and drafts. Startup log shows
markdown
```
Detected MTP model. Sharing target model embedding weights / lm_head weights with the draft model
```
. Draft acceptance, read from vllm:spec_decode_num_{accepted,draft}_tokens_total: 74.4 % (1,377 accepted / 1,852 draft) at num_speculative_tokens=1, a running average over mixed chat and agent traffic (acceptance is workload-, protocol-, and depth- dependent; published values for this model span roughly 0.5–0.9 at num_speculative_tokens 1–3). A 0 % rate would mean the head was not loaded.
KV-cache scales (served, verified): the checkpoint ships 32 calibrated k_scale/v_scale tensors (16 full-attention layers × k/v), written by the GPTQ one-shot pass and consumed by vLLM's compressed-tensors KV path (
markdown
```
kv_cache_scheme: {type: float, num_bits: 8, strategy: tensor, symmetric: true}
```
). Served with in the image: startup clean (no uncalibrated-scale warnings), fp8 KV cache allocated (492,830 tokens at on GB10), generation correct, MTP draft acceptance (80/113 over a 300-token request) — the calibrated K/V path is on the draft-model acceptance path, so this also covers the MTP→KV interaction. PPL and lm-eval are unchanged vs the uncalibrated W4A4 build (identical weights; see Quality Verification).

Quality Verification (measured 2026-08-29; KV build re-verified 2026-08-31)

Both the BF16 source and this build were benchmarked under identical conditions: lm-eval 0.4.12 (vLLM backend) in the vllm/vllm-openai:qwen38-x86_64-cu130 image (torch 2.13.0+cu130), single RTX PRO 6000 Blackwell (96 GB), --gpu-memory-utilization 0.85, max_model_len 4096, greedy decoding. Prompts are 0-shot raw task prompts — no chat template, no thinking (the lm-eval runs recorded chat_template: null; equivalent to the "Instruct mode" the upstream card benches in).

Table with columns: Task, BF16, NVFP4 W4A4, Δ
Task	BF16	NVFP4 W4A4	Δ
arc-challenge	0.597	0.606	+0.9 pt
arc-easy	0.845	0.851	+0.7 pt
boolq	0.853	0.888	+3.5 pt
hellaswag (acc-norm)	0.830	0.806	−2.4 pt
openbookqa (acc-norm)

* gsm8k strict-match is regex-extracted against the #### <number> format; the +8.2 pt is a format-compliance gain (the quantized build more reliably emits the exact marker), not a math-capability gain.

Reading: knowledge/prose-bound tasks (MMLU humanities, hellaswag, wikitext PPL) carry the expected 3–4 % W4A4 cost — consistent with the +10.7 % wikitext PPL and with the highest-compression NVFP4 scheme — while reasoning/common-sense and generation-bound tasks (arc, piqa, boolq, gsm8k, lambada) are within ±3 % of the BF16 source. No task degraded by more than 3.3 %. Absolute values differ from third-party harness numbers (e.g. the upstream card's llama.cpp benches); compare against the BF16 column, which was measured under the same conditions.

The KV-calibrated build in this repo was re-benchmarked under identical conditions on 2026-08-31 (same image, same flags, RTX PRO 6000): the NVFP4 W4A4 column above applies to it as well — wikitext-2 test PPL 7.3372 (same n_windows) and all lm-eval task scores reproduced within sample noise, as expected from bit-identical weights. (PPL here is a long-context/decode-side metric and is sensitive to KV-cache behavior, which makes it the sharpest single check that the calibrated scales load and are actually applied at serve time.)

Serve with vLLM (MTP enabled)

bash
vllm serve <this-repo> \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --kv-cache-dtype fp8 --enable-prefix-caching \
  --max-num-batched-tokens 2048 --max-num-seqs 2 \
  --gpu-memory-utilization 0.44 \
  --reasoning-parser qwen3 --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

(Flag values are a tested GB10 / 128 GB unified-memory configuration — 256K context works at this setting. On a dedicated GB10 you can raise --gpu-memory-utilization toward ~0.8 for a larger KV pool; avoid ~0.95, which starves the OS on a unified pool.)

Lineage

Qwen/Qwen3.8-27B → DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 → this. Apache 2.0, unbroken from the base model.

Pipeline note: this quantization run (calibration, QC, card, upload) was performed with an LLM agent (unsloth/Qwen3.8-27B-NVFP4) operating the operator's own toolchain.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

hyssra

Model Tree

Base

DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality