zm3

Qwen3.8-27B-Uncensored-NVFP4A16

README

License: apache-2.0

Format

Table

Scheme	`NVFP4A16` (4-bit weights, bf16 activations)
Elements	FP4 `E2M1`, packed 2/byte
Block scale	`float8_e4m3`, group_size 16
Global scale	FP32, per tensor
Strategy	`tensor_group`
Checkpoint format	`nvfp4-pack-quantized` (compressed-tensors)

Effective ~4.5 bits/weight (4 + 8/16). The block-16 FP8 scale is what distinguishes NVFP4 from MXFP4, which uses block-32 with a power-of-two E8M0 scale — the finer, non-power-of-two scale retains noticeably more accuracy.

What was quantized

496 Linear modules — 24.351 B params, 87.7% of the model.

Table with columns: Component, Params, Precision
Component	Params	Precision
MLP (64 layers)	17.113 B	NVFP4
Linear-attention proj (48 layers)	5.562 B	NVFP4
Full-attention proj (16 layers)	1.678 B	NVFP4
`lm_head`	1.271 B	bf16
Embeddings	1.271 B	bf16
Vision tower (27 blocks)

The vision tower, lm_head, and embeddings are held at bf16 deliberately — together only 12.3% of parameters, but the usual accuracy casualties of 4-bit. The MTP head ships unmodified as model-mtp.safetensors, so speculative decoding is preserved.

A_log, dt_bias, conv1d, and all norms are not nn.Linear and remain untouched, which matters here: the model keeps its SSM state in fp32 (mamba_ssm_dtype: float32).

Verified at the tensor level: 496 weight_packed / 496 weight_scale (F8_E4M3) / 496 weight_global_scale (F32), 0 in model.visual.*.

Memory

Max context is 262,144 tokens. Only 16 of 64 layers use full attention; the other 48 are linear-attention layers whose recurrent state is constant in sequence length.

markdown
KV cache        64 KiB/token   (16 full-attn layers, GQA 4 kv-heads x 256 head_dim)
KV @ 256K       16.00 GiB
linear state     0.141 GiB     <- constant, at 1 token or at 262,144
                (all 64 layers as full attention would be 64.0 GiB)

Peak at full 256K context, single sequence:

Table with columns: Weights, KV, State, Peak, 256K seqs in 80 GB
	Weights	KV	State	Peak	256K seqs in 80 GB
bf16	51.75	16.00	0.14	~71.4 GiB	1
NVFP4A16	19.14	16.00	0.14	~38.8 GiB	3

An FP8 KV cache halves the 16 GiB and takes you to ~5–6 concurrent 256K sequences.

Measured on H100 (SM 9.0) — not the target hardware

transformers reference path, greedy, 40 tokens, causal-conv1d / flash-linear-attention fast path absent in both runs:

Table with columns: bf16, NVFP4A16
	bf16	NVFP4A16
Disk	51.8 GiB	19.15 GiB
VRAM after load	50.96 GiB	18.36 GiB
Peak VRAM	51.17 GiB	55.58 GiB
Throughput	5.1 tok/s	2.6 tok/s

Without NVFP4 tensor cores the runtime unpacks weights to bf16 while still holding the packed copies — hence higher peak VRAM and ~2× slower than bf16. Disk and load-time savings are real; inference savings are not. On Blackwell this table should invert. vLLM has quantized kernels transformers lacks and may behave differently on Hopper — benchmark your own serving stack rather than extrapolating from the above.

Reproducing

python
from transformers import AutoModelForImageTextToText
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "JonathanColetti/Qwen3.8-27B-Uncensored",
    dtype=torch.bfloat16, device_map="cuda:0",
)
recipe = QuantizationModifier(
    targets="Linear", scheme="NVFP4A16",
    # NOTE: entries are matched as LITERAL module names unless prefixed "re:".
    # Without the prefix these regexes match nothing and the vision tower is
    # quantized anyway -- while still exiting 0 with a valid-looking checkpoint.
    ignore=["lm_head", r"re:model\.visual\..*", r"re:mtp\..*", r"re:.*embed_tokens.*"],
)
oneshot(model=model, recipe=recipe, output_dir="out", save_compressed=True)

Data-free RTN — no calibration set. NVFP4's global scale derives from the weights themselves, and group_size 16 is fine-grained enough that RTN holds up. This also avoids a calibration-distribution mismatch against a fine-tuned model, and avoids pushing calibration batches through a hybrid linear-attention stack that llm-compressor has not been validated against.

model-mtp.safetensors must be copied through manually: transformers 5.14.1 does not instantiate the MTP module, so save_pretrained silently drops it.

Produced with llmcompressor 0.13.0, compressed-tensors 0.18.0, transformers 5.14.1, torch 2.13.0+cu130.

Caveats

Not validated on target hardware. Produced and tested on Hopper only. Blackwell's NVFP4 GEMM has its own accumulation behavior; numbers here are indicative, not identical.
Long-context accuracy is unmeasured. 48 of 64 layers are recurrent, so quantization error can compound along the sequence rather than staying bounded per token as in softmax attention. Published NVFP4 recipes were validated on pure transformers. Evaluate at 2K / 32K / 128K — a single short-context perplexity number will not surface this.
No accuracy evaluation has been run. Only format correctness and a smoke generation.
Quantization perturbs model behavior generally, including tone and refusal patterns. This inherits the base model's behavior, which is an abliterated fine-tune with no safety filtering; nothing here adds or restores any.
transformers_version reads 5.14.1 (the tooling pin) vs 5.15.0 upstream. A config round-trip diff confirmed zero dropped or altered keys otherwise.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

zm3

Model Tree

Base

JonathanColetti/Qwen3.8-27B-Uncensored

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality