zm3

Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4A16

README

License: apache-2.0

Format

Table

Scheme	`NVFP4A16` (4-bit weights, bf16 activations)
Elements	FP4 `E2M1`, packed 2/byte
Block scale	`float8_e4m3`, group_size 16
Global scale	FP32, per tensor
Strategy	`tensor_group`
Checkpoint format	`nvfp4-pack-quantized` (compressed-tensors)

Effective ~4.5 bits/weight (4 + 8/16). The block-16 FP8 scale is what distinguishes NVFP4 from MXFP4, which uses block-32 with a power-of-two E8M0 scale — the finer, non-power-of-two scale retains noticeably more accuracy.

What was quantized

496 Linear modules — 24.351 B params, 87.7% of the model.

Table with columns: Component, Params, Precision
Component	Params	Precision
MLP (64 layers)	17.113 B	NVFP4
Linear-attention (48 layers)	5.562 B	NVFP4
Full-attention (16 layers)	1.678 B	NVFP4
`lm_head`	1.271 B	bf16
Embeddings	1.271 B	bf16
Vision tower

The vision tower, lm_head, and embeddings are held at bf16 deliberately — together 12.3% of parameters with the MTP head, but the usual accuracy casualties of 4-bit. The MTP head ships unmodified as model-mtp.safetensors, so speculative decoding is preserved.

A_log, dt_bias, conv1d, and all norms are not nn.Linear and remain untouched, which matters here: the model keeps its SSM state in fp32 (mamba_ssm_dtype: float32).

Verified at the tensor level: 496 weight_packed / 496 weight_scale (F8_E4M3) / 496 weight_global_scale (F32), 0 in model.visual.*, and 15 mtp.* tensors carried through.

Memory

Max context is 262,144 tokens. Only 16 of 64 layers use full attention; the other 48 are linear-attention layers whose recurrent state is constant in sequence length.

markdown
KV cache        64 KiB/token   (16 full-attn layers x 2 x 4 kv-heads x 256 head_dim x bf16)
KV @ 256K       16.00 GiB
linear state     0.141 GiB     <- constant, at 1 token or at 262,144
                (48 layers x 48 v-heads x 128 x 128, fp32)

Peak at full 256K context, single sequence:

Table with columns: Weights, KV, State, Peak, 256K seqs in 80 GB
	Weights	KV	State	Peak	256K seqs in 80 GB
bf16	51.75	16.00	0.14	~67.9 GiB	1
NVFP4A16	19.15	16.00	0.14	~35.3 GiB	3

An FP8 KV cache halves the 16 GiB.

Smoke test

Measured on H100 80GB (SM 9.0) — not the target hardware. transformers reference path, greedy, 40 new tokens, causal-conv1d / flash-linear-attention fast path absent:

Table

Load time	6.7 s
VRAM after load	18.36 GiB
Peak VRAM	55.59 GiB
Throughput	5.4 tok/s

Peak VRAM exceeds the checkpoint size because, without NVFP4 tensor cores, the runtime unpacks weights to bf16 while still holding the packed copies. Output was coherent. This is a correctness check, not a benchmark — vLLM has quantized kernels transformers lacks, so benchmark your own serving stack.

Reproducing

python
from transformers import AutoModelForImageTextToText, AutoProcessor
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16", dtype=torch.bfloat16, device_map="cuda:0",
)
recipe = QuantizationModifier(
    targets="Linear", scheme="NVFP4A16",
    # NOTE: entries are matched as LITERAL module names unless prefixed "re:".
    # Without the prefix these regexes match nothing and the vision tower is
    # quantized anyway -- while still exiting 0 with a valid-looking checkpoint.
    ignore=["lm_head", r"re:model\.visual\..*", r"re:mtp\..*", r"re:.*embed_tokens.*"],
)
oneshot(model=model, recipe=recipe, processor=AutoProcessor.from_pretrained("AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16"),
        output_dir="out", save_compressed=True)

Data-free RTN — no calibration set. NVFP4's global scale derives from the weights themselves, and group_size 16 is fine-grained enough that RTN holds up. This also avoids a calibration-distribution mismatch, and avoids pushing calibration batches through a hybrid linear-attention stack that llm-compressor has not been validated against.

The MTP tensors must be copied through manually: transformers 5.14.1 does not instantiate the MTP module, so save_pretrained silently drops them. torchvision must be installed or the Qwen3VLProcessor fails to initialize and the processor files are silently omitted from the output.

Produced with llmcompressor 0.13.0, compressed-tensors 0.18.0, transformers 5.14.1, torch 2.13.0+cu130, on an NVIDIA H100 80GB.

Caveats

Not validated on target hardware. Produced and tested on Hopper only. Blackwell's NVFP4 GEMM has its own accumulation behavior.
No accuracy evaluation has been run. Only format correctness and a smoke generation. No perplexity, no benchmark suite, no vision-path evaluation.
Long-context accuracy is unmeasured. 48 of 64 layers are recurrent, so quantization error can compound along the sequence rather than staying bounded per token as in softmax attention. Published NVFP4 recipes were validated on pure transformers. Evaluate at 2K / 32K / 128K — a single short-context perplexity number will not surface this.
Quantization perturbs model behavior generally, including tone and refusal patterns.
Base-model provenance. The base is tagged early-access / draft and is an abliterated, refusal-removed fine-tune. Its shards are named inconsistently (model-00003-of-00003.safetensors alongside a 2-shard set); the index maps it correctly — it is the MTP head — but be aware if you script against those names. This checkpoint inherits the base model's behavior, which has no safety filtering; nothing here adds or restores any.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

zm3

Model Tree

Base

AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality