PassingByPixels

Qwen3.8-27B-heretic-ara-NVFP4

README

License: apache-2.0

What we did

Download trohrbaugh/Qwen3.8-27B-heretic-ara (51.8 GB bf16).
PTQ to NVFP4 with NVIDIA ModelOpt on a DGX Spark, Foundry path, vlm=true (vision tower left in higher precision), 512 calib samples, seq_len 2048.
Restore the MTP head. ModelOpt exported text_config.mtp_num_hidden_layers: 1 but wrote zero of the 15 mtp.* tensors, including mtp.fc.weight. That is the known silent-failure shape: vLLM will load, draft from an uninitialised projection, and only get slower. We spliced all 15 mtp.* tensors unchanged (bf16) from the source checkpoint into an extra shard (model-mtp-fc.safetensors, 849.4 MB) and listed those modules in quantization_config.ignore so they are not read as NVFP4.
Nothing else. Same tokenizer, chat template, vision processor files (borrowed from a same-arch donor at register time where ModelOpt omitted them). Language-model and vision-tower tensor names match the official PassingByPixels/Qwen3.8-27B-NVFP4 export except the restored MTP set.

What this quantization is

NVFP4 stores each weight as an E2M1 4-bit float (1 sign, 2 exponent, 1 mantissa bit), with an 8-bit scale shared across each block of 16 weights.

Mixed precision, by design. ModelOpt leaves these in higher precision:

Table with columns: Component, Precision
Component	Precision
Gated DeltaNet (linear attention)	bf16
MTP head (spliced back)	bf16
Vision tower	bf16
Embeddings / LM head	bf16
FFN + full-attention projections	NVFP4

Table with columns: bf16 source, this checkpoint
	bf16 source	this checkpoint
On disk	51.8 GB	~20 GB (2 NVFP4 shards + MTP shard)
Weight keys	1,199	2,399 (weights + NVFP4 scales + 15 MTP)
`mtp.*` tensors	15/15	15/15 (spliced)
`mtp.fc.weight`	present	present

MTP validation (this artifact)

Served on DGX Spark GB10 with vLLM 0.25.1-gb10.2:

markdown
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Smoke generation (96 tokens, temperature 0): spec_drafted=86, spec_accepted=53, accept rate 0.616. If the head were missing or random, drafted would be 0 or accept would sit near 0. That is the functional check. It is not a quality bench.

Source

Quantized from: trohrbaugh/Qwen3.8-27B-heretic-ara
That model is a Heretic (custom fork) v1.2.0+custom ARA decensor of Qwen/Qwen3.8-27B
Official (non-heretic) NVFP4 of the same architecture: PassingByPixels/Qwen3.8-27B-NVFP4

Serve (vLLM)

bash
vllm serve PassingByPixels/Qwen3.8-27B-heretic-ara-NVFP4 \
  --served-model-name qwen3.8-27b-heretic-ara-nvfp4 \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.72 \
  --reasoning-parser qwen3 \
  --trust-remote-code \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Thinking mode is on by default. Disable per request with chat_template_kwargs.enable_thinking: false.

Limitations

Lossy 4-bit conversion. No side-by-side quality eval vs the bf16 heretic source.
No throughput sweep was run on this repo (the table on the official Qwen3.8-27B NVFP4 card is a different checkpoint).
Client max_tokens must be less than the served max_model_len.
MTP + FlashInfer on GB10 has a known crash class at some GQA / k settings; this checkpoint was loaded and drafted at k=2 with FlashInfer on v0.25.1-gb10.2.

License

Apache 2.0, inherited from trohrbaugh/Qwen3.8-27B-heretic-ara and Qwen/Qwen3.8-27B.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

PassingByPixels

Model Tree

Base

trohrbaugh/Qwen3.8-27B-heretic-ara

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality