claymorecrystal

Qwen3.6-35B-A3B-PrismaQuant-4.75bit-vllm

README

License: apache-2.0

At a glance

Table with columns: Metric, BF16 source, This artifact, Delta
Metric	BF16 source	This artifact	Delta
Size on disk	70 GB	22 GB	−69 %
Fraction of original weights	100 %	31 %
Average bits per param	16	4.76
Multimodal (vision + text)	✓	✓
MTP speculative decoding heads	✓	✓
Loads in vLLM (stock `compressed-tensors`)	✓	✓
Runtime backend	any	vLLM only

Precision mix

This checkpoint uses three precisions, selected per-Linear by the allocator from measured sensitivity — not chosen uniformly:

Table with columns: Format, W, A, Use, Count
Format	W	A	Use	Count
NVFP4	4-bit (FP4, group_size=16 with per-group FP8 scale + per-tensor global)	4-bit (dynamic)	Bulk MoE experts + medium-sensitivity dense Linears + full visual encoder	192 Linears (52 dense + 80 per-expert + 108 visual + 2 visual un-probed stamped + MTP entries)
MXFP8	8-bit (E4M3, group_size=32 with per-group E8M0 scale)	8-bit (dynamic)	High-sensitivity dense Linears the allocator won't risk at 4-bit	45 Linears
BF16	16-bit	16-bit

The allocator couples MoE gate_up_proj / down_proj so siblings share a scheme (vLLM's FusedMoE requires this), and fused attention siblings (q_proj/k_proj/v_proj) share one per-tensor global scale so the packed qkv_proj loads without the "accuracy mismatch" warning.

Activation-aware passes applied during export

On every NVFP4 weight the exporter runs, in order:

GPTQ-OBS one-shot rounding — block-wise error propagation along the group-quant structure using the calibration Hessian. Closed-form, not iterative. Handles cross-column activation coupling.
Closed-form per-group scale sweep — for each 16-weight NVFP4 group, enumerate grid=32 candidate scales spanning [0.5·s₀, 1.5·s₀], round each weight to its nearest codebook neighbor at every candidate scale, pick the (scale, rounding-set) configuration minimizing activation-weighted per-group MSE sum_j a_j² · (w_orig,j - w_q,j)². Improve-or-keep gate against the post-GPTQ weight. Handles within-group weight-distribution variation that GPTQ takes as fixed.

Scale_sweep is the closed-form analog of Intel's AutoRound — where AutoRound learns per-weight continuous rounding offsets V via 200 SGD iterations on a relaxed loss, scale_sweep enumerates the discrete scale dimension directly and lets RTN pick rounding conditional on scale. No gradient descent, sub-second per Linear.

Measured per-Linear output-MSE vs RTN baseline (Qwen3.6-35B, mixed visual + MTP Linears, geomean):

Table with columns: Pipeline variant, out_mse ratio vs RTN
Pipeline variant	out_mse ratio vs RTN
RTN (no passes)	1.00
GPTQ only	0.41
GPTQ + act_round polish (prior pipeline)	0.99 (act_round undid GPTQ)
scale_sweep only	0.33
GPTQ + scale_sweep (this artifact)	0.33

The prior pipeline's act_round polish was a closed-form per-weight Δw²·E[a²] minimization at the fixed group scale. It turned out to systematically undo GPTQ's cross-column error propagation — the per-weight metric minima don't respect GPTQ's compensation structure. scale_sweep replaces it as a strict improvement.

AWQ's γ-fold is not applied. On NVFP4's 16-channel groups, AWQ's per-channel rescaling pushes mixed-scale values into the same group and inflates per-group quant noise rather than reducing it (measured: baseline PPL 4.97, AWQ-only 16.44 — +230 %).

Which layers are quantized

Text body (DeltaNet linear-attention + dense MoE, 40 layers)

Full attention Linears (q_proj, k_proj, v_proj, o_proj): mixed NVFP4 / MXFP8 / BF16 per-Linear by sensitivity
DeltaNet linear-attention Linears (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj): same
MoE experts (gate_up_proj, down_proj): per-expert NVFP4 with joint per-tensor scale across the gate_up pair so vLLM FusedMoE loads them
Shared expert MLP: same per-Linear policy

Multi-token-prediction (MTP) head

Speculative-decoding head (1 layer) + its own MoE block: same per-Linear policy, so --speculative-config method=mtp drafts at the same precision as the body.

Visual encoder (27 blocks — Qwen3.6-VL vision tower)

Fisher-driven per-Linear allocation: 108 of 110 visual Linears got placed by the full DP allocator on the basis of per-Linear activation-weighted cost (8 multimodal calibration samples, 110 Linears tracked via the model.visual.* module tree).
Remaining 2 un-probed visual Linears (patch_embed.proj edges the probe didn't tap) stamped at NVFP4 uniformly.
model.visual.pos_embed stays BF16 — it's a learnable parameter, not an nn.Linear, and vLLM's compressed-tensors loader cannot consume a quantized Parameter layout. The allocator's discover pass excludes it explicitly.
This is the same treatment body Linears get. There is ONE incremental code path: the streaming multimodal probe keeps the visual tower (~2 GB) resident on GPU while it streams the 70 GB body layer-by-layer, capturing Fisher through inputs_embeds.backward(grad) that propagates into visual weights.

Passthrough (unquantized)

lm_head — kept at BF16 because vLLM's ParallelLMHead module only accepts a single weight parameter. The allocator measures lm_head's Fisher sensitivity and would pick NVFP4 for it (saving ~770 MB), but the compressed-tensors runtime rejects a compressed lm_head with KeyError: lm_head.input_global_scale because its scheme registry doesn't include ParallelLMHead. This is a vLLM runtime limitation, not a PrismaQuant design decision.
RMSNorm weights (all layers + MTP + visual)
All biases
embed_tokens
model.visual.pos_embed (Parameter/Embedding, see above)

Serving (vLLM only)

This artifact is only runnable via vLLM's stock compressed-tensors support — there is no transformers-native runtime path for mixed NVFP4 + MXFP8 with packed-MoE experts today. vLLM 0.11+ or equivalent is required.

bash
vllm serve rdtand/Qwen3.6-35B-A3B-PrismaQuant-4.75bit-vllm \
    --trust-remote-code \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

FlashInfer NVFP4 attention is picked up automatically; set VLLM_USE_FLASHINFER_NVFP4=1 to make the preference explicit.
MTP speculative decoding at n=3 is the measured optimum for this family on DGX Spark (n=2 leaves ~10 % tok/s on the table, n=4 regresses).
Visual inputs work via vLLM's standard image-text-to-text chat API — no special flags.

Reproducing this artifact

Full pipeline is in the PrismaQuant repo:

Sensitivity probe — streaming per-shard empirical-Fisher trace (diagonal) across body + MTP + visual Linears. Each shard holds only its ~2 layers resident; the rest of the model is on disk or meta. 8 multimodal calibration samples drive visual Fisher through one unified streaming context.
Per-(Linear, format) cost measurement — for each Linear and each candidate format, the per-group RTN error weighted by cached input activations. Incremental: same per-shard streaming as the probe.
Multi-choice knapsack allocator — picks one format per Linear minimizing total predicted Δloss under the bit budget. Target 4.75 bpp; achieved 4.755 bpp here. Known-non-Linear rank-2 tensors (pos_embed, rotary_emb) are excluded from the visual pool.
Export — streams each body / visual / MTP shard, applies GPTQ + activation-weighted rounding to its NVFP4 entries, writes the compressed-tensors format. lm_head passthrough at BF16 enforced at this stage (see known issues).

Wall-clock on a DGX Spark (128 GB unified memory): ~15 min on cached probe + cost + activation shards (body shards are invariant across export-pass flag changes, so only the final export stage reruns when you change a flag).

Known issues / limitations

vLLM only at serve time. No transformers-runtime path for this precision mix today.
lm_head stays BF16 because vLLM's ParallelLMHead does not register the NVFP4/MXFP8 compressed-tensors schemes. Allocator measured it and would have picked NVFP4; the runtime limitation forces BF16. Costs ~770 MB on the disk footprint.
MTP n=4 regresses on this family. Stick to n=3 unless you verify against the draft-head acceptance-rate trace.

Citation

bibtex
@software{prismaquant2026,
  title        = {PrismaQuant: per-Linear sensitivity-driven mixed-precision
                  quantization for LLMs},
  author       = {Tand, Rob},
  year         = 2026,
  url          = {https://github.com/RobTand/prismaquant},
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

claymorecrystal

Model Tree

Base

Qwen/Qwen3.6-35B-A3B

Quantized

this model

Input Modalities

Text

Image

Video

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

At a glance

Table with columns: Metric, BF16 source, This artifact, Delta
Metric	BF16 source	This artifact	Delta
Size on disk	70 GB	22 GB	−69 %
Fraction of original weights	100 %	31 %
Average bits per param	16	4.76
Multimodal (vision + text)	✓	✓
MTP speculative decoding heads	✓	✓
Loads in vLLM (stock `compressed-tensors`)	✓	✓
Runtime backend	any	vLLM only

Precision mix

This checkpoint uses three precisions, selected per-Linear by the allocator from measured sensitivity — not chosen uniformly:

Table with columns: Format, W, A, Use, Count
Format	W	A	Use	Count
NVFP4	4-bit (FP4, group_size=16 with per-group FP8 scale + per-tensor global)	4-bit (dynamic)	Bulk MoE experts + medium-sensitivity dense Linears + full visual encoder	192 Linears (52 dense + 80 per-expert + 108 visual + 2 visual un-probed stamped + MTP entries)
MXFP8	8-bit (E4M3, group_size=32 with per-group E8M0 scale)	8-bit (dynamic)	High-sensitivity dense Linears the allocator won't risk at 4-bit	45 Linears
BF16	16-bit	16-bit

Activation-aware passes applied during export

On every NVFP4 weight the exporter runs, in order:

GPTQ-OBS one-shot rounding — block-wise error propagation along the group-quant structure using the calibration Hessian. Closed-form, not iterative. Handles cross-column activation coupling.
Closed-form per-group scale sweep — for each 16-weight NVFP4 group, enumerate grid=32 candidate scales spanning [0.5·s₀, 1.5·s₀], round each weight to its nearest codebook neighbor at every candidate scale, pick the (scale, rounding-set) configuration minimizing activation-weighted per-group MSE sum_j a_j² · (w_orig,j - w_q,j)². Improve-or-keep gate against the post-GPTQ weight. Handles within-group weight-distribution variation that GPTQ takes as fixed.

Measured per-Linear output-MSE vs RTN baseline (Qwen3.6-35B, mixed visual + MTP Linears, geomean):

Table with columns: Pipeline variant, out_mse ratio vs RTN
Pipeline variant	out_mse ratio vs RTN
RTN (no passes)	1.00
GPTQ only	0.41
GPTQ + act_round polish (prior pipeline)	0.99 (act_round undid GPTQ)
scale_sweep only	0.33
GPTQ + scale_sweep (this artifact)	0.33

Which layers are quantized

Text body (DeltaNet linear-attention + dense MoE, 40 layers)

Full attention Linears (q_proj, k_proj, v_proj, o_proj): mixed NVFP4 / MXFP8 / BF16 per-Linear by sensitivity
DeltaNet linear-attention Linears (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj): same
MoE experts (gate_up_proj, down_proj): per-expert NVFP4 with joint per-tensor scale across the gate_up pair so vLLM FusedMoE loads them
Shared expert MLP: same per-Linear policy

Multi-token-prediction (MTP) head

Speculative-decoding head (1 layer) + its own MoE block: same per-Linear policy, so --speculative-config method=mtp drafts at the same precision as the body.

Visual encoder (27 blocks — Qwen3.6-VL vision tower)

Fisher-driven per-Linear allocation: 108 of 110 visual Linears got placed by the full DP allocator on the basis of per-Linear activation-weighted cost (8 multimodal calibration samples, 110 Linears tracked via the model.visual.* module tree).
Remaining 2 un-probed visual Linears (patch_embed.proj edges the probe didn't tap) stamped at NVFP4 uniformly.
model.visual.pos_embed stays BF16 — it's a learnable parameter, not an nn.Linear, and vLLM's compressed-tensors loader cannot consume a quantized Parameter layout. The allocator's discover pass excludes it explicitly.
This is the same treatment body Linears get. There is ONE incremental code path: the streaming multimodal probe keeps the visual tower (~2 GB) resident on GPU while it streams the 70 GB body layer-by-layer, capturing Fisher through inputs_embeds.backward(grad) that propagates into visual weights.

Passthrough (unquantized)

lm_head — kept at BF16 because vLLM's ParallelLMHead module only accepts a single weight parameter. The allocator measures lm_head's Fisher sensitivity and would pick NVFP4 for it (saving ~770 MB), but the compressed-tensors runtime rejects a compressed lm_head with KeyError: lm_head.input_global_scale because its scheme registry doesn't include ParallelLMHead. This is a vLLM runtime limitation, not a PrismaQuant design decision.
RMSNorm weights (all layers + MTP + visual)
All biases
embed_tokens
model.visual.pos_embed (Parameter/Embedding, see above)

Serving (vLLM only)

bash
vllm serve rdtand/Qwen3.6-35B-A3B-PrismaQuant-4.75bit-vllm \
    --trust-remote-code \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

FlashInfer NVFP4 attention is picked up automatically; set VLLM_USE_FLASHINFER_NVFP4=1 to make the preference explicit.
MTP speculative decoding at n=3 is the measured optimum for this family on DGX Spark (n=2 leaves ~10 % tok/s on the table, n=4 regresses).
Visual inputs work via vLLM's standard image-text-to-text chat API — no special flags.

Reproducing this artifact

Full pipeline is in the PrismaQuant repo:

Sensitivity probe — streaming per-shard empirical-Fisher trace (diagonal) across body + MTP + visual Linears. Each shard holds only its ~2 layers resident; the rest of the model is on disk or meta. 8 multimodal calibration samples drive visual Fisher through one unified streaming context.
Per-(Linear, format) cost measurement — for each Linear and each candidate format, the per-group RTN error weighted by cached input activations. Incremental: same per-shard streaming as the probe.
Multi-choice knapsack allocator — picks one format per Linear minimizing total predicted Δloss under the bit budget. Target 4.75 bpp; achieved 4.755 bpp here. Known-non-Linear rank-2 tensors (pos_embed, rotary_emb) are excluded from the visual pool.
Export — streams each body / visual / MTP shard, applies GPTQ + activation-weighted rounding to its NVFP4 entries, writes the compressed-tensors format. lm_head passthrough at BF16 enforced at this stage (see known issues).

Known issues / limitations

vLLM only at serve time. No transformers-runtime path for this precision mix today.
lm_head stays BF16 because vLLM's ParallelLMHead does not register the NVFP4/MXFP8 compressed-tensors schemes. Allocator measured it and would have picked NVFP4; the runtime limitation forces BF16. Costs ~770 MB on the disk footprint.
MTP n=4 regresses on this family. Stick to n=3 unless you verify against the draft-head acceptance-rate trace.

Citation

bibtex
@software{prismaquant2026,
  title        = {PrismaQuant: per-Linear sensitivity-driven mixed-precision
                  quantization for LLMs},
  author       = {Tand, Rob},
  year         = 2026,
  url          = {https://github.com/RobTand/prismaquant},
}

Qwen3.6-35B-A3B-PrismaQuant-4.75bit-vllm

README

At a glance

Precision mix

Activation-aware passes applied during export

Which layers are quantized

Text body (DeltaNet linear-attention + dense MoE, 40 layers)

Multi-token-prediction (MTP) head

Visual encoder (27 blocks — Qwen3.6-VL vision tower)

Passthrough (unquantized)

Serving (vLLM only)

Reproducing this artifact

Known issues / limitations

Links

Citation

Explore FriendliAI today

README

At a glance

Precision mix

Activation-aware passes applied during export

Which layers are quantized

Text body (DeltaNet linear-attention + dense MoE, 40 layers)

Multi-token-prediction (MTP) head

Visual encoder (27 blocks — Qwen3.6-VL vision tower)

Passthrough (unquantized)

Serving (vLLM only)

Reproducing this artifact

Known issues / limitations

Links

Citation