Highlights
-
Benchmarks. On a deterministic, prefix-cache-OFF tool-evaluation
harness it match or exceed the
rdtand/Qwen3.6-27B-PrismaAURA-5.5bit-vllm
baseline while being smaller:
Table with columns: Model, Tool-eval (3 metrics), Size| Model | Tool-eval (3 metrics) | Size |
|---|
| This model (PrismaQuant 5.2bpp) | 94 / 66.7 / 92.3 | 22 GB |
| PrismaAURA 5.5bit (baseline) | 93 / 66.7 / 80.8 | 23 GB |
Same harness for both, run solo with prefix caching disabled for
determinism. The largest gap is on the third metric (92.3 vs 80.8).
Method
PrismaQuant, 5.2 bpp target, mixed-precision:
- Bit placement by Fisher-diagonal sensitivity allocation — more bits
are spent on the parameters the loss is most sensitive to.
- Per-candidate selection by measured KL against a validated surrogate
(the allocator chooses the quantization that minimizes measured output-
distribution divergence, rather than relying on a proxy heuristic).
- Provenance: stock PrismaQuant allocator, run under our
production render-stream cost mode.
The result is a heterogeneous mix of element types across tensors:
- NVFP4 (W4A4) for the bulk of the linear weights,
- FP8 for sensitivity-flagged tensors,
- BF16 for the most sensitive / small tensors kept at full precision.
config.json carries the full quantization_config
(quant_method: compressed-tensors, format: mixed-precision); the exact
per-tensor scheme is described by mixed_native_manifest.json.
Serving (vLLM)
The model loads via vLLMs native compressed-tensors path:
vllm serve JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm
For fast GDN / linear-attention (gated-delta) serving, install the optional
kernel:
pip install causal_conv1d
(Without it, vLLM falls back to a slower path for the linear-attention layers;
correctness is unaffected.)
Contents
model-0000{1..5}-of-00005.safetensors + model.safetensors.index.jsonconfig.json (with quantization_config), generation_config.jsonmixed_native_manifest.json (per-tensor precision map)- MTP module weights (
mtp.*) and the Qwen3.6 vision tower (model.visual.*)
- tokenizer + chat template
License
Inherited from the base model
Qwen/Qwen3.6-27B. Refer to the base
model card for the governing license terms.
Provenance (added 2026-07-04 — recorded after the fact, verified from the build environment)
- Calibration dataset:
diverse-v1.jsonl (4.6 MB, sha256[:16] cc76f4a13c413398) —
mixed prose/code/math corpus built by tools/build_diverse_calibration.py.
Exact bytes archived in the PrismaQuant repo (calibration-datasets/).
- Known caveat (discovered 2026-07-03, applies retroactively): the Fisher
probe consumed only 32x1024 tokens (~3%) of that corpus (the "calibration
keyhole"). The result stands on its benchmarks; the caveat matters if you
try to attribute the win to the calibration mix.
- Pipeline: PrismaQuant (Fisher-diagonal allocation, production-render-stream
cost, validated-surrogate measured-KL selection, exportable_to_vllm gate);
TARGET_BITS selected agentically (5.2 beat the 4.9 KL-kneedle on the agentic
harness: 94 vs 89).
- Benchmark context: deterministic harness (temp 0, prefix caching OFF),
June 2026 stack. Scores from other stacks/harnesses are not comparable;
see the repo maintainer for the full serve flags.