leoncca

Qwen3.5-122B-A10B-Heretic-Selective-AWQ

README

License: apache-2.0

Frozen source revisions

Table with columns: Component, Repository, Revision
Component	Repository	Revision
Heretic BF16 mother	`trohrbaugh/Qwen3.5-122B-A10B-heretic`	`356e5abe1fbc83bb3613b46e470f6e2a5565ca47`
Native MTP tensors	`Qwen/Qwen3.5-122B-A10B`	`dc4d348443bc740c68e2d77492492c11606384d5`
Quantizer	`modelcloud/GPTQModel` 7.3.2	`f6f1ab12fe8115f84c52253871ca039ac6c425ee` plus the included build patch

Exactly 785 official mtp.* tensors were grafted from the pinned Qwen source and retained in BF16. See mtp-graft-manifest.json for tensor-level provenance.

Quantization recipe

Table with columns: Item, Value
Item	Value
Method	Activation-aware AWQ, GEMM W4A16
Weight bits	4
Group size	128
Zero point	Asymmetric, enabled
Activation order	Disabled (`desc_act=false`)
Packed dtype	INT32 `qweight` and `qzeros`
Scale dtype	BF16

The calibration run used a frozen 320-conversation corpus with 190,590 active tokens: 96 English reference samples, 64 Chinese reference samples, 64 code samples, 64 structured/tool/math samples, and 32 image-conditioned samples. Its pinned inputs were derived from WikiText-103, Chinese Wikipedia, CodeParrot-clean, and held-out COCO assets. The corpus itself is not included; the source manifests and corpus digest are included for provenance.

Natural router coverage was supplemented only for 46 expert/layer pairs that were not reached by the frozen calibration pass. This selective routing bypass was used to collect activation statistics; it did not exempt those expert weights from quantization. All 36,096 target modules were packed.

Precision and artifact audit

The final artifact audit reports:

36,096/36,096 target modules packed;
2,621 protected tensors unchanged byte-for-byte;
333 vision tensors and 785 MTP tensors retained;
10 weight shards fully SHA-256 hashed;
qweight, qzeros, and scales stored as INT32, INT32, and BF16;
simulated V100 BF16-to-FP16 loading of every AWQ scale found zero non-finite values, zero FP16 overflows, and zero nonzero values underflowing to zero.

The last point is a safety audit, not a claim that BF16-to-FP16 casting is mathematically lossless. The checkpoint itself was not globally converted to FP16.

See artifacts/final-output-audit.json, artifacts/quantization-report.json, quantization-provenance.json, and SHA256SUMS.

V100 TP4 performance

Validated profile:

4x Tesla V100 PCIe 32 GB, tensor parallel size 4;
no CPU weight offload;
FP16 KV cache, one active sequence;
greedy streaming generation, 256 output tokens, MTP disabled for the table;
local 1Cat-vLLM 1.2.2 SM70/TurboMind AWQ path.

Table with columns: Actual prompt tokens, Prefill/TTFT-derived tok/s, Decode tok/s
Actual prompt tokens	Prefill/TTFT-derived tok/s	Decode tok/s
8,051	3,482.74	58.01
64,052	1,817.16	49.00
128,053	1,166.45	41.77

These are single-request local engineering measurements, not standardized cross-model benchmark scores. The raw result JSON files are included under benchmarks/.

A separate 262,144-token engine profile with native MTP1 also loaded and ran successfully. The longest completed request in that acceptance pass contained 211,042 prompt tokens and completed without OOM or worker restart. This is not a claim that a near-full 262,144-token prompt was tested.

Local quality checks and known limits

The following local deterministic checks passed:

7/7 short agent/tool behavior cases without repetition or length-cap hits;
8/8 exact records plus the correct aggregate at 33,137 prompt tokens;
33/33 completed frozen NeedleBench/RULER retrieval cases around 127K-131K;
32/32 authoritative-vs-foil records at 123,457 prompt tokens;
structured tool calls and basic single-image inference.

Important failures and limits are retained rather than hidden:

at 120,369 prompt tokens, all 8 exact records were retrieved but the derived sum was wrong (173 instead of 137);
the strict multimodal fixture passed 5/6 cases; one arrow-route case failed;
multi-hop long-context reasoning was inconsistent and some thinking-enabled reruns reached their output cap;
no standardized paired coding/agent evaluation established quality non-inferiority to the BF16 mother or to a retained 27B full-precision model.

Accordingly, this checkpoint was retained locally as a secondary cold-standby, not promoted as the preferred general-purpose model. Parameter count alone must not be treated as proof that this quantization is better than a smaller full-precision model.

Download

bash
hf download leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ \
  --local-dir Qwen3.5-122B-A10B-Heretic-Selective-AWQ

Verify the download from inside the model directory:

bash
sha256sum -c SHA256SUMS

Runtime flags depend on the inference engine. Do not assume that a loader which supports dense AWQ also supports this selectively quantized Qwen3.5 MoE multimodal checkpoint on SM70.

License and use

The included model files are released under Apache-2.0, following the source repositories. Review the upstream model cards and license before use.

The Heretic base intentionally changes refusal behavior. This checkpoint is not safety-tuned, and its outputs may be inaccurate, offensive, or unsafe. Downstream users are responsible for evaluation, safeguards, and compliance in their own deployment context.

Acknowledgements

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

leoncca

Model Tree

Base

trohrbaugh/Qwen3.5-122B-A10B-heretic

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality