Quantization recipe
- GPTQ-Pro / GPTQModel
- 4 bits, group size 128, symmetric
desc_act=false, true sequential
- FOEM alpha
0.25, beta 0.2
- MSE
2.0
- damp percent
0.05, auto increment 0.01
- 256 calibration samples at sequence length 2048
- 288,482 accepted calibration tokens
- CPU packing
- Dense layers assigned exclusively to one NVIDIA A100 80GB
- Vision, MTP, LM head, embeddings, and normalization tensors preserved unquantized
The calibration artifact is identified by SHA-256
5d8d9a2fe6ecc47dc937358784b5698564d3b217deaf13dab418c7c2fb06eaaa.
Validation contract
The upload workflow fails closed unless all of the following pass:
- durable checkpoint coverage through all 64 text layers;
- exactly 400 expected quantized linear modules;
- strict FOEM audit with no zero-sample, sample-mismatch, fallback, failed, duplicate,
or non-finite records;
- complete safetensor index and shard inventory;
- preserved vision, MTP, LM-head, embedding, and norm tensors;
- post-save reload and generation on GPU;
- complete private Hugging Face upload verified through the Hub API.
MTP and vision status
The source tensors are preserved so the artifact is complete, but preservation alone
does not establish MTP speculative-decoding or vision runtime compatibility. Treat
those paths as experimental until separately validated with the target serving stack.
The source model card reports faster decoding and high MTP acceptance for its own
GGUF/llama.cpp evaluation. Those figures are not claimed for this GPTQ-Pro artifact.
Publish measured GPTQ/vLLM results only after a controlled run on the target hardware.
Credits