Weights are divided into groups of 128 elements. Each group gets its own scale and zero-point:
4-bit weights are packed into int32 (8 values per int32). 8-bit weights stored as uint8.
-
Frantar, E., et al. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." ICLR 2023. — Foundation for group-wise weight quantization with second-order error compensation.
-
Lin, J., et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." MLSys 2024. — Activation-aware weight scaling to protect salient channels. Inspired the sensitivity analysis approach.
-
Ashkboos, S., et al. "QuaRot: Outlier-free 4-bit Quantization of Large Language Models." arXiv 2403.02747. — Hadamard-based rotation to remove outlier features. Explored but skipped due to hook conflicts.
-
Dettmers, T., et al. "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale." NeurIPS 2022. — Mixed-precision decomposition for 8-bit quantization. Basis for 8-bit attention/embedding treatment.
-
Xiao, G., et al. "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models." ICML 2023. — Activation smoothing for outlier mitigation. Informed the group quantization strategy.
-
Chee, J., et al. "QuIP: 2-Bit Quantization of Large Language Models with Incoherence Processing." arXiv 2307.13304. — Incoherence processing and vector quantization. Theoretical basis for rotation-based approaches.
-
Note: This model uses a custom quantization format (custom_affine_group). The quantized weights are stored as packed int32/uint8 with separate scales and zero-points. Dequantization happens on-the-fly during forward pass via QuantizedLinear and QuantizedMoeExperts modules.