Quantization configuration
- Method: OmniQuant
- Weight precision: 2-bit
- Activation precision: 16-bit
- Group size: 128
- Optimization epochs: 40
- Learnable weight clipping (LWC): enabled
- Learnable equivalent transformation (LET): disabled
- Calibration dataset: C4 English validation split
- Calibration samples: 128
- Calibration sequence length: 512
- Calibration seed: 42
Evaluation
Table with columns: Dataset, Split, Sequence length, Perplexity| Dataset | Split | Sequence length | Perplexity |
|---|
| WikiText2 | test | 2048 | 671.2557 |
The evaluation used the full tokenized WikiText2 test corpus with non-overlapping
2048-token windows.
This follows OmniQuant's fake-quantized Hugging Face save path. It is not a
packed low-bit runtime checkpoint, so its storage and loading memory can remain
close to FP16 despite representing W2A16 quantized weights.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "yw223/Meta-Llama-3.1-8B-Instruct-OmniQuant-2bit"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
Transformers may report unused weight_quantizer.scales and
weight_quantizer.zeros entries when loading. The fake-quantized model weights
still load through the standard Transformers path used for the reported PPL.
License
Use of this checkpoint is subject to the license and terms of the base model.