Quantization
- Method: AWQ
- Weight format: INT4
- Compute scheme: W4A16
- Calibration samples: 128
- Max calibration length: 4096 tokens
- Calibration style: document QA, OCR-style text, structured JSON output, coding prompts, table and chart descriptions, transcript-style prompts, and long-context retrieval prompts
- Tooling: LLM Compressor 0.12.0, Transformers 5.10.1, PyTorch 2.11.0+cu128
- GPU used: NVIDIA A100-SXM4-80GB
Local Check
A short generation check was run after saving the weights.
- Test prompt: return a JSON object with
status: ok
- Result: passed
- Peak VRAM during local load test: 22.98 GB
Notes
These weights are intended for practical inference tests with lower memory use than the original precision. Quantization can affect exact wording, numeric precision, and long-context recall. Full 256K context behavior should only be assumed after testing in your own setup.