WikiText-2 (test, 2048-token windows, 65504 tokens)
Measured on NVIDIA GeForce RTX 3090 (vast-rtx3090) against the original fp16 snapshot.
Numbers copied from jobs/20261107T033248Z-1b1409/benchmark.json.
Table with columns: Quantized, fp16 snapshot | Quantized | fp16 snapshot |
|---|
| Perplexity | 11.121597 | 9.657566 |
| NLL loss | 2.408889 | 2.267742 |
| Tokens/s (2048 prefill, after warmup) | 6588.5 | 3493.5 |
| Peak VRAM (GB) | 11.998 | 18.462 |
ppl_ratio: 1.151594 (quality_ok=True)
improved_throughput: True
improved_vram: True
Pull
from huggingface_hub import snapshot_downloadpath = snapshot_download("chris320211/prefixquant-qwen3-8b-vastrtx3090")
This is not a drop-in AutoModelForCausalLM.from_pretrained checkpoint.
Reload with the included quant_agent_inference_adapter.py after the method
repo (and overlay, if any) is on QUANT_AGENT_METHOD_REPO. See
quantization_config.json.
License
Inherits the license of Qwen/Qwen3-8B
(apache-2.0), plus the method repository license. Keep LICENSE /
NOTICE.md from the snapshot when present.