WikiText-2 (test, 2048-token windows, 65504 tokens)
Measured on NVIDIA GeForce RTX 3090 (vast-rtx3090) against the original fp16 snapshot.
Numbers copied from jobs/20261107T062044Z-8b5fee/benchmark.json.
Table with columns: Quantized, fp16 snapshot | Quantized | fp16 snapshot |
|---|
| Perplexity | 10.060141 | 9.657566 |
| NLL loss | 2.308581 | 2.267742 |
| Tokens/s (2048 prefill, after warmup) | 3352.2 | 3450.9 |
| Peak VRAM (GB) | 8.891 | 18.462 |
ppl_ratio: 1.041685 (quality_ok=True)
improved_throughput: False
improved_vram: True
Pull
from huggingface_hub import snapshot_downloadpath = snapshot_download("chris320211/autoround-qwen3-8b-vastrtx3090")
This is not a drop-in AutoModelForCausalLM.from_pretrained checkpoint.
Reload with the included quant_agent_inference_adapter.py after the method
repo (and overlay, if any) is on QUANT_AGENT_METHOD_REPO. See
quantization_config.json.
Serve with vLLM
A native vLLM build of these weights: chris320211/autoround-qwen3-8b-vastrtx3090-vllm.
vllm serve chris320211/autoround-qwen3-8b-vastrtx3090-vllm
vLLM PPL ratio 1.0417 against vLLM fp16; prefill
0.97×, decode 2.64× (batch 1), 2.19× (batch 16). Measured on
NVIDIA GeForce RTX 3090 with vLLM 0.30.0.
License
Inherits the license of Qwen/Qwen3-8B
(apache-2.0), plus the method repository license. Keep LICENSE /
NOTICE.md from the snapshot when present.