Validation
Saved-checkpoint reload and generation smoke passed with the GPTQ-Pro native backend on an RTX 3090. All packed projections, finite scales, exact protected source tensors and tokenizer/template preservation were checked. English, Spanish and Chinese smoke outputs are in VALIDATION.json.
A disjoint 24-conversation public heldout smoke, using the first 512 tokens per conversation, produced NLL 1.479804 nats/token and perplexity 4.392086. No original-BF16 quality comparison was performed.
Load
Use the GPTQ-Pro fork, not an assumed stock GPTQ runtime. Tested with PyTorch 2.11.0+cu130 and Transformers 5.19.0.
import torchfrom gptqmodel import GPTQModel, BACKENDfrom transformers import AutoTokenizer, AutoProcessor repo = "groxaxo/Qwopus3.5-9B-Coder-Flash-Heretic-GPTQPro-W4-G64"model = GPTQModel.load(repo, backend=BACKEND.GPTQ_PRO, device="cuda:0", dtype=torch.bfloat16)tokenizer = AutoTokenizer.from_pretrained(repo)processor = AutoProcessor.from_pretrained(repo)text = tokenizer.apply_chat_template([{"role": "user", "content": "Hello!"}], tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, add_special_tokens=False, return_tensors="pt").to("cuda:0")output = model.generate(**inputs, max_new_tokens=256, do_sample=False, use_cache=True)print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
The upstream checkpoint is an ARA abliteration of Jackrong/Qwopus3-5-9B-Coder-Flash (Apache-2.0). Abliteration and quantization do not independently establish the upstream publisher's claims. SHA256SUMS covers the published model files.
Deployment: vLLM with native MTP (verified on 1× RTX 3090)
vLLM ≥ 0.24 loads this checkpoint's MTP head natively — no separate draft model is needed:
vllm serve groxaxo/Qwopus3.5-9B-Coder-Flash-Heretic-GPTQPro-W4-G64 \ --served-model-name qwopus-heretic \ --max-model-len 120 --max-num-seqs 8 \ --gpu-memory-utilization 0.85 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --port 8011
Measured on one RTX 3090 (24 GB), 80-token greedy completions, warm runs:
Table with columns: Speculative config, 8 concurrent, Single stream, Draft acceptance| Speculative config | 8 concurrent | Single stream | Draft acceptance |
|---|
| MTP, 3 draft tokens | ~835 tok/s | ~140 tok/s | 73–78% |
| MTP, 1 draft token | ~730–835 tok/s | ~134 tok/s | 93% |
| No speculative decoding | ~735 tok/s | ~104 tok/s | — |
- Startup confirms:
Detected MTP model. Sharing target model embedding/lm_head weights with the draft model.
- VRAM 19.1/24 GB at
--gpu-memory-utilization 0.85; GPU KV cache 4,560 tokens (~4.7× the 960 tokens needed for 8 concurrent × 120 ctx).
- The BF16 checkpoint (groxaxo/Qwopus3.5-9B-Coder-Flash-Heretic) also runs with MTP but is ~3.5× slower at concurrency 8 (~242 tok/s).