Quantization
- llm-compressor AWQ, scheme
W4A16_ASYM, lm_head kept in bf16 (see recipe.yaml)
- calibration: 256 samples × 512 tokens, drawn from 512 long passages of one Budapest district's official sources and climate strategy. Using text that looks like what the model will actually read keeps the important weights in the right place. (512 × 1024 ran out of memory on a 24 GB A10G.)
- ~2.7 GB of weights. It fits on an 8 GB GPU with room for the KV cache.
- scripts:
scripts/ml/quantize/
These numbers are for the full Paper Trail extractor: this model plus the receipt check, filters and dedupe. The gold set gold-v2 has 834 commitments from 9 real climate strategies: 7 Hungarian (Budapest districts and Pécs) and 2 English (Leicester, Kensington and Chelsea). Scores are macro-averaged over the 6 holdout documents, which were never used for prompt work.
Table with columns: Recall, Precision, F1| Recall | Precision | F1 |
|---|
| 0.866 | 0.929 | 0.896 |
Caveats: 159 of the 834 gold rows were pooled from model answers that people later confirmed (TREC-style pooling), which flatters recall a little. Precision is a floor, because the key may still be missing real commitments. Nine documents is a small test. Details are in the Models wiki page.
Serving
Any OpenAI-compatible server that reads compressed-tensors checkpoints (vLLM, SGLang) can serve it. Paper Trail's settings:
- temperature 0,
max_tokens 2048, strict JSON schema (response_format)
- turn off thinking:
chat_template_kwargs: {"enable_thinking": false}. Otherwise the model spends its whole budget reasoning and never answers.
- context 12288, which leaves about 6k tokens of KV cache on 8 GB. That's enough for 3–4 pages in flight at once.
vllm serve unnobatroo/paper-trail-qwen3-4b-instruct-hu-awq --max-model-len 12288
Limits
- Output must be grounded, so don't trust it without a check. The model sometimes paraphrases or invents section codes, and Paper Trail strips those.
- Calibrated on Hungarian municipal text. It works in English, but it was tuned for neither language.
- Same license as the base model: Apache-2.0.