⚠ Use the tokenizer from this repository
Always load the tokenizer from this repository, not from the base model. The base model's
tokenizer_config.json names Qwen2Tokenizer; with transformers older than 5.18 – including the version that vLLM
installs – that applies Qwen2's built-in split rule, which cuts Devanagari vowel signs (मात्राएँ) into separate
pieces. A Hindi prompt then becomes about 1.7× more tokens than the model saw in training and the model drops
letters and misspells words (e.g. निष्कर्ष- for निष्कर्ष :-, अब तक क अनुसंधान). The tokenizer files here name
Qwen3_5Tokenizer and tokenise exactly as in training under every transformers version from 5.17 on.
Quick check – this sentence must come out as 84 tokens (140 means the wrong tokenizer):
from transformers import AutoTokenizertok = AutoTokenizer.from_pretrained("Probabilistr/qwen3.5-4b-supervision-note-lora")s = ("यह कांड वादी परीक्षण कुमार के लिखित आवेदन के आधार पर अज्ञात चोर के विरूद्ध मोटरसाईकिल चोरी के आरोप में " "अंकित किया गया है। घटनास्थल का निरीक्षण, साक्षियों का बयान एवं पर्यवेक्षण से यह कांड सत्य प्रतीत होता है।")print(type(tok).__name__, len(tok(s, add_special_tokens=False)["input_ids"])) # Qwen3_5Tokenizer 84
Serving
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer REPO = "Probabilistr/qwen3.5-4b-supervision-note-lora"tok = AutoTokenizer.from_pretrained(REPO) # tokenizer from THIS repomodel = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, REPO).merge_and_unload().eval() messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": case_facts}]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)out = model.generate(**ids, max_new_tokens=4096, do_sample=False)print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))
Install flash-linear-attention and causal-conv1d (pip install --no-build-isolation causal-conv1d) for
Qwen 3.5's linear-attention layers; without them transformers falls back to a much slower implementation.
vLLM (several users at once)
Merge the adapter into a full checkpoint with merge_adapter.py from this repository. It folds the LoRA weights into
a copy of the base model and installs this repository's tokenizer, so the result can be served directly:
pip install torch safetensors huggingface_hubpython merge_adapter.py --adapter Probabilistr/qwen3.5-4b-supervision-note-lora --out ./supervision-4bVLLM_NO_USAGE_STATS=1 vllm serve ./supervision-4b --max-model-len 20480 \ --limit-mm-per-prompt '{"image": 0, "video": 0}'
Then confirm the server tokenises correctly – POST /tokenize with the sentence above must return 84 tokens.
If you serve some other copy of the weights, pass --tokenizer Probabilistr/qwen3.5-4b-supervision-note-lora.
For full control you can also tokenise on the client with this repo's tokenizer and send token ids as the
prompt of /v1/completions. Sent that way, vLLM's greedy output matched transformers character for character in
our tests; the tokenizer installed by merge_adapter.py produces the same ids inside vLLM.
(Serving the base model with vLLM's LoRA support was not tested.)
Loop guard
In about one case in five greedy decoding falls into a loop, writing the same direction again and again. Stop
generation once three earlier lines have been repeated and drop the repeated lines.
System prompt (used in training; use it unchanged):
आप झारखण्ड पुलिस के पर्यवेक्षण पदाधिकारी हैं। नीचे एक कांड की प्राथमिकी, घटनास्थल, साक्षियों के बयान एवं अनुसंधान की प्रगति दी गई है। इसके आधार पर पर्यवेक्षण टिप्पणी का निष्कर्ष, त्रुटि तथा अनुसंधानकर्ता को निर्देश लिखें।
User turn: the note's text up to (not including) the conclusion, in Unicode Hindi, one section per paragraph with its
heading, in the usual order – case line (police station, case number, date, sections of law), header fields
(place, distance and direction, dates, investigating officer), then प्राथमिकी :-, घटनास्थल :-,
अभियोजन साक्ष्य :-, साक्षी :- (one paragraph per witness), अनुसंधान में प्रगति :-, गिरफ्तारी :-,
समर्पण :-, तलाशी सह जप्ती :-, प्रतिरक्षा साक्ष्य :- (शून्य। where there is nothing).
Text typed in the legacy Kruti Dev font must be converted to Unicode first.
Output:
निष्कर्ष :- … त्रुटि :- … निर्देश :-1. …2. …
Training
Table | |
|---|
| Data | 4,127 supervision notes (≤ 16,384 tokens) from one district, 2020–2026; split by case |
| Method | LoRA r = 16, α = 32, dropout 0.05 on q/k/v/o and MLP projections; loss on the answer only |
| Schedule | 3 epochs, effective batch 8, learning rate 1e-4, cosine, 3% warm-up; bf16 |
| Validation loss | 0.225 → 0.204 → 0.201 (epochs 1–3) |
Evaluation (233 held-out cases)
Table with columns: Metric, Base Qwen 3.5 4B, This adapter, + loop guard| Metric | Base Qwen 3.5 4B | This adapter | + loop guard |
|---|
| Well-formed (all three parts) | 8% | 97% | 99% |
| chrF vs. officer's text | 43 | 78 | 80 |
| chrF, conclusion | 15 | 86 | 86 |
| Officer's directions covered | 0% | 58% | 58% |
| Same sections of law cited (F1) | 0.16 | 0.88 | 0.88 |
| Same finding (true / false / unproved) | 28% | 91% | 92% |
| Numbers/dates/vehicle nos. not found in the input | 19% | 0.8% | 0.8% |
| Ran to the token limit | 29% | 24% | 2% |
Text drawn from a different case scores chrF 39, 24% directions covered, F1 0.03 and 55% finding agreement – the
floor from boilerplate and chance.
Limitations
- Directions are the weak part: the model writes fewer directions than officers do, most of all on long, complex
cases; the loop guard removes repetition but does not add the missing directions.
- Trained on one district's notes; one office contributes about half the examples, so its style dominates.
- Inputs longer than about 16,000 tokens were not seen in training.
- The finding and the sections of law must always be checked against the case diary.