p4ulbr4dl3y

qwen3-0.6b-contextual-sft-v2

README

License: apache-2.0

Results

Heuristic word-F1 against the teacher reference, same 60 held-out validation rows before and after SFT:

Table with columns: metric, base Qwen3-0.6B (4-bit), + LoRA SFT
metric	base Qwen3-0.6B (4-bit)	+ LoRA SFT
word-F1 mean	0.1893	0.5373 (2.8x)
word-F1 median	0.1764	0.534
word-F1 p10	0.0639	0.3188
zero-F1 share	0.017	0.000
hallucinated numbers	0	0

Per-category word-F1 (base -> tuned):

construction: 0.097 -> 0.529
food: 0.143 -> 0.484
it: 0.232 -> 0.574
pharma: 0.137 -> 0.445
security: 0.079 -> 0.54

word-F1 above 0.10 for 60 of 60 rows (baseline: 47 of 60)

Training

base model: Qwen/Qwen3-0.6B, 4-bit (unsloth), LoRA r=16 alpha=16 dropout=0, 10.09M trainable params (1.665%)
dataset: p4ulbr4dl3y/ru-contextual-chunks-v2 - 2701 train / 433 val rows, split by window_hash (no near-duplicate leakage)
response-only masking, max_seq_length 16384, batch 2 x grad-accum 4 = 8, lr 2e-4 (linear, warmup 5), 2 epochs = 676 steps
~9.5 h of GPU time on a single free Tesla T4 (fp16), ~50 s/step including periodic validation, spread over 4 resume-safe sessions (3 Colab spot-quota kills; every checkpoint is mirrored to a private HF repo); best checkpoint by eval_loss = step 640
eval_loss:
- 40: 0.75, 80: 0.6554, 120: 0.5923, 160: 0.573, 200: 0.5414, 240: 0.5262, 280: 0.5163, 320: 0.5119, 360: 0.5231, 400: 0.5151, 440: 0.5075, 480: 0.4908, 520: 0.4882, 560: 0.4824, 600: 0.4769, 640: 0.4761

Run log and per-row comparison are in this repo (log_history.json, per_row_comparison.jsonl).

Prompt format

Both training and inference use the same prompt (Qwen3 chat template with enable_thinking=False):

text

<document>{document}</document>Вот фрагмент, который нужно соотнести со всем документом:<chunk>{chunk}</chunk>Сформулируй краткий контекст (1-2 предложения), чтобы привязать этот фрагмент к документу в целом для улучшения поисковой выдачи. Ответь только кратким контекстом и ничем больше.

Usage

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel BASE = "Qwen/Qwen3-0.6B"REPO = "p4ulbr4dl3y/qwen3-0.6b-contextual-sft-v2"tok = AutoTokenizer.from_pretrained(REPO)model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.float16, device_map="cuda"), REPO).eval() msgs = [{'role': 'user', 'content': TEMPLATE.format(document=document, chunk=chunk)}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to(model.device)out = model.generate(ids, max_new_tokens=300, do_sample=False)print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

GGUF / llama.cpp

The merged checkpoint is also exported to GGUF, so the model runs in llama.cpp, Ollama or LM Studio on any machine, not only through MLX on Apple Silicon. Quantization ablation on the same 60 held-out validation rows (word-F1 against the teacher reference, as above; MLX bf16 reference on those rows is 0.537):

Table with columns: quant, size, word-F1, vs f16 (95% CI)
quant	size	word-F1	vs f16 (95% CI)
f16	1143 MB	0.526	-
Q8_0	610 MB	0.520	-0.007 [-0.021, +0.008]
Q6_K	472 MB	0.502	-0.024 [-0.053, +0.002]
Q5_K_M	424 MB	0.500	-0.027 [-0.062, +0.006]

Q8_0 is the recommended default: its difference from f16 is not significant, and no quantization produced an empty response or a number absent from the document. Only Q8_0 is uploaded here. Q6_K is a fair pick under a tight size budget; Q4_K_M loses quality measurably on a 0.6B model. Intervals are a 95% paired bootstrap over the 60 rows.

Prompt assembly matters: the model expects the training prompt wrapped in the Qwen3 chat template with enable_thinking=False. The exact rendering used for the numbers above lives in experiments/gguf/eval_gguf.py of the ocrpipe repo, and the model is invoked in raw completion mode:

bash

llama-cli -m qwen3-0.6b-contextual-sft-v2-Q8_0.gguf -ngl 99 -c 16384 -p "<chat-templated prompt>"

Limitations

0.6B student: the context is a retrieval aid, not a summary; expect occasional invented contract numbers (2/60 rows here).
Russian-language, contract/procurement/technical domain.
word-F1 is measured against a teacher reference, not against end-to-end retrieval recall.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

p4ulbr4dl3y

Model Tree

Base

Qwen/Qwen3-0.6B

Adapter

this model

Input Modalities

Text

Output Modalities