oshyshatskyi

ukr-council-extractor-qwen2.5-1.5b

README

License: apache-2.0

Fields

decision_no · date (ISO YYYY-MM-DD) · title · signatory · council_name

Results

Evaluated on 10 councils held out entirely from training (none of their documents were seen), so the number measures generalization to unseen councils, not memorization. The metric is strict macro all-fields-exact (afe): the fraction of documents where all five fields match the gold exactly (text fields lower-cased + punctuation-folded, then exact; decision_no space-insensitive exact; date ISO exact), averaged per council so large councils can't dominate.

Table with columns: all-fields-exact, decision_no, date, title, signatory, council_name
	all-fields-exact	decision_no	date	title	signatory	council_name
this adapter (seed 1)	0.903	0.984	0.997	0.981	0.952	0.987
base, zero-shot	0.066	—	—	—	—	—

Across-seed quality (3 training seeds, full 41-council pool): afe 0.889 ± 0.016. The single-number table above is this published seed-1 checkpoint.
JSON validity: 100.0%.
Distillation generalizes: the base model scores 0.066 zero-shot with the identical prompt; the adapter lifts it to 0.903 on councils it never trained on.

Usage

python
import json, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "Qwen/Qwen2.5-1.5B-Instruct"
ADAPTER = "oshyshatskyi/ukr-council-extractor-qwen2.5-1.5b"

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER).eval()

SYSTEM = "Ти витягуєш дані з шапки та підпису РІШЕННЯ місцевої ради України. Поверни СТРОГО JSON з рівно такими полями: decision_no, date, title, signatory, council_name. Використовуй null, якщо поле відсутнє у тексті.\n- decision_no: номер рішення БЕЗ префікса «№» і без слова «№». Напр. «№ 4227» → «4227»; «№62-2275/2025» → «62-2275/2025». Зберігай складений формат (дефіси, косі риски, роки). null, якщо у шапці немає номера.\n- date: дата ухвалення у форматі ISO РРРР-ММ-ДД. Перетвори з української: «23 грудня 2025 року» → «2025-12-23»; «24.12.2025р.» → «2025-12-24».\n- title: заголовок «Про …» (повністю, без номера).\n- signatory: ПІБ особи, що підписала (зазвичай голова/секретар ради), зазвичай в кінці. null, якщо не видно.\n- council_name: повна назва ради з шапки."

def head_tail(text, head=2200, tail=800):
    # feed the model the same header+signature window it trained on; elide the middle
    return text if len(text) <= head + tail else text[:head] + "\n…\n" + text[-tail:]

def extract(document_text):
    msgs = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": head_tail(document_text)}]
    prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    ids = tok(prompt, return_tensors="pt").to(model.device)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False)
    return json.loads(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

# print(extract(open("some_decision.txt", encoding="utf-8").read()))

Training

Method: QLoRA — 4-bit NF4 base (double-quant, bf16 compute) + LoRA (r=16, α=32, dropout=0.05) on all attention and MLP projections.
Optimizer: lr 2e-4, 3 epochs, input window 2560 tokens.
Data: ~1.6k teacher-labelled decision spans pooled from 41 training councils (10 councils fully held out for evaluation). One prompt for teacher, base, and student, so the comparison is fair.
Hardware: single H100 (Modal); a training run costs well under $2.

Limitations

Ukrainian, decision headers only. Built for the рішення header/signature block; not for appendices, voting protocols, or other document types, and not for other languages.
Strict-exact metric. The ~0.88 afe is conjunctive over five fields; individual fields are 0.95–0.99. The remaining errors are dominated by signatory and by verbatim title/council_name phrasing.
Prompt-coupled. Use the exact system prompt and head+tail window above; other prompts degrade it.
Provenance: part of a cross-council generalization study (frozen 51-council dataset). Each field's score above is read from this adapter's own eval artifact.

License

Adapter weights: apache-2.0. You must also comply with the base model's license.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

oshyshatskyi

Model Tree

Base

Qwen/Qwen2.5-1.5B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities