illuminator22

openbrief-nano-qwen35-entity-extractor

README

License: apache-2.0

What It Does

Given a chunk of legal contract text, OpenBrief Nano returns structured JSON with 8 entity types:

party — named legal entities, organizations, persons
date — agreement, effective, expiration dates (with subtypes)
deadline — notice periods, renewal terms, payment windows (with subtypes)
jurisdiction — venue, forum, arbitration seat
obligation — duties imposed on a party
monetary_amount — dollar values, liability caps, liquidated damages (with subtypes)
governing_law — applicable law
termination_right — rights to terminate (with subtypes)

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch, json

# Load base model + adapter
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-9B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, "illuminator22/openbrief-nano-qwen35-entity-extractor")
model.eval()

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-9B", trust_remote_code=True)

# Build prompt
chunk = "Your legal text here..."
prompt = (
    "<|im_start|>system\n"
    "You are OpenBrief Nano, a legal entity extraction model. "
    "Return only valid JSON. Do not include explanations, commentary, or hidden reasoning.<|im_end|>\n"
    "<|im_start|>user\n"
    "Extract entities from this legal text:\n\n"
    f"{chunk}<|im_end|>\n"
    "<|im_start|>assistant\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=False,
                              eos_token_id=tokenizer.eos_token_id,
                              pad_token_id=tokenizer.pad_token_id)

gen_tokens = outputs[0][inputs["input_ids"].shape[1]:]
result = json.loads(tokenizer.decode(gen_tokens, skip_special_tokens=True))
print(json.dumps(result, indent=2))

Evaluation: OpenBrief Nano vs GPT-5.4 Nano

Test Setup

50 chunks from 51 held-out CUAD contracts (contract-level split, no data leakage)
25 chunks manually reviewed with gold labels for per-type accuracy
Both models evaluated on identical chunks

Overall Metrics

Table with columns: Metric, OpenBrief Nano, GPT-5.4 Nano (prompted)
Metric	OpenBrief Nano	GPT-5.4 Nano (prompted)
Valid JSON rate	100%	100%
Total entities extracted (45 comparable chunks)	407	386

Per-Type Extraction Volume (45 comparable chunks)

Table with columns: Entity Type, OpenBrief Nano, GPT-5.4 Nano, Winner
Entity Type	OpenBrief Nano	GPT-5.4 Nano	Winner
party	196	146	Nano
date	67	57	Nano
deadline	30	13	Nano
jurisdiction	1	13	GPT-Nano

Per-Type Accuracy (25 gold-reviewed chunks)

Based on manual expert review of 25 held-out chunks, comparing each model's output against human-verified gold labels:

Table with columns: Entity Type, Nano Wins, GPT-Nano Wins, Tie, Assessment
Entity Type	Nano Wins	GPT-Nano Wins	Tie	Assessment
party	12	4	9	Nano stronger — finds more real parties, occasionally over-extracts
date	5	3	17	Nano slightly better — sometimes over-extracts temporal references
deadline	7	1	17	— catches notice periods and payment windows GPT misses

Key Findings

OpenBrief Nano wins on: party extraction (12-4), deadline detection (7-1), termination rights (5-2). These are high-frequency, CUAD-anchored entity types where training data was strongest.

GPT-5.4 Nano wins on: jurisdiction (0-5), obligation volume (2-6). Jurisdiction was fully GPT-assisted during training with limited examples. Obligation extraction finds fewer entities but with better party attribution.

Both models struggle with: jurisdiction in general (sparse in contracts), monetary amounts (often redacted in CUAD source data).

Qualitative observations:

Nano produces better party attribution in obligations (uses defined party names)
GPT-Nano occasionally extracts signature blocks as obligations and documents as parties
Nano occasionally over-extracts section titles and generic temporal references as entities
Both models correctly return empty arrays when no entities are present

Cost Comparison

Table with columns: Model, Cost per chunk, Monthly cost (1000 docs), Self-hosted
Model	Cost per chunk	Monthly cost (1000 docs)	Self-hosted
OpenBrief Nano	$0.00	$0.00	Yes
GPT-5.4 Nano	~$0.002	~$40-60	No
GPT-5.4	~$0.02	~$400-600	No

Training Details

Base model: Qwen/Qwen3.5-9B
Method: LoRA (bf16, not QLoRA — Qwen3.5 architecture has higher quantization loss at 4-bit)
Training data: 708 contract chunks from 407 CUAD contracts
Labeling: GPT-5.4 with CUAD expert annotations as anchors (hybrid silver labeling)
LoRA config: rank=32, alpha=64, dropout=0.05
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Epochs: 3
Learning rate: 2e-4 (cosine schedule)
Final training loss: 0.62
Hardware: NVIDIA A100-SXM4-80GB (Google Colab)
Trainable parameters: 58M / 9B (0.65%)

Training Data Pipeline

510 CUAD commercial contracts split at contract level (407 train / 51 val / 51 test)
Contracts chunked into 512-token windows with 50-token overlap
CUAD expert annotations matched to chunks as anchors for 6 entity types (party, date, governing_law, termination_right, deadline, monetary_amount)
GPT-5.4 used for schema conversion, gap-filling, and labeling 2 additional types (obligation, jurisdiction)
Post-processing cleanup to remove generic party references and noisy labels
Balanced sampling: 558 anchor-rich chunks + 150 no-anchor chunks = 708 training examples

Label Reliability Tiers

Table with columns: Tier, Entity Types, Label Source, Quality
Tier	Entity Types	Label Source	Quality
Tier 1 (high confidence)	party, date, governing_law	CUAD expert annotations + GPT formatting	High
Tier 2 (anchor-partial)	deadline, termination_right, monetary_amount	CUAD clause anchors + GPT extraction	Medium-High
Tier 3 (GPT-assisted)	obligation, jurisdiction	GPT-5.4 only	Medium

Limitations

Jurisdiction extraction is weak (0 wins vs GPT-Nano in gold review) — limited training signal
Obligation extraction lags GPT — fewer examples found, though party attribution is often better
English only — trained exclusively on English commercial contracts from CUAD
Contract-focused — not trained on court opinions, legislation, or other legal document types
Silver labels — training data labeled by GPT-5.4 with CUAD anchors. Model inherits systematic biases from the labeling model
Occasional over-extraction — may extract section titles, generic temporal references, or product names as entities
Recommended max_new_tokens: 1024 — use 1024 or higher for entity-dense chunks to avoid output truncation

Known Issues and Future Improvements

Add more jurisdiction training examples from contracts with explicit forum selection clauses
Add court case chunks to expand beyond commercial contracts
Post-processing regex to strip think blocks as safety net (model was trained without thinking mode)

Project

OpenBrief Nano is part of OpenBrief, an open-source legal document intelligence platform featuring agentic RAG, persistent memory, multi-provider LLM support, and transparent evaluation metrics.

Author: Ivan Arshakyan / BrainX Corp

Citation

markdown
@misc{openbrief-nano-2026,
  title={OpenBrief Nano: LoRA Fine-Tuned Legal Entity Extraction on Qwen3.5-9B},
  author={Ivan Arshakyan},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/illuminator22/openbrief-nano-qwen35-entity-extractor}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

illuminator22

Model Tree

Base

Qwen/Qwen3.5-9B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality