ashish24142

nightwing-14b-cuad-extractive

README

License: apache-2.0

Honest scoreboard (same harness, same metric, same test split)

Table with columns: Model, AUPR
Model	AUPR
claude-opus-4-8	0.561
claude-opus-4-6	0.498
gpt-5.2	0.423
gpt-4o	0.421
nightwing-v2-14b (this adapter)	0.389
nightwing-v1-14b (generative, same base/data/budget)	0.291

Beats gpt-4o in 25/40 categories and gpt-5.2 in 22/40; beats all four frontier models simultaneously in 6 (e.g. Agreement Date 0.829 vs best frontier 0.168).
The v1 -> v2 delta (+9.8 pts) comes from task framing alone: generation -> extraction, same base model, same data, ~$32 of A100 time, one epoch.
A pre-registered prediction that this run clears 0.50 missed; the deficit is the deliberately lean recipe (1 epoch, 2:1 negative subsampling, no tuning). Details, scaling curve (0.5B-14B), and every raw prediction file: github.com/ashish24142/nightwing.

IMPORTANT: loading correctly

Qwen3ForQuestionAnswering names its backbone transformer.* while the base checkpoint stores model.*, so a vanilla AutoModelForQuestionAnswering.from_pretrained("Qwen/Qwen3-14B") silently initialises a random backbone and this adapter will produce garbage. Load the backbone explicitly (or use harness/extractive.py::load_qa_model from the repo, which asserts it):

python
import torch
from transformers import AutoModel, AutoModelForQuestionAnswering, AutoTokenizer
from peft import PeftModel

BASE = "Qwen/Qwen3-14B"
qa = AutoModelForQuestionAnswering.from_pretrained(BASE, dtype=torch.bfloat16)
src = AutoModel.from_pretrained(BASE, dtype=torch.bfloat16)
missing, _ = qa.transformer.load_state_dict(src.state_dict(), strict=False)
assert not [m for m in missing if "embed" in m or "layers" in m]
del src

model = PeftModel.from_pretrained(qa, "ashish24142/nightwing-14b-cuad-extractive")
tok = AutoTokenizer.from_pretrained("ashish24142/nightwing-14b-cuad-extractive")
model.eval().to("cuda")

Inference over long contracts uses 512-token sliding windows (stride 128) with truncation="only_second"; span confidence for PR-curve scoring is sigmoid(best_span_score - min_null_score). The exact windowing, postprocess and official-metric scoring code is in the repo (harness/extractive.py, harness/eval_extractive.py), using it is the recommended path and reproduces 0.3885 exactly.

Training

LoRA r16, alpha 32, dropout 0.05, all attention+MLP projections; randomly initialised qa_outputs head trained via modules_to_save.
bf16, effective batch 16, lr 2e-4 cosine, 2000 steps (~1 epoch), gradient checkpointing; single A100 80GB, ~2.6 h.
Data: CUAD train split only, 368 contracts (40-contract dev carve-out excluded), zero train/test contamination asserted in code at every launch.
Trained only on CUAD ground truth, never on frontier model outputs.

Intended use & limitations

Research benchmark artifact for legal-NLP extraction. Not a lawyer, not legal advice, not production-ready: overall accuracy trails frontier models by 17 pts, recall at high thresholds is weak (p@80recall = 0), and the head predicts one span per window while some CUAD questions have multiple gold spans.

Citation

CUAD: Hendrycks et al., 2021 (arXiv:2103.06268), CC BY 4.0, The Atticus Project. Base model: Qwen3-14B (Apache-2.0).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ashish24142

Model Tree

Base

Qwen/Qwen3-14B

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer