n0rths7ar

bioevidence-qwen3-4b-extraction-lora-v1

README

License: apache-2.0

Intended use

Input consists of a biomedical query, a PubMed title, an abstract, and the v1 JSON Schema. Output is exactly one JSON object containing:

query-specific evidence status;
study design;
population, intervention or exposure, and comparator when supported;
outcome direction and result text;
verbatim evidence spans copied from the abstract;
a short evidence summary.

Use the adapter for reproducible portfolio experiments, offline extraction evaluation, and optional enrichment inside BioEvidence Copilot. Validate every response against the schema and verify evidence spans against the source text.

Do not use it for diagnosis, treatment selection, clinical decision-making, or unsupported claims beyond the supplied title and abstract.

Base model

Base: unsloth/Qwen3-4B-Instruct-2507-unsloth-bnb-4bit
Pinned training snapshot: 7744afa8566e264af1a92a806d8d9aae00cc7c78
Adapter format: PEFT LoRA
Base-model license: Apache-2.0

Training data

The v1 experiment contains 60 query-document annotations built from PubMed titles and abstracts:

Table with columns: Label, Rows
Label	Rows
direct	8
indirect	29
none	23

Labels are model-assisted drafts. The 20-row pilot received a second-model review; 40 expansion rows were produced in Claude Desktop and passed JSON, Pydantic schema, duplicate-pair, and exact-span validation. Splits are assigned by PMID to prevent the same document from crossing train, dev, and test.

Table with columns: Split, Rows, Unique PMIDs
Split	Rows	Unique PMIDs
train	46	43
dev	7	5
test	7	5

Dataset provenance and split IDs are tracked in the project repository. The adapter package does not redistribute the training dataset.

Training procedure

GPU: NVIDIA RTX 5070 12 GB
Python: 3.12.13
Quantization: 4-bit base weights
Compute: BF16
LoRA rank / alpha: 16 / 16
Target modules: attention and MLP projections
Trainable parameters: 33,030,144 of 4,055,498,240 (0.81%)
Effective batch size: 4
Optimizer: 8-bit AdamW
Learning rate: 2e-4 with linear decay
Steps / epochs: 36 / 3
Maximum sequence length: 4,096
Loss: response-only; the supervised response begins with {
Training time: 238.3 seconds
Peak PyTorch allocated VRAM: 5.81 GiB
Dev loss: 0.971 before training, 0.244 after training

Held-out evaluation

All systems were evaluated on the same seven PMID-held-out draft annotations with deterministic decoding.

Table with columns: Metric, Rules, Prompted base, QLoRA adapter
Metric	Rules	Prompted base	QLoRA adapter
Strict JSON parse rate	1.000	1.000	1.000
Schema validity	1.000	1.000	1.000
Grounding rate	n/a	0.714	1.000
Evidence-status accuracy	0.429	0.714	0.429

These results demonstrate strict JSON specialization and a working local training/inference path. Seven draft test rows are too few for a general claim about biomedical extraction quality, and evidence-status classification did not improve over the prompted base model.

Known limitations

All 60 labels are draft, model-assisted annotations rather than expert gold.
The held-out set contains only seven rows.
Direct labels are absent for the asthma and melanoma query groups.
Training and evaluation use titles and abstracts, not full text.
Evidence-span grounding checks copying, not whether a selected span is the best scientific interpretation.
The adapter can misclassify unrelated evidence as indirect.
Results are specific to the pinned base model, schema, prompt, and split.

Reproduction and inference

The repository contains the exact environment lock, schema, annotations, split manifest, training script, evaluator, and aggregate report. After placing the adapter in a local directory:

powershell
conda activate bioevidence-training
python scripts/run_extraction_eval.py `
  --backend local `
  --adapter-path path/to/adapter `
  --dataset artifacts/training/evidence_extraction/training_v1_sft/test.annotations.jsonl

For optional product inference, set EXTRACTION_BACKEND=local and EXTRACTION_ADAPTER_PATH=path/to/adapter. BioEvidence Copilot validates schema and verbatim spans and falls back to its deterministic structured extractor when the optional local runtime is unavailable or invalid.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

n0rths7ar

Model Tree

Base

unsloth/Qwen3-4B-Instruct-2507-unsloth-bnb-4bit

Adapter

this model

Input Modalities

Text

Output Modalities