OmkarShewale

clintrial-qwen2.5-7b-sft

README

License: apache-2.0

What it does

Table with columns: Task, Description
Task	Description
Eligibility extraction	Free-text inclusion/exclusion criteria → structured JSON
Plain-language summary	Trial description → patient-friendly summary
Condition Q&A	"What conditions does this trial study?"
Phase classification	Identify the trial phase

Results

Evaluated on a held-out test set split by trial ID (no trial appears in both train and test), 500 sampled examples.

Table with columns: Task, Metric, Base Qwen2.5-7B, This model (SFT)
Task	Metric	Base Qwen2.5-7B	This model (SFT)
Eligibility extraction	criterion F1	0.840	0.968
Eligibility output	JSON validity	0.986	1.000
Phase classification	exact match	0.000	0.794
Condition Q&A	token F1	0.298	0.742
Plain-language summary	ROUGE-L	0.195	0.290

Eligibility extraction reaches precision 0.962 / recall 0.978 with 100% schema-valid JSON.

The base model scores 0.000 on phase classification not because it lacks the knowledge, but because it won't answer in the required format — fine-tuning buys format discipline and faithfulness.

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "OmkarShewale/clintrial-qwen2.5-7b-sft"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16, device_map="auto")

SYSTEM = (
    "You are a clinical research assistant. You help patients and clinicians understand "
    "clinical trials. Answer only from the information provided, be precise, and never "
    "invent eligibility criteria, conditions, or outcomes."
)

criteria = """Inclusion Criteria:
1. Adults aged 18 years or older with confirmed type 2 diabetes
2. HbA1c between 7.0% and 10.5% at screening

Exclusion Criteria:
1. History of severe hypoglycaemia within the last 6 months
2. Pregnancy or breastfeeding
"""

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content":
        'Extract the eligibility criteria from the trial text below into JSON with two lists, '
        '"inclusion" and "exclusion". Copy each criterion verbatim; do not add any.\n\n'
        f"Trial eligibility text:\n{criteria}"},
]

enc = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
                                    return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**enc, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Note: eligibility answers can be long (up to ~1.7k tokens). Use max_new_tokens >= 1024 or the JSON will be truncated and fail to parse.

Training

Table

Base model	Qwen2.5-7B-Instruct
Method	QLoRA — 4-bit NF4 + LoRA (r=16, α=32, dropout 0.05)
Target modules	q/k/v/o_proj, gate/up/down_proj
Trainable params	~0.5% of total
Precision	bf16 compute, gradient checkpointing
Effective batch	32 (8 × 4 grad-accum)
LR / schedule	2e-4, cosine, 3% warmup
Hardware	1× NVIDIA H100 NVL (95 GB), ~19 GB used
Framework

Best-checkpoint selection matters here: eval loss bottomed around step 500 and then rose while train loss kept falling (overfitting). load_best_model_at_end on eval_loss selected the step-500 checkpoint — this model. Training for 3 epochs was more than necessary.

A GRPO / RLVR variant (RL against a programmatic reward: JSON validity + criterion F1) was also trained; it matched this SFT model but did not beat it, because SFT had already saturated the reward. Details in the repo.

Data

Built from 8,000 real ClinicalTrials.gov studies. Targets come from the registry's own structured fields (not a teacher LLM), so labels are auditable. Splits are partitioned by trial, not by example, to prevent leakage.

Limitations

Not medical advice. Outputs may be wrong; a human expert must review anything clinical.
Trained on English registry text only; performance on other formats/languages is unknown.
Summary quality (ROUGE-L 0.29) is the weakest task.
Only 8k of ~500k available trials were used — more data would likely improve generalisation.

License

Apache-2.0, inherited from Qwen2.5. Source registry data is public-domain U.S. government work.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

OmkarShewale

Model Tree

Base

Qwen/Qwen2.5-7B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities