oshinmiranda26

qwen2.5-0.5b-lora-clinical-note-specialty

Deploy Dedicated

README

License: apache-2.0

Intended use

Intended: research and education on parameter-efficient fine-tuning for clinical text classification.
Out of scope: any clinical, billing, or operational use. Not validated on real-world clinical notes.

Training

Base model: Qwen2.5-0.5B (about 496 million parameters), frozen
Method: LoRA, rank 16, on the attention projections; about 0.44% of parameters trained
Data: MTSamples, publicly available transcribed medical reports (via Kaggle), specialty labels; patient-level details are not involved. Check the source dataset's terms before reuse.
Imbalance handling: class-weighted loss; inputs up to 512 tokens
Compute: about 56-62 minutes on a laptop GPU (Apple MPS)
Reproducibility: retraining from scratch with the same seed reproduced the reported metrics exactly

Results (held-out test set)

Table with columns: Model, Macro-F1, Accuracy
Model	Macro-F1	Accuracy
TF-IDF + logistic regression	0.820	0.890
Bio_ClinicalBERT, 512 tokens, class weights	0.794	0.874
This adapter (Qwen2.5-0.5B + LoRA)	0.783	0.879

The simple baseline performed best: specialty labels in this dataset are largely vocabulary-driven, which keyword features capture well. The adapter was strongest on surgery (F1 0.97), orthopedic, and urology notes (0.92 each) and weakest on neurology (0.42), whose notes overlap with other specialties.

Limitations

No separate validation set was used for model selection (stated in the repository as a limitation).
MTSamples are sample reports, cleaner and less varied than real clinical documentation.
Small base model; results do not represent larger LLMs.

How to load

python

from peft import PeftModelfrom transformers import AutoModelForSequenceClassification, AutoTokenizer base = AutoModelForSequenceClassification.from_pretrained("Qwen/Qwen2.5-0.5B", num_labels=NUM_LABELS)model = PeftModel.from_pretrained(base, "oshinmiranda26/qwen2.5-0.5b-lora-clinical-note-specialty")tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

Set NUM_LABELS and the label order to match the training code in the repository.

Author

Oshin Miranda, PhD. GitHub | LinkedIn

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

oshinmiranda26

Model Tree

Base

Qwen/Qwen2.5-0.5B

Adapter

this model

Input Modalities

Text

Output Modalities