Training data
saurabhb2412/eka-clinical-sft-train — 219,662 train / 1,103 val conversations (~259M packed tokens, 4096-token BFD packing):
- 99,509 lab reports rendered from ekacare/NidaanKosha-100k-V1.0 (CC-BY-SA-4.0): user message = the subject's report as a test list with reference ranges; assistant = abnormal-value summary generated from the data itself. Multi-line categorical ranges (lipid/HbA1c/vitamin-D tables) are never flagged numerically.
- 120,153 MCQAs from openlifescienceai/medmcqa train (Apache-2.0), single-choice rows with non-null options, answered as
(letter) + source explanation.
Training
TRL 1.18 SFTTrainer, 1 epoch, packed 4096, effective batch 16 (8×2), lr 2e-5 cosine, 50 warmup steps, bf16, kernels-community/flash-attn2. Run on 1× H200: 4,341 steps at 2.45 s/step, 25.9k tokens/s. Final train loss 0.3297, eval loss 0.3162, eval token accuracy 0.914. Live curves: Trackio dashboard.
Evaluation (ekacare MCQA battery, greedy letter-parse, full test splits)
Table with columns: Set, n, accuracy| Set | n | accuracy |
|---|
| ekacare/indian_drug_mcqa | 1,512 | 0.397 |
| ekacare/Eka_NFI_MCQA | 921 | 0.455 |
| ekacare/MedMCQA-Indic en | 4,183 | 0.438 |
| ekacare/MedMCQA-Indic hi | 4,183 | 0.348 |
| mmlu-medical (8 configs: anatomy, clinical-knowledge, college-biology/medicine, medical-genetics, nutrition, professional-medicine, virology) | 1,553 | 0.466–0.671 |
Parse rate ≥ 0.98 on every set. eval_results.json is in this repo. Honest read: the base Qwen3-1.7B already scores mid-50s on MedMCQA-style MCQA, so SFT on this mix traded general MCQA accuracy for lab-report skill — the drug-MCQA drop below the base model is expected and noted, not hidden. Hindi MCQA degrades most (0.348).
Limitations
- Lab-report assistant outputs are template-generated from NidaanKosha values; the model learns flagging and format, not clinical judgment.
- MCQA accuracy is moderate and below the base instruct model on several sets — use for lab-report workflows, not as a replacement for a general medical LLM.
- No Hindi lab-report training data; Hindi MCQA degradation is unmeasured for report-style inputs.
Quick start
from transformers import pipeline report = """Patient: 45-year-old maleLab report:- Hemoglobin: 9.8 g/dL (reference: 13.0 - 17.0) - LOW- Total Leucocyte Count: 7.8 10^3/uL (reference: 4.0 - 10.0) - NORMAL Summarise this lab report: list any abnormal values with their reference ranges, note the normal results in one line, and state what the patient should do next."""generator = pipeline("text-generation", model="saurabhb2412/qwen3-1.7b-eka-clinical-sft", device_map="auto")out = generator([{"role": "user", "content": report}], max_new_tokens=256, return_full_text=False)[0]print(out["generated_text"])
Framework versions
- TRL 1.18.0 · Transformers 5.23.0 · PyTorch 2.15.0 · Datasets 5.2.0 · Tokenizers 0.23.3