Model summary
A LoRA adapter for Qwen/Qwen3-8B, trained with QLoRA (4-bit NF4 base) on
MedMCQA training questions. Released as one arm of a controlled six-arm
benchmark, not as a general medical assistant.
Table | |
|---|
| Base model | Qwen/Qwen3-8B @ b968826d9c46dd6066d109eabc6255188de91218 |
| Method | QLoRA — LoRA r=16, α=32, dropout 0.05, over 4-bit NF4 |
| Target modules | q,k,v,o,gate,up,down_proj (attention and MLP) |
| Trainable params | 43,646,976 of 4,761,498,624 (0.92%) |
| Training data | 30,000 MedMCQA train rows (Apache 2.0) |
| Compute | 1,875 steps, 1 epoch, 5h04m on one NVIDIA A40 |
| Final loss | train 1.287, best eval 1.201 (on a held-out validation split) |
| Precision | trained in 4-bit NF4; evaluated merged into bf16 |
Intended use
In scope. Reproducing the benchmark in
the accompanying repository;
research on fine-tuning versus retrieval trade-offs; a baseline for further
medical-QA experiments.
Out of scope. Any clinical, diagnostic, triage, or treatment context.
Patient-facing applications. Generating medical content for consumption without
expert review. Any use where being confidently wrong carries a cost.
Results
Scored on a frozen, subject-stratified 1,000-item test set carved from MedMCQA's
labelled validation split. (MedMCQA's own test split has cop == -1 —
labels are withheld — so it cannot be used for evaluation.)
Table with columns: Arm, Accuracy, 95% CI, p50 latency, Prompt tokens| Arm | Accuracy | 95% CI | p50 latency | Prompt tokens |
|---|
base — Qwen3-8B zero-shot | 56.8% | [53.8, 59.9] | 103 ms | 113 |
qlora — this adapter | 62.9% | [60.0, 65.9] | 102 ms | 113 |
rag-parity — retrieval, no fine-tune | 67.0% | [64.1, 69.9] | 171 ms | 656 |
+6.1 points over base, paired McNemar p < 0.0001 over identical items.
The headline finding is not flattering to this adapter. Retrieving the same
training explanations at inference time scores higher (67.0%) than absorbing
them into these weights (62.9%), and the gap holds across three training seeds
(+4.2 points, 95% CI [+1.0, +7.2], p = 0.008).
Same base model, same prompt, same information — one in an index, one in the
weights, and the index wins.
The adapter's advantage is cost, not accuracy: merged, it has identical
inference latency to the base model while serving 113-token prompts instead of
656. It repays its training cost at roughly 200,000 queries.
Training data
MedMCQA, Apache
2.0 — Indian medical entrance exam questions (AIIMS/NEET-PG). No PHI: exam
questions with de-identified vignettes.
Preprocessing, all documented and reproducible:
- OCR repair. MedMCQA's text has systematic extractor corruption in which
the
rt bigram is dropped, and the corrupted spelling is more common than
the correct one (aery 11,203 vs artery 7,009; hea 7,626 vs heart
4,957). Repaired with a 225-entry human-reviewed lexicon covering 93,174
tokens.
- Answer-key stripping. Many explanations open with the answer letter
("Ans. is 'd' i.e., ..."). Training on those teaches the letter, not the
medicine, so the boilerplate is removed and explanations reduced to nothing
are dropped.
- Leakage gate. Rows are filtered on content hash, not id — MedMCQA repeats
items across splits under different ids. 0 leaked rows reached training.
- Subsampling. A seeded random 30,000 of the ~179,600 clean rows.
Targets are the answer letter followed by the explanation. The letter comes
first so that first-token scoring grades this arm identically to every other
arm; the explanation is included so the fine-tune sees the same text the
retrieval arm serves.
Limitations
-
Contamination is present, and measured. MedMCQA predates Qwen3 and is a
widely mirrored public benchmark, so it almost certainly appears in
pretraining. Three probes were run rather than assumed:
- Verbatim reproduction. ~9% of test stems are regenerated well above a
shuffled-reference chance baseline (base 10.0% vs 1.3% control;
this adapter 8.7% vs 2.0%). Some absolute accuracy above is recall.
- Option permutation. Shuffling answer positions moves the base model by
-1.6 points — it is not relying on memorised answer labels. This
adapter loses 3.2 points, so fine-tuning introduced a mild positional
shortcut the base model did not have.
- Position bias. Negligible for both (max excess 0.029 and 0.045).
Because contamination is roughly constant across arms — they share a base
model and a test set — the between-arm comparisons are largely unaffected,
which is what the benchmark exists to measure. Absolute accuracy is inflated;
the deltas are not. Full detail in
REPORT.md.
-
Three training seeds, reported at one. Every table reports seed 42.
Retrained at seeds 1 and 2, this recipe scores 62.9%, 63.9% and 61.7%
(62.8 ± 1.1). Differences between configurations smaller than that spread —
including every LoRA-rank and epoch ablation in the repository — are not
evidence of anything.
-
The MCQ gain does not survive free-text generation. Every accuracy
above is 4-option multiple choice scored by constrained log-probability.
Asked the same 300 test questions open-ended (options hidden) and graded by
a cross-family 70B judge against the gold answer, this adapter scores
below the base model it was trained from — 0.396 vs 0.454 graded (p =
0.024), −4.0 points and not significant when scored strictly as
correct-or-not — while retrieval over the same explanations gains +9.3
graded / +12.0 strict. A 50-item human check of the judge gave κ = 0.51 on
the three-way rubric and 0.76 on correct-or-not; the judge is lenient on
partial credit, which is why both scorings are reported. The adapter
learned to rank candidates and an exam-answer register, not the facts
needed to produce an answer. A stricter replication on MIRIAD (the same
passages in the weights and in the index) puts the same adapter design
significantly below base under both scorings (−8.8 graded, −7.3 strict).
Do not use this adapter for generation; it is an MCQ artefact. No
citation or calibration assessment beyond confidence margins. Full detail
in REPORT.md §2.4–2.5.
-
31.3% of the evaluation pool is dentistry, not general medicine. Results
excluding it are meaningfully different (base 59.8%, this adapter 66.5%).
-
English only, and the questions reflect the Indian medical curriculum.
-
Merged evaluation. Latency figures assume the adapter is merged into the
base. Served unmerged it is ~4.9× slower.
Usage
from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer BASE = "Qwen/Qwen3-8B"REV = "b968826d9c46dd6066d109eabc6255188de91218" # pragma: allowlist secret tok = AutoTokenizer.from_pretrained(BASE, revision=REV)model = AutoModelForCausalLM.from_pretrained(BASE, revision=REV, dtype="bfloat16")model = PeftModel.from_pretrained(model, "<this-repo>").merge_and_unload()
The benchmark pins enable_thinking=False in the chat template for every arm,
because variable-length reasoning traces confound latency and cost measurement.
Reproducing the numbers above requires the same setting.
Licences
- This adapter: Apache 2.0, inherited from the Qwen3-8B base.
- Training data: MedMCQA, Apache 2.0.
- Benchmark code: MIT.
MedQA was deliberately not used anywhere in this project: bigbio/med_qa
declares its licence "unknown", and a re-uploader's cc-by-4.0 tag does not
launder upstream copyright on USMLE board-prep material.
Citation
@software{koli_finetune_vs_rag_2026, author = {Koli, Viresh}, title = {Fine-Tune vs. Retrieve: A Controlled Benchmark in Clinical QA}, year = {2026}, url = {https://github.com/vireshkoli/Fine-Tune-vs-RAG}}