Model Summary
A LoRA adapter that fine-tunes Qwen/Qwen3-4B-Instruct-2507
on medical chain-of-thought reasoning. It is the supervised-fine-tuned ("after") checkpoint from a
controlled calibration study of small medical LLMs. The headline finding: this fine-tune makes the
model's internal (token-probability) confidence markedly better calibrated, while its
verbalized (self-reported) confidence does not improve - the two diverge. Trained entirely on a
single free Kaggle T4.
Model Characteristics
- Base: Qwen3-4B-Instruct-2507 (Apache-2.0), 4B params, text-only.
- Method: QLoRA - 4-bit NF4 (double-quant), LoRA r=16, α=32, dropout 0.05 on
q/v/o/gate/up/down projections. Trainable params well under 1% of the model.
- Training: 500 steps (~1.33 epochs), seq len 512, effective batch 16, lr 2e-4, fp16.
Final/avg
train_loss = 1.477 (from ~1.7), grad-norm stable ~0.27, no divergence. ~3.5 h on one T4.
- Format: PEFT LoRA adapter (
adapter_config.json + adapter_model.safetensors). Not a standalone model -
load on top of the base. (Base tokenizer is used; not bundled.)
Data Overview
Evaluation Results
Before (base) vs. after (this adapter) on MedQA + MedMCQA (0-shot, n≈800), with 95% paired bootstrap
CIs (2000 resamples). Bold = the change is statistically significant (CI excludes 0).
Internal confidence (max-softmax over option logits - the deployable signal):
Table with columns: metric, base, +this adapter, Δ [95% CI]| metric | base | +this adapter | Δ [95% CI] |
|---|
| Accuracy ↑ | 0.611 | 0.576 | -0.036 [-0.063, -0.007] |
| ECE ↓ (calibration error) | 0.351 | 0.085 | -0.266 [-0.287, -0.220] |
| AURC ↓ (selective prediction) | 0.223 | 0.236 | +0.013 [-0.013, +0.031] (n.s.) |
| Abstention rate | 0.125 | 0.086 |
Verbalized confidence (the "Confidence: NN" it writes after reasoning; unparsed dropped): ECE rises
0.232 to 0.342 (+0.073 [+0.009, +0.130]) - i.e. internal calibration improves while spoken confidence
does not. Trust the logits, not the self-report.
Takeaway: internal calibration error roughly quartered (no "confidence tax" in selective
prediction), at a small accuracy cost. Replicated on MedGemma-4B (ECE 0.304 to 0.126). Full method, CIs,
and the matched-accuracy decomposition: full method, CIs, and code: https://github.com/Mr-Neutr0n/confidence-tax-study
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = "Qwen/Qwen3-4B-Instruct-2507"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, "<this-adapter-path>")
License & citation