Contents
adapter_model.safetensors, adapter_config.json: trained rank-16 LoRA (43,278,336 parameters).
joint_head.safetensors, joint_head_config.json: trained head (121,762,820 parameters).
inference.py: portable loader and prediction interface using AutoTokenizer only.
joint_schema_model.py: unmodified Apache-2.0 Clef implementation from the pinned base release.
clef_med/: input encoding, loss, training, evaluation and data preparation source.
release_config.json, training_config.json, split_manifest.json: reproducibility metadata.
calibration.json: temperature and empirically selected abstention thresholds.
metrics.json, comparison.csv: evaluation results.
SHA256SUMS.json: SHA-256 and size of each release file except the checksum file itself.
The adapter and head occupy approximately 660 MB. The full base backbone is not
duplicated. Raw dataset examples, access credentials and machine-specific paths
are not included.
Training and data
Table with columns: Item, Value| Item | Value |
|---|
| Training | 100 original questions from MedHallu pqa_artificial, expanded to 200 answer examples |
| Validation | 200 other questions / 400 examples; minimum validation NLL selects the checkpoint |
| Calibration | 200 other questions / 400 examples; temperature and risk thresholds only |
| Final evaluation | All 1,000 pqa_labeled questions / 2,000 answer examples |
| Epochs / optimizer updates | 3 / 75; selected epoch 3 |
| Seed | 42 (one pilot run) |
| LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
| Targets | Text-layer full-attention, linear-attention and MLP projections |
| Trainable modules | LoRA + complete Joint Schema Head; original backbone stays frozen |
| Batch / gradient accumulation | 1 / 8 |
| Maximum input length | 2,048 tokens |
| Learning rates | Head 5e-5; LoRA 1e-4 |
| Loss | CE with label smoothing 0.05 + 0.1 x multiclass Brier |
| Precision | BF16 backbone; FP32 trained head/adapter with BF16 autocast |
| Hardware | One NVIDIA RTX A6000 |
The CE/Brier objective is a baseline adaptation choice, not claimed as a novel method.
The positive class is hallucinated. Each question contributes one supported
answer and one synthetic hallucinated answer. Ground Truth is used as the
supported candidate answer, never as an additional answer key in the input.
Question and abstract overlap are prevented by grouping connected components of
normalized question/abstract identities before splitting. This is a study-defined
assignment; both original Hugging Face configurations call their split train.
Larger 500/2,000-question sets were prepared but were not trained for this release.
Evaluation
Final held-out pqa_labeled results, before temperature scaling:
Table with columns: Method, Accuracy, Hallucination F1, AUROC, Binary Brier (lower better)| Method | Accuracy | Hallucination F1 | AUROC | Binary Brier (lower better) |
|---|
| Unadapted Clef-Flash | 0.7870 | 0.7311 | 0.9230 | 0.1628 |
| Head-only | 0.8135 | 0.7733 | 0.9362 | 0.1262 |
| LoRA + Head (this release) | 0.9565 | 0.9567 | 0.9896 | 0.0348 |
The uploaded checkpoint is the LoRA + Head row. The primary reported accuracy is
95.65%, with 1,913 correct decisions out of 2,000. Temperature scaling used only the
separate calibration set; fitted temperature is 0.9272774008.
Raw and temperature-scaled results are stored separately in metrics.json.
Reported binary Brier is (p_hallucinated - label)^2; the two-class Brier loss
used in training sums over classes and is twice that quantity.
LoRA + Head inference averaged 355.9 ms/example
at batch size 1 on the recorded A6000 run, with the adapter unmerged. Timing
includes collation, device transfer and synchronized forward; tokenization,
model loading and warmup are excluded. Treat this as one hardware/workload
measurement, not a general latency guarantee.
Usage
Use Python 3.12 and install the provided requirements. The verified environment
used PyTorch 2.11.0+cu128, Transformers 5.10.2 and PEFT 0.19.1.
CUDA inference is required by the supplied loader. No torchvision/video processor
is needed for this text-only task. Optional fast-path libraries were absent in
the evaluation run; the torch fallback was used.
import sysfrom huggingface_hub import snapshot_download path = snapshot_download("aliAs09/clef-flash-medhallu-lora-r16-100q") # authenticate with read access for this private reposys.path.insert(0, path)from inference import load_model detector = load_model(path) # downloads the pinned Clef-Flash base if neededresult = detector.predict( question="Did the intervention group show a higher response rate?", knowledge="In this illustrative study, response occurred in 70% of the intervention group and 40% of controls.", answer="The intervention group showed a higher response rate than controls.",)print(result)
The example above is synthetic and only demonstrates the input format.
To reuse an existing base snapshot, pass base_path containing the exact revision below.
p_hallucinated_raw reproduces the uncalibrated prediction; the second returned
probability applies the stored temperature. The decision threshold is 0.5.
If an input exceeds the token limit, only the abstract tail is truncated and the
result flags this. A question/answer/schema that alone exceeds the limit raises an error.
Scope and limitations
This is a one-seed, small-data research pilot on MedHallu, not a clinical product
or a guarantee of medical correctness. Hallucinated answers in pqa_labeled are
synthetic, even though its original questions come from PubMedQA's labeled portion.
The benchmark is balanced and may differ from deployed answer distributions.
High scores may partly reflect dataset-specific synthetic-answer patterns;
external datasets, multiple seeds and broader error analysis remain to be evaluated.
Any pretrained-base exposure to related medical benchmarks is not controlled here.
Calibration-set risk targets are empirical and provide no population or clinical
safety guarantee. This release does not claim that using LoRA makes the backbone
smaller or inherently faster.
Provenance and license
- Base:
Cloudflare/clef-flash, revision 17f0b0ad64efb65d273590632833508766b2aae6.
- Dataset:
UTAustin-AIHealth/MedHallu, revision 515060458a945c633debc6fd5baac7764416b724.
- Clef-Flash and its code: Apache-2.0; the license is included.
- MedHallu data card specifies MIT; no raw dataset records are redistributed here.
- Training and adaptation code in this release is distributed under Apache-2.0.
See Clef-Flash,
MedHallu, and
the MedHallu paper.