GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
segestic
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
segestic
Model Tree
microsoft/Phi-3.5-mini-instruct
Input Modalities
Output Modalities
Supported Functionality
| Artifact | Repository |
|---|---|
| Merged model (this repository) | segestic/MediliteQA-phi3.5-mini-all-med |
| LoRA adapter | segestic/phi3.5-mini-4k-qlora-medical-seg-vall_med |
| GGUF quantizations (CPU inference) | segestic/MediLITE-QA-GGUF |
Code and evaluation logs: https://github.com/segestic/Medilite-QA
Research on resource-efficient medical language models, including the accuracy, latency, memory, and energy trade-offs of quantized models on commodity CPUs. Clinical decision support and patient-facing applications are out of scope.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "segestic/MediliteQA-phi3.5-mini-all-med"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16, device_map="auto") question = "..."choices = {"A": "...", "B": "...", "C": "...", "D": "..."}options = "\n".join(f"{k}. {v}" for k, v in choices.items()) messages = [{"role": "user", "content": f"{question}\n\n Input: {options}"}]inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device) with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=4, do_sample=False)print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))The model answers multiple-choice questions with a single letter; use the prompt layout above for best results.
QLoRA fine-tuning of Phi-3.5-mini-instruct on 393,000 records (about 104.5 million tokens) from the training splits of three public datasets:
| Dataset | Share of corpus tokens |
|---|---|
| PubMedQA (PQA-A subset) | 86.06% |
| MedMCQA | 11.46% |
| MedQA-USMLE-4-options | 2.49% |
| Setting | Value |
|---|---|
| LoRA | r = 32, alpha = 256, dropout 0.10; q/k/v/o, gate, up, and down projections |
| Learning rate / schedule | 2e-4, linear |
| Batch size | 4 per device × 8 gradient-accumulation steps (effective 32) |
| Epochs / sequence length | 3 / 512 tokens |
| Precision / hardware | fp16 mixed precision / 1 × NVIDIA Quadro RTX 4000 |
The complete configuration is given in Appendix A of the paper.
Zero-shot accuracy (%) averaged over nine medical benchmarks (7,045 items), scored with lm-evaluation-harness;
brackets show bootstrap 95% confidence intervals. Per-benchmark results are in the paper.
| Model | Average accuracy |
|---|---|
| MediLITE-QA (fp16) | 72.79 [70.95, 74.66] |
| MediLITE-QA (Q4_K_M GGUF) | 71.87 [70.05, 73.76] |
| Phi-3.5-mini-instruct (base) | 68.73 [66.75, 70.62] |
| OpenBioLLM-8B | 72.09 [70.22, 73.94] |
| LLaMA 3 8B | 70.52 [68.65, 72.37] |
Fine-tuning improved average accuracy by 4.06 percentage points over the base model (95% CI 2.40 to 5.70). The average was not significantly different from that of OpenBioLLM-8B.
Q4_K_M GGUF file (2.39 GB), llama-cpp-python, 4 threads, no GPU; Intel Core i5-1135G7 with 6.96 GiB RAM.
| Metric | Value |
|---|---|
| Steady-state time to first token | 1.34 s |
| Time per output token | 117 ms (8.19 tokens/s) |
| Peak resident memory (2,000-token context) | 4,403 MiB (61.8% of RAM) |
| CPU-package energy | 3.17 J per generated token |
The 7–8B open models measured on the same hardware used 1.56–1.98× more CPU energy per token.
Fine-tuning used one NVIDIA Quadro RTX 4000 for 1,136.6 h; GPU-board energy was approximately 139 kWh, estimated from logged power telemetry.
bibtex
@article{odewole2026medilite, author = {Odewole, Olusegun Oladele and Başaran, Seren}, title = {MediLITE-QA: Energy-Efficient Medical Question Answering with a Quantized Domain-Specific Language Model on a Commodity Laptop CPU}, journal = {Sustainability}, volume = {18}, number = {21}, pages = {11245}, year = {2026}, doi = {10.3390/su182111245}}import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "segestic/MediliteQA-phi3.5-mini-all-med"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16, device_map="auto") question = "..."choices = {"A": "...", "B": "...", "C": "...", "D": "..."}options = "\n".join(f"{k}. {v}" for k, v in choices.items()) messages = [{"role": "user", "content": f"{question}\n\n Input: {options}"}]inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device) with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=4, do_sample=False)print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))@article{odewole2026medilite, author = {Odewole, Olusegun Oladele and Başaran, Seren}, title = {MediLITE-QA: Energy-Efficient Medical Question Answering with a Quantized Domain-Specific Language Model on a Commodity Laptop CPU}, journal = {Sustainability}, volume = {18}, number = {21}, pages = {11245}, year = {2026}, doi = {10.3390/su182111245}}