xenkrypt

MedLlama-India-70B

README

License: apache-2.0

Benchmark Results

AutoScientist Internal Evaluation

Table with columns: Metric, Base Model, MedLlama-India-70B, Improvement
Metric	Base Model	MedLlama-India-70B	Improvement
Win Rate (on dataset)	20	80	+300% relative
Medical Domain Win Rate	26	74	+185% relative

External Baseline Evaluation

Table with columns: Model, Accuracy, Notes
Model	Accuracy	Notes
Mistral 7B Instruct v0.2 (zero-shot)	44.68%	Evaluated on MedMCQA validation set (4,183 examples)
MedLlama-India-70B	~74% (AutoScientist medical win rate)	Llama 3.3 70B fine-tuned

Benchmark: openlifescienceai/medmcqa validation set — 4,183 questions across 21 medical subjects.

Training Details

Table with columns: Parameter, Value
Parameter	Value
Base model	meta-llama/Llama-3.3-70B-Instruct
Method	LoRA (PEFT)
LoRA rank	64
LoRA alpha	128
Target modules	all-linear
Epochs	3
LR scheduler	cosine
Warmup ratio	0.05
Gradient clipping	1

AutoScientist Training Metrics

Win rate: 20 → 80 (+300%)
Medical win rate: 26 → 74 (+185%)
Final training loss: 0.73
Training steps: 624
Loss curve: clean convergence, no overfitting

Adaptive Data Processing

Source: MedMCQA (170,000 questions)
Quality grade: B → A (28.9 → 57.7 percentile)
Recipes: Reasoning Traces, Hallucination Mitigation, Prompt Rephrase, Metadata Injection, House Special
Localization: India (English)
Blueprint: AIIMS/NEET-PG medical examiner persona

Dataset

xenkrypt/MedLlama-India-Dataset

38,000+ instruction-formatted NEET-PG and AIIMS-PG questions processed via Adaption Labs Adaptive Data with reasoning traces, hallucination mitigation, and Indian medical context localization.

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("xenkrypt/MedLlama-India-70B")
model = PeftModel.from_pretrained(base, "xenkrypt/MedLlama-India-70B")

prompt = """### Instruction:
You are a medical expert for AIIMS/NEET-PG examinations.
Answer this multiple choice question.

Question: Most common cause of mitral stenosis?
A) Rheumatic fever
B) Infective endocarditis
C) Congenital
D) SLE

### Response:
The correct answer is"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=150, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Limitations

LoRA adapter — requires base model meta-llama/Llama-3.3-70B-Instruct
Trained on exam MCQs — validate before clinical use
Not a substitute for professional medical advice

Demo

https://huggingface.co/spaces/xenkrypt/MedLlama-India-Demo

Submitted To

Adaption Labs AutoScientist Hackathon — Healthcare, Part 1, July 2025

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

xenkrypt

Model Tree

Base

meta-llama/Llama-3.3-70B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities