GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ramanareddyg
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ramanareddyg
Model Tree
google/gemma-4-E4B-it
Input Modalities
Output Modalities
Supported Functionality
NEET-UG is India's medical entrance exam. About 2.3 million candidates sit for it each year; the questions are drawn from the NCERT Class 11 and 12 textbooks, and roughly 330,000 candidates (~15%) write it in Hindi. Good coaching costs money, and the best AI tutors are paid APIs that need a network connection.
The goal here: an open dataset and an open model on Hugging Face, free, ideally running on a student's own laptop with no internet.
The question I wanted to answer: how close can an open model get to the commercial ones on the real exam, and what does it take?
The inputs are public: NTA's question papers and final answer keys for 2020–2025, and the NCERT Physics, Chemistry and Biology books for Classes 11 and 12.
The happy path is one vision-model call per page, about $0.15 per paper. The last 5% took most of the first day.
The answer keys are parsed, not guessed. No model is asked what the answer is. The keys are read deterministically from NTA's PDFs: 208 booklet-code tables, all complete, including the dropped questions (one per booklet in 2022) and the 2023 question where two options were accepted.
Hindi is never text. Even in NTA's bilingual PDFs, the Devanagari is embedded as vector outlines. Zero Hindi characters can be extracted from 48 pages. Several English papers are scans too. So every page goes through a vision model.
The recitation filter. NEET papers are copied all over the web, and newer Gemini models refuse to transcribe them verbatim. Every Flash model after 2.5 refused whole pages. Gemini 2.5 Flash works on most pages but refused 7 of 32 pages of the 2024 paper. Older papers trip it more, presumably because they have been copied for longer. The fix was a fallback ladder: whole page, then two bands, then three, then a crop per question across three different models, then the whole page asking for one question only. Seven questions out of 1,160 were refused by every model at every rung; they are listed for typing by hand, not guessed.
The model translated instead of transcribing. On one page of an English-only paper, the model returned fluent Hindi for all ten questions. None of it was printed on the page. Hindi now only comes from bilingual booklets, where it actually exists.
Small bugs that would have poisoned training silently:
\rho became a control character plus ho. Repaired deterministically.A free accuracy check. The 2025 English booklet and the 2025 bilingual booklet carry the same English text, extracted independently from different scans. Median similarity between the two transcriptions is 1.00; the differences are LaTeX spacing and table styles.
Result: 1,160 questions from nine booklets, official answers, figures cropped per language, Hindi where NTA printed it, and every question linked to an NCERT passage (passage-level for about 80%, chapter-level for most of the rest).
The NCERT books were split into 1,923 passages (684,000 words), each about a page, and embedded for search. On top of that, Gemini 3.8 Flash generated training material:
Two things worth knowing if you do this. The recitation filter never fired on generation, only on transcription. And the generated assertion-reason questions were 81% "both true, and R explains A", against about 50% in real NEET papers, so they had to be rebalanced.
The test set is 660 items: all 180 questions of NEET 2025 in English and in Hindi, plus 150 generated questions held out from training, also in both languages. 46 items have figures. Every model gets the same prompt, the same images and the same answer parser.
Starting point, plain prompt for everyone:
| Model | All | NEET 2025 EN | NEET 2025 HI | With figure |
|---|---|---|---|---|
| Gemma 4 E4B (open, ~4.5B effective) | 82.1 | 81.7 | 70.0 | 47.8 |
| Claude Haiku 4.5 | 87.0 | 88.3 | 80.0 | 47.8 |
| Gemma 4 31B (open) | 95.0 | 95.6 | 92.2 | 82.6 |
| GPT-6 Luna | 96.1 | 95.6 | 92.8 | 87.0 |
| Gemini 3.8 Flash | 98.3 | 97.8 | 97.2 | 95.7 |
| GPT-6.1 Sol | 98.5 | 98.3 | 96.7 | 95.7 |
| Claude Opus 5.5 | 99.1 | 98.9 | 97.8 | 95.7 |
Even with a plain prompt, the open 31B is within about a point of GPT-6 Luna. The small model's two gaps are Hindi (11.7 points below English) and figures (under 50%).
A note on the parser: Haiku first scored 69%. It was echoing the Hindi answer template ("Answer: <विकल्प संख्या 2>"), which my first regex missed. After the fix it scored 87%. If your eval has a parser, read its failures before you read its scores.
Two changes, neither of which touches the model's weights.
Gemma 4's chat template has thinking off by default. I had not noticed. Turning it on:
Embed the question, find the closest NCERT passage within the subject, paste it above the question. E4B: 82.1 → 86.8, mostly from Chemistry and Biology, the recall-heavy subjects. The search found the passage a human would pick only 64% of the time; when it missed, accuracy barely dropped.
The two fix different things, so the gains add up. With an open embedding model for the search, Qwen3-Embedding-0.6B, the whole pipeline runs with no internet: 90.8%. The open embedder even found the right passage slightly more often than Gemini's embedding API (192 vs 188 of 300).
This setup is also the honest way to give a student facts. The app can show which NCERT page an answer came from, which no amount of training can do.
Same setup for every open model, no training. Commercial models with a plain prompt.
| Model | Runs on | All | NEET EN | NEET HI | Figures |
|---|---|---|---|---|---|
| Gemma 4 E4B, Google's 4-bit build | 16 GB laptop (6.1 GB download) | 87.6 | 83.3 | 78.3 | 50.0 |
| Claude Haiku 4.5 | API | 87.0 | 88.3 | 80.0 | 47.8 |
| Gemma 4 E4B | 16 GB laptop | 90.8 | 88.3 | 83.9 | 47.8 |
| Qwen3.5-4B | 16 GB laptop (~3 GB at 4-bit) | 93.3 | 93.3 | 86.1 | 63.0 |
| Gemma 4 26B-A4B (mixture of experts, 4B active) | 32 GB laptop or 24 GB Mac (15.6 GB download) | 95.0 | 91.1 | 93.3 | 82.6 |
| Qwen3.5-9B | 16 GB laptop (~5.5 GB at 4-bit) | 95.3 | 94.4 | 91.1 | 71.7 |
| GPT-6 Luna | API | 96.1 | 95.6 | 92.8 | 87.0 |
| Gemma 4 31B | 24 GB GPU (18.9 GB download) | 97.7 | 96.1 | 96.1 | 91.3 |
| Gemini 3.8 Flash | API | 98.3 | 97.8 | 97.2 | 95.7 |
| GPT-6.1 Sol | API | 98.5 | 98.3 | 96.7 | 95.7 |
| Claude Opus 5.5 | API | 99.1 | 98.9 | 97.8 | 95.7 |
What stands out:
Notes on the table: only the first row is quantised; the other open models are full-precision runs, and their 4-bit accuracy is not yet measured. Gemma 4 12B is left out because it reasoned past 16,000 tokens on 64 questions in this setup, which says more about my prompt than about the model. Qwen3.5 thinks long, a median of about 3,000 tokens per answer for the 4B against 1,000 for E4B, so on a laptop CPU it takes roughly three times as long.
| E4B max output tokens | Accuracy | Answers cut off | Of those, still right |
|---|---|---|---|
| 8,192 | 90.8 | 1 | – |
| 2,560 | 87.0 | 40 | 2 |
| 1,536 | 75.6 | 143 | 9 |
Every truncated answer is a lost question. A product needs "stop thinking and give your best answer now", not a hard limit. I have not built that yet.
Before finding the setup, I spent most of two days fine-tuning Gemma 4 E4B. All three runs were LoRA (rank 16) on Modal H100s. None beat the untrained model, and each failed for a different reason worth knowing.
Training data: the Gemini explanations plus the generated questions, about 16,000 examples. Validation loss fell beautifully, from 1.96 to 0.68. Accuracy fell on every cut.
The cause was visible in one number: the median answer shrank from 1,537 characters to 308. The targets were short explanations written by a teacher that already knew the answer. The base model reasons at length and checks itself; the tuned model learned the teacher's concise style and stopped thinking before answering. On one microscope question, the base model computed 145, noticed it was not an option, tried the textbook simplification and got 125, correct. The tuned model computed 145 and picked the nearest option.
Lesson: distilling explanations is not distilling reasoning.
Training data: E4B answered every training prompt itself, four times each; I kept the attempts that reached the official answer, plus Gemma 31B's answers where E4B never succeeded. 17,363 examples. The dev score fell from 81.2% to 71.3% before I stopped it, while training loss sat at 0.20.
89% of the examples were the model's own text on questions it already got right every time, so there was nothing to learn. Yet when I measured how far each adapter had moved the weights, v2 had moved as fast per unit of learning rate as v1, spread evenly over all 42 layers, the pattern of accumulated noise rather than a learned behaviour. Adam takes full-sized steps whether or not the gradient means anything.
Lesson: a flat loss curve is not a safe one. Measure how much of your data the model already gets right before training on it.
v3 applied everything above and two outside reviews of the plan: train in the shipping setup (thinking on, NCERT page in the prompt), use Gemma 31B's own thinking traces as targets, train only on the ~1,000 prompts E4B gets wrong plus 45% already-correct ones, at a 5× lower learning rate, with a 280-item held-out validation set and paired comparisons.
It worked as designed: on held-out questions the model used to fail, it went from 13% right to about 30%. But it lost the same number of points elsewhere—on Chemistry and Hindi in the 660 eval, and on figures in the validation set. On the 660 eval: 90.0 against 90.8 untrained, a difference of −0.8 with a 95% interval of −3.0 to +1.7.
One more finding. The 150 held-out generated questions all come from passages whose other questions were in training, so they test "same fact, asked a new way". Training on four questions per passage did not help the model answer a fifth (v1 made it worse, 89.7 → 85.3). Handing it the passage did (96.3).
Four days, 4–7 October 2026.
| Item | Cost |
|---|---|
| Gemini: transcription, linking, explanations, question generation and checking | ~$65* |
| API comparison runs (Claude, OpenAI, Gemini) | $10.50 |
| Modal GPUs: all evals, sampling, three fine-tunes and the false starts | ~$55 |
| Total | ~$130 |
* Estimated. The Gemini bill is not split by task. The project hit its ₹5,000 (~57)spendingcappartwaythroughcheckingthegeneratedquestions;finishingthatcheckafteratop−upcostanother 7.
This was the third of three fine-tunes. The first, on short explanations, made the model 8 points worse; the second, on its own answers, also made it worse. Why each one failed is in the write-up linked from the GitHub repository.
python
import torchfrom peft import PeftModelfrom transformers import AutoModelForMultimodalLM, AutoProcessor base = "google/gemma-4-E4B-it"processor = AutoProcessor.from_pretrained(base)model = AutoModelForMultimodalLM.from_pretrained(base, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "ramanareddyg/gemma-4-E4B-neet-lora-v3") messages = [{"role": "user", "content": [{"type": "text", "text": "NCERT reference: ...\n\nAnswer this NEET-UG multiple-choice question. ..."}]}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=4096)print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))vLLM can also load it as a LoRA adapter on top of the base model (enable_lora=True, max_lora_rank=32).
import torchfrom peft import PeftModelfrom transformers import AutoModelForMultimodalLM, AutoProcessor base = "google/gemma-4-E4B-it"processor = AutoProcessor.from_pretrained(base)model = AutoModelForMultimodalLM.from_pretrained(base, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "ramanareddyg/gemma-4-E4B-neet-lora-v3") messages = [{"role": "user", "content": [{"type": "text", "text": "NCERT reference: ...\n\nAnswer this NEET-UG multiple-choice question. ..."}]}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=4096)print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))