GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
jashu1326
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
jashu1326
Model Tree
openai/whisper-medium
Input Modalities
Output Modalities
Supported Functionality
There are two evaluation runs here and only one of them is valid.
| Run | n | Status | Where |
|---|---|---|---|
| Decontaminated | 482 | Use this | results/ |
| Original random split | 709 | Contaminated - do not cite | superseded/ |
The original 5% random split was made at the utterance level. Shrutilipi is mined from All India Radio bulletins, where the same news item is read repeatedly across editions, so 227 of the 709 held-out utterances (32.0%) had a near-duplicate in the training set. Every metric computed on that split measures partial memorisation as much as generalisation.
The results/ numbers are computed on the 482 utterances that survived screening. Those
are the numbers in the paper.
| System | BLEU [95% CI] | chrF | METEOR | BERTScore F1 |
|---|---|---|---|---|
| Whisper Medium, zero-shot | 5.68 [4.62, 6.72] | 29.05 | 21.89 | 86.42 |
| + LoRA (this adapter) | 23.74 [22.02, 25.38] | 50.74 | 52.26 | 92.79 |
| Absolute improvement | +18.07 | +21.69 | +30.37 |
Paired bootstrap over 482 segments, 1000 resamples: dBLEU = +18.07, p = 0.001. No resampled draw favoured the zero-shot baseline.
Distribution, not just the mean. Median sentence-level BLEU rises 3.9 to 17.0.
71 utterances (14.7%) clear BLEU 40 after adaptation, against 2 before. 42 utterances
(8.7%) score lower after adaptation - the gain is a large net effect, not a uniform one.
See results/fig3_segment_distribution.png.
results/paper_values.txt carries every figure quoted in the paper, regenerated from the
committed predictions.
text
README.md <- you are hereadapter_config.json <- the reported model: LoRA adapter foradapter_model.safetensors checkpoint-4000, 37.8 MBprocessor_config.jsontokenizer.jsontokenizer_config.jsontraining_args.bin <- torch.load() for the full TrainingArgumentsresults/ <- AUTHORITATIVE: everything the paper reports base_vs_finetuned.csv 482 rows: reference, base_whisper, finetuned base_vs_finetuned_summary.csv the metrics table above per_segment_scores.csv per-utterance BLEU, sorted worst-delta first fig3_segment_distribution.png BLEU distribution, before vs after paper_values.txt every figure quoted in the papersuperseded/ <- contaminated 709-run, kept only for provenance README.md why these must not be cited eval_results_n709_contaminated.csv eval_summary_n709_contaminated.csvfinal_processed_data/ <- the split the model was trained on train/ 13,467 utterances test/ 709 utterances (the contaminated held-out portion)whisper-sanskrit-en/ <- training checkpoints, every 20 steps checkpoint-4000/ <- THE REPORTED MODEL checkpoint-340 ... checkpoint-3980 intermediate, not used in the paper.virtual_documents/ __notebook_source__.ipynb <- the training notebookIf you only want one thing: results/base_vs_finetuned.csv plus the top-level adapter
files. Everything else is provenance.
Recoverable from training_args.bin and
whisper-sanskrit-en/checkpoint-4000/trainer_state.json.
| Base model | openai/whisper-medium (769M params) |
| Method | LoRA via PEFT 0.18.1 |
| Rank / alpha / dropout | 32 / 64 / 0.05 |
| Target modules | q_proj, v_proj - encoder self-attn, decoder self- and cross-attn |
| Adapted projections | 144 |
| Trainable params | ~9.44M (1.2% of base), 37.8 MB adapter |
| Train / test | 13,467 / 709 () |
No checkpoint selection was performed. The 4000-step budget was fixed in advance and validation loss fell monotonically to the last step, so the reported model is simply the terminal checkpoint - not a best-scoring intermediate one picked on held-out data.
Both systems decoded over identical inputs with identical settings: beam search with
5 beams, max 444 new tokens, no-repeat 4-gram, repetition penalty 1.3, and a forced
decoder prompt of language="sanskrit", task="translate".
Metrics - quote these signatures if you report the numbers:
text
BLEU nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0chrF nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.6.0METEOR: HuggingFace evaluate (NLTK backend), scaled x100.
BERTScore: roberta-large, baseline rescaling DISABLED.
Rescaled and unrescaled BERTScores are not comparable. The zero-shot model scores 86.42 not because it is 86% accurate but because unrescaled BERTScore is bounded well away from zero for any pair of fluent English sentences. Treat it as relative only.
1. References are machine-generated. Shrutilipi supplies Sanskrit audio aligned to Sanskrit text, not English. The English side was produced by IndicTrans2. Every metric here measures agreement with IndicTrans2, not accuracy against a human standard. There are documented cases where the reference is wrong and this model is right and is penalised for it - so the lexical scores are a lower bound.
2. Inference precision is not matched. The zero-shot baseline ran in fp32; the adapted model ran 8-bit quantised with fp16 activations. Quantisation degrades output, so the confound biases against the adapted model - the reported margin is conservative.
3. Decontamination is coarse. The screen hashes the first ten tokens of each target label. It misses paraphrases that diverge early and over-removes distinct utterances that share a formulaic opening. Treat 32.0% as a lower bound on contamination.
4. Two utterances are reproduced verbatim. The adapted model outputs its reference exactly for two items, including a 13-word sentence with two proper nouns. That is either genuine quality or residual memorisation the prefix screen missed; the present evidence cannot separate them. A full-string similarity screen is the fix.
5. No WER/CER. The acoustic stage is unmeasured, so claims about error propagation from recognition to translation are argued, not measured.
6. Domain is narrow. Read broadcast news, professional newsreaders. No claim is made about liturgical, pedagogical or conversational Sanskrit.
No model needs to be loaded - the predictions are committed.
python
import pandas as pd, evaluatefrom huggingface_hub import hf_hub_download csv = hf_hub_download("jashu1326/whisper-full-backup-checkpoint-1100", "results/base_vs_finetuned.csv", repo_type="model")df = pd.read_csv(csv)refs = df["reference"].tolist()bleu = evaluate.load("sacrebleu")for col in ["base_whisper", "finetuned"]: print(col, bleu.compute(predictions=df[col].tolist(), references=[[r] for r in refs])["score"])Runs on CPU in seconds.
python
from transformers import WhisperProcessor, WhisperForConditionalGenerationfrom peft import PeftModel REPO = "jashu1326/whisper-full-backup-checkpoint-1100"processor = WhisperProcessor.from_pretrained("openai/whisper-medium", language="sanskrit", task="translate")model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-medium")model = PeftModel.from_pretrained(model, REPO) # decode with: num_beams=5, max_new_tokens=444,# no_repeat_ngram_size=4, repetition_penalty=1.3Kaggle notebook (jashwanth113/) | Produced |
|---|---|
notebook92eb7ffbd6 | results/base_vs_finetuned.csv - decontaminated, 5-beam |
notebook1d9e855503 | superseded/ - 709 contaminated, greedy decoding |
.virtual_documents/__notebook_source__.ipynb | the training run |
The two evaluation runs differ in three ways at once - sample, contamination and decoding strategy - so their numbers are not comparable to each other. Only the decontaminated run is reported.
bibtex
@inproceedings{jashwanth2026sanskrit, title = {Fine-Tuning Whisper Medium for Sanskrit-to-English Speech Translation: A Low-Resource Case Study with Error Analysis}, author = {Jashwanth, Veerlapati Sai and Vamshi, Seelam Shiva and Singh, Yuvraj and Darshith}, booktitle = {Proc. 2nd Int. Conf. on Computational Intelligence and Data Communication}, year = {2026}}Built on Whisper (Radford et al., 2023), LoRA (Hu et al., 2022), Shrutilipi (Bhogale et al., 2023) and IndicTrans2 (Gala et al., 2023).
| +6.37 |
train_test_split(test_size=0.05)| Batch size | 8 x grad-accum 2 = effective 16 |
| Peak LR / schedule | 1e-4, linear decay to zero, no warmup |
| Budget | 4000 steps (~4.75 epochs), no early stopping |
| Reported checkpoint | terminal checkpoint at step 4000 |
| Precision | fp16 mixed |
| Final train / val loss | 1.934 / 1.227 |
| Hardware | Kaggle notebook, single NVIDIA Tesla T4 |