GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
theshivam7
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
theshivam7
Model Tree
openai/whisper-medium
Input Modalities
Output Modalities
Supported Functionality
A sibling model, theshivam7/whisper-medium-indian-english, was fine-tuned and evaluated on TIE_shorts' official splits, which have speaker overlap between train and test. That model showed no net WER improvement over the pretrained baseline. This repo re-runs the same fine-tune with speaker overlap eliminated, to check whether the earlier result was simply masked by a favorable split.
It wasn't, and the honest result is worse than a null. This checkpoint (training seed 42) raises WER from 14.42% (pretrained, same decoding pipeline) to 16.17%, a +1.75pp regression that survives Holm-Bonferroni correction for multiple comparisons (p = 0.048, paired speaker-cluster bootstrap, 985 clips / 280 speakers).
Training-set size vs. speaker-disjointness. TIE_shorts' official test speakers are so
entangled with train that removing them keeps only 567 of 7,200 train clips (3.8 of 46.9 hours;
51 of 331 speakers), so this run differs from the official-split fine-tune in both speaker
overlap and training-set size (~13× smaller). A size-matched speaker-overlapping control (same
567-clip budget, 3 seeds, speaker overlap preserved) isolates the two effects: all 3 control seeds
land flat at 14.33–14.40%, indistinguishable from the 14.42% pretrained baseline, no significant
effect in any seed. Since the control trains on the same ~567-clip budget as the disjoint runs
below (the shared no-embedded-audio filter can drop a clip or two per seed; e.g. this seed-42 run
trains on 566), a one-clip difference cannot explain the disjoint runs' regression. Speaker-disjointness
is the cause: removing speaker overlap exposes that part of the official split's fine-tuning
signal was speaker memorization, not genuine acoustic/language adaptation. Full breakdown in the
GitHub repo's finetune_comparison.md.
This project trained the same recipe with 3 seeds to check whether a null/negative result from one run was just seed variance. It wasn't uniform:
| Seed | WER (transcript_clean) | Δ vs pretrained | Holm-adjusted p |
|---|---|---|---|
| 42 (this checkpoint) | 16.17% | +1.75 pp | 0.048 (significant) |
| 43 | 14.80% | +0.38 pp | 0.116 (not significant) |
| 44 | 15.20% | +0.79 pp | 0.163 (not significant) |
Only this seed (42) reaches significance; seeds 43/44 fall within the study's minimum detectable effect (≈1.2pp) and are indistinguishable from no effect. The seed-to-seed spread (1.37pp) is itself larger than the effect being measured. The defensible claim from the full study is: fine-tuning on the speaker-disjoint training subset (567 clips) shows no evidence of improving WER, and at least one seed (this one) shows evidence of making it measurably worse. Per the size-matched control above, that regression traces to speaker-disjointness itself, not to the 13×-smaller training set. Do not treat this checkpoint's 16.17% as "the" expected WER of speaker-disjoint fine-tuning: treat it as one of three seed outcomes, published for exact reproducibility of that specific result.
Full methodology, all 3 seeds' raw transcripts, the size-matched control, and the statistical analysis: indian-asr-bench.
This is a research artifact documenting an evaluation-validity and fine-tuning-stability finding, not a recommended checkpoint for transcription. If you need Indian-English Whisper inference, plain openai/whisper-medium performs at least as well per this study, with no fine-tuning risk.
train split of raianand/TIE_shorts, with every speaker present in test removed:
567 clips / 3.8 hours / 51 speakers after the standard ≤30s/non-empty filters (the official
split keeps 7,200 clips / 46.9 hours / 331 speakers; this seed's run trains on 566 after the
no-embedded-audio filter).test split of raianand/TIE_shorts (986 clips): identical set used for every other
model in the benchmark, so results are directly comparable.Transcript field (English, transcribe task).Apache 2.0, matching the base model. The TIE_shorts dataset carries its own license.