dylan01163104

whisper-medium-dora-mix6

README

License: apache-2.0

Usage

python
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel

processor = WhisperProcessor.from_pretrained(
    "openai/whisper-medium",
    revision="abdf7c39ab9d0397620ccaea8974cc764cd0953e",
)
base = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-medium",
    revision="abdf7c39ab9d0397620ccaea8974cc764cd0953e",
    torch_dtype="float32",
)
model = PeftModel.from_pretrained(base, "dylan01163104/whisper-medium-dora-mix6")
model.eval()
# ...standard Whisper inference from here

Training

Base model: openai/whisper-medium revision abdf7c39ab9d0397620ccaea8974cc764cd0953e
Method: DoRA (Weight-Decomposed LoRA), peft==0.19.1
Adapter target: decoder self_attn + encoder_attn q_proj, k_proj, v_proj, out_proj (all 24 layers). Encoder frozen.
Rank: 32
Alpha: 64
Data: FLEURS revision d7c758a6dceecd54a98cac43404d3d576e721f07, 6 target languages + 10% English anchor. Temperature-sampling T=0.5.
Steps: 2000 · LR: 1e-5 · Warmup: 200 steps · Batch: 8 × 2 grad_accum × 2 GPUs = effective 32
Hardware: 2× consumer GPU, DDP via torchrun
Seed: 42

Full training script: tmm_asr/train/dora.py.

Results (WER, mean of 264 FLEURS test clips)

Table with columns: Language, Base whisper-medium, + DoRA-mix6, Δ
Language	Base whisper-medium	+ DoRA-mix6	Δ
Lingala	83.2	52.2	−31.0
Javanese	60.7	44.7	−16.0
Hausa	66.5	53.0	−13.5
Maltese	51.9	46.0

Token merging on top of this adapter costs at most +0.59 pp WER at TRR = 0.40 (Lingala). See paper §5.3.

Generalisation to held-out languages

Applied to 10 languages the adapter was NOT trained on (English, French, German, Spanish, Thai, Swahili, Afrikaans, Icelandic, Welsh, Kazakh):

Anchors preserved (+0.02 pp mean drift from base)
Untrained mid/low-res drift +1.77 pp mean, max +3.78 pp (Kazakh)
Merging still cheap: mean +0.23 pp, max +0.97 pp (Icelandic)

Full held-out table in paper §5.3 ¶3.

Limitations

Encoder unchanged, so any encoder-language-code mismatch (e.g. Whisper's jw for Javanese) still needs to be handled at the processor level.
Wall-clock latency at batch=1 does NOT improve with encoder-side merging on modern GPUs — the merge op overhead dominates the compute saved. See paper Limitations.
Trained only on FLEURS narrow-band read speech. Domain shift to noisy / conversational speech not evaluated.

Citation

bibtex
@inproceedings{holyoak2026tokenmerging,
  title     = {Token Merging for Multilingual Speech Recognition:
               A Systematic Study Across Model Scale and Fine-Tuning},
  author    = {Holyoak, Dylan Luke},
  booktitle = {Proceedings of ICNLSP 2026},
  year      = {2026},
  address   = {University of California, Los Angeles}
}

License

Apache-2.0. Base Whisper checkpoint carries its own upstream MIT license.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

dylan01163104

Model Tree

Base

openai/whisper-medium

Adapter

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer