thorhojhus

whisper-small-danish

Usage

python
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

model_id = "thorhojhus/whisper-small-danish"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, dtype=dtype, low_cpu_mem_usage=True,
).to(device)

transcribe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    dtype=dtype,
    device=device,
)
result = transcribe(
    "audio.wav",
    generate_kwargs={"language": "da", "task": "transcribe", "num_beams": 1},
)
print(result["text"])

Training

Training was started from stock Whisper Small. Filtered pseudo-labelled Danish speech was combined with the Danish portions of CoRal v3, Common Voice 26.0, FLEURS, FTSpeech and NST. Pseudo-labels were generated with whisper-large-v3-turbo-danish. The retained pool contained approximately 13,411 unique audio hours. Labels were filtered by confidence, no-speech probability, hallucination rules and a Danish/English vocabulary check.

Pseudo-labelled speech accounted for 100% of the sampling weight through step 50,000, 80% through step 140,000 and 70% thereafter. Human-labelled sources made up the remainder. Targets retained their casing, punctuation and numeral style; normalization was applied only for scoring.

The use of weak supervision and no-speech examples was inspired by the original Whisper paper. Waveform augmentation, SpecAugment and synthetic no-speech examples were included.

Table with columns: Setting, Value
Setting	Value
Steps	200,000
Optimizer	Muon for hidden matrices; AdamW for remaining parameters
Peak learning rate	Muon `5e-5`; AdamW `2.5e-5`
Weight decay	`0.01`
Schedule	1,000-step warmup, stable to 50,000, linear decay to 10% at 200,000
EMA	Per-step decay `0.9999`, applied every 50 steps
Precision	BF16 parameters with compensated updates; FP32 EMA
Maximum training segment	30 seconds

The following figures are cumulative audio sampled on the final training trajectory, including repeated passes. They are not unique dataset durations.

Table with columns: Source, Hours seen
Source	Hours seen
CoRal conversation	1,151.29
CoRal read-aloud	2,584.61
Common Voice Danish	139.94
FLEURS Danish	140.21
FTSpeech	4,182.40
NST Danish	1,938.15
Pseudo-labelled data	44,344.90
Total	54,481.50

Additional settings and provenance are recorded in training_recipe.json.

Evaluation

The checkpoint was evaluated using the public Danish ASR Leaderboard protocol with Transformers FP16 greedy Danish transcription. Standard decoding was used for short clips; timestamped sequential decoding was used above 30 seconds. Scoring included the leaderboard's Danish numeral normalization and did not remove filler words.

Table with columns: Test set, Samples, WER (%), CER (%)
Test set	Samples	WER (%)	CER (%)
CoRal conversation	8,438	24.01	15.02
CoRal read-aloud	9,122	14.16	5.73
Common Voice (`cv17_da`)	2,756	9.53	3.65
FLEURS `da_dk`	930

Limitations

Performance outside Danish has not been validated after fine-tuning.
Long-form behavior was not directly trained and should be checked for the intended use.
Silence, noise and unfamiliar names can still produce incorrect text.

License

The model weights are released under the MIT License. Training data are not redistributed and remain subject to their source terms. Attribution and the upstream Whisper notice are included in THIRD_PARTY_NOTICES.md.

Citation

bibtex
@article{radford2022robust,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2022}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

thorhojhus

Model Tree

Base

openai/whisper-small

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Usage

python
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

model_id = "thorhojhus/whisper-small-danish"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, dtype=dtype, low_cpu_mem_usage=True,
).to(device)

transcribe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    dtype=dtype,
    device=device,
)
result = transcribe(
    "audio.wav",
    generate_kwargs={"language": "da", "task": "transcribe", "num_beams": 1},
)
print(result["text"])

Training

The use of weak supervision and no-speech examples was inspired by the original Whisper paper. Waveform augmentation, SpecAugment and synthetic no-speech examples were included.

Table with columns: Setting, Value
Setting	Value
Steps	200,000
Optimizer	Muon for hidden matrices; AdamW for remaining parameters
Peak learning rate	Muon `5e-5`; AdamW `2.5e-5`
Weight decay	`0.01`
Schedule	1,000-step warmup, stable to 50,000, linear decay to 10% at 200,000
EMA	Per-step decay `0.9999`, applied every 50 steps
Precision	BF16 parameters with compensated updates; FP32 EMA
Maximum training segment	30 seconds

The following figures are cumulative audio sampled on the final training trajectory, including repeated passes. They are not unique dataset durations.

Table with columns: Source, Hours seen
Source	Hours seen
CoRal conversation	1,151.29
CoRal read-aloud	2,584.61
Common Voice Danish	139.94
FLEURS Danish	140.21
FTSpeech	4,182.40
NST Danish	1,938.15
Pseudo-labelled data	44,344.90
Total	54,481.50

Additional settings and provenance are recorded in training_recipe.json.

Evaluation

Table with columns: Test set, Samples, WER (%), CER (%)
Test set	Samples	WER (%)	CER (%)
CoRal conversation	8,438	24.01	15.02
CoRal read-aloud	9,122	14.16	5.73
Common Voice (`cv17_da`)	2,756	9.53	3.65
FLEURS `da_dk`	930

Limitations

Performance outside Danish has not been validated after fine-tuning.
Long-form behavior was not directly trained and should be checked for the intended use.
Silence, noise and unfamiliar names can still produce incorrect text.

License

Citation

bibtex
@article{radford2022robust,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2022}
}

whisper-small-danish

README

Usage

Training

Evaluation

Limitations

License

Citation

Explore FriendliAI today

README

Usage

Training

Evaluation

Limitations

License

Citation