MosesJoshuaCoker

Novax_Music_Lyrics_Krio

Deploy Dedicated

README

License: apache-2.0

Model Details

Model Description

Developed by: Moses Joshua Coker
Model type: Encoder–decoder speech-to-text transformer (Whisper architecture, ~242M parameters)
Language(s): Krio (kri)
License: Apache 2.0 (inherited from the base model)
Finetuned from model: openai/whisper-small

Model Sources

Repository: this Hub repo
Training data: MosesJoshuaCoker/30_hours_krio_voice

Uses

Direct Use

Transcribing Krio speech and sung/musical vocals to Krio text: song lyrics, radio, interviews, voice notes.

Out-of-Scope Use

Transcribing languages other than Krio — including English. The fine-tune repurposed the <|en|> token, so English audio will come out "Krio-flavored".
Translation (the translate task was not trained).
Timestamped output was not part of training and may be unreliable.

Bias, Risks, and Limitations

30-second window: Whisper processes audio in fixed 30 s chunks. Clips longer than 30 s are truncated unless you use chunking (see the long-audio example below).
Music is hard: heavy instrumentation, overlapping vocals, and autotune degrade accuracy relative to clean speech.
Single-source data: the model reflects the speakers, recording conditions, and orthography conventions of the training dataset; unusual accents, noisy phone recordings, or different Krio spelling conventions may see higher error rates.
Hallucination: like all Whisper models, it can produce fluent but wrong text on silence, music-only passages, or very noisy audio.

Recommendations

For full songs, use the chunked pipeline (below). Spot-check transcriptions before downstream use, especially on instrumental-heavy sections.

How to Get Started with the Model

Quick test (pipeline)

The forced Krio decoding is baked into generation_config, so no language arguments are needed:

python
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="MosesJoshuaCoker/Novax_Music_Lyrics_Krio",
    device=0,  # use device=-1 for CPU
)

print(asr("my_krio_clip.wav")["text"])

Any format ffmpeg can read works (wav, mp3, m4a, ...).

Full songs / audio longer than 30 seconds

python
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="MosesJoshuaCoker/Novax_Music_Lyrics_Krio",
    chunk_length_s=30,
    device=0,
)

print(asr("full_song.mp3")["text"])

Manual usage (processor + model)

python
import librosa
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model_id = "MosesJoshuaCoker/Novax_Music_Lyrics_Krio"
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id).to("cuda").eval()

audio, _ = librosa.load("my_krio_clip.wav", sr=16000)
features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")

with torch.no_grad():
    # language="en" is the Krio anchor token — see the warning at the top.
    ids = model.generate(features, language="en", task="transcribe", max_new_tokens=225)

print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Reproduce the evaluation (WER on the held-out split)

Training held out 5% of the dataset (seed 42, after dropping empty transcriptions). This rebuilds that exact split and scores the model on it:

python
import evaluate
import torch
from datasets import Audio, load_dataset
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model_id = "MosesJoshuaCoker/Novax_Music_Lyrics_Krio"
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id).to("cuda").eval()

ds = load_dataset("MosesJoshuaCoker/30_hours_krio_voice", split="train")
ds = ds.filter(lambda t: t is not None and len(t.strip()) > 0, input_columns=["transcriptions"])
ds = ds.cast_column("audio_path", Audio(sampling_rate=16000))
eval_ds = ds.train_test_split(test_size=0.05, seed=42)["test"]

wer = evaluate.load("wer")
preds, refs = [], []
for i in range(0, len(eval_ds), 8):
    batch = eval_ds[i : i + 8]
    audios = [a["array"] for a in batch["audio_path"]]
    features = processor(audios, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")
    with torch.no_grad():
        ids = model.generate(features, language="en", task="transcribe", max_new_tokens=225)
    preds += processor.batch_decode(ids, skip_special_tokens=True)
    refs += batch["transcriptions"]

print(f"WER: {100 * wer.compute(predictions=preds, references=refs):.2f}%")

Note: this split is only truly "unseen" for checkpoints trained with the same filter and seed as the training notebook.

Training Details

Training Data

MosesJoshuaCoker/30_hours_krio_voice: 6,807 clips (~30 h) with columns audio_path (16 kHz audio) and transcriptions (Krio text). Empty transcriptions were dropped, then split 95/5 train/eval with seed 42.

Training Procedure

Preprocessing

Audio resampled to 16 kHz; log-mel features computed on the fly, padded/truncated to Whisper's 30 s window.
Labels tokenized with the Whisper tokenizer configured with language="en" (the Krio anchor) and task="transcribe", truncated to 448 tokens.
Trained with distributed data parallel (DDP) across 2 GPUs via torchrun.

Training Hyperparameters

Training regime: fp16 mixed precision
Effective batch size: 32 (16 per GPU × 2 GPUs)
Learning rate: 1e-5, linear schedule with 10% warmup
Epochs: up to 8, with early stopping on eval WER (patience 3, eval every 200 steps)
Gradient checkpointing: enabled
Selection: best checkpoint by lowest eval WER

Speeds, Sizes, Times

~242M parameters, ~967 MB checkpoint (fp32)
Roughly 1.5–2.5 h wall-clock on the hardware below

Evaluation

Testing Data, Factors & Metrics

Testing data: the held-out 5% split of the training dataset (see the reproduction snippet above).
Metrics: Word Error Rate (WER), plus a case/punctuation-normalized WER — WER is the standard ASR metric and Krio orthography varies, so the normalized number is often the fairer one.

Results

Table with columns: Metric, Score
Metric	Score
WER	TODO %
WER (normalized)	TODO %

Environmental Impact

Hardware Type: 2× NVIDIA T4
Hours used: ~2
Cloud Provider: Kaggle
Carbon Emitted: can be estimated with the ML Impact calculator

Technical Specifications

Model Architecture and Objective

Whisper Small: a sequence-to-sequence transformer trained with a next-token prediction objective on (log-mel spectrogram → text) pairs. This fine-tune keeps the architecture unchanged and updates all weights.

Compute Infrastructure

Hardware: Kaggle GPU T4 ×2 instance
Software: transformers (Seq2SeqTrainer), datasets <4.0, evaluate/jiwer, PyTorch with torchrun DDP

Citation

If you use this model, please cite the dataset and the Whisper paper:

BibTeX:

bibtex
@misc{radford2022whisper,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  year={2022},
  eprint={2212.04356},
  archivePrefix={arXiv},
  primaryClass={eess.AS}
}

Model Card Contact

Moses Joshua Coker — via the Hugging Face profile

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

MosesJoshuaCoker

Model Tree

Base

openai/whisper-small

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Model Description

Developed by: Moses Joshua Coker
Model type: Encoder–decoder speech-to-text transformer (Whisper architecture, ~242M parameters)
Language(s): Krio (kri)
License: Apache 2.0 (inherited from the base model)
Finetuned from model: openai/whisper-small

Model Sources

Repository: this Hub repo
Training data: MosesJoshuaCoker/30_hours_krio_voice

Uses

Direct Use

Transcribing Krio speech and sung/musical vocals to Krio text: song lyrics, radio, interviews, voice notes.

Out-of-Scope Use

Transcribing languages other than Krio — including English. The fine-tune repurposed the <|en|> token, so English audio will come out "Krio-flavored".
Translation (the translate task was not trained).
Timestamped output was not part of training and may be unreliable.

Bias, Risks, and Limitations

30-second window: Whisper processes audio in fixed 30 s chunks. Clips longer than 30 s are truncated unless you use chunking (see the long-audio example below).
Music is hard: heavy instrumentation, overlapping vocals, and autotune degrade accuracy relative to clean speech.
Single-source data: the model reflects the speakers, recording conditions, and orthography conventions of the training dataset; unusual accents, noisy phone recordings, or different Krio spelling conventions may see higher error rates.
Hallucination: like all Whisper models, it can produce fluent but wrong text on silence, music-only passages, or very noisy audio.

Recommendations

For full songs, use the chunked pipeline (below). Spot-check transcriptions before downstream use, especially on instrumental-heavy sections.

How to Get Started with the Model

Quick test (pipeline)

The forced Krio decoding is baked into generation_config, so no language arguments are needed:

python
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="MosesJoshuaCoker/Novax_Music_Lyrics_Krio",
    device=0,  # use device=-1 for CPU
)

print(asr("my_krio_clip.wav")["text"])

Any format ffmpeg can read works (wav, mp3, m4a, ...).

Full songs / audio longer than 30 seconds

python
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="MosesJoshuaCoker/Novax_Music_Lyrics_Krio",
    chunk_length_s=30,
    device=0,
)

print(asr("full_song.mp3")["text"])

Manual usage (processor + model)

python
import librosa
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model_id = "MosesJoshuaCoker/Novax_Music_Lyrics_Krio"
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id).to("cuda").eval()

audio, _ = librosa.load("my_krio_clip.wav", sr=16000)
features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")

with torch.no_grad():
    # language="en" is the Krio anchor token — see the warning at the top.
    ids = model.generate(features, language="en", task="transcribe", max_new_tokens=225)

print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Reproduce the evaluation (WER on the held-out split)

Training held out 5% of the dataset (seed 42, after dropping empty transcriptions). This rebuilds that exact split and scores the model on it:

python
import evaluate
import torch
from datasets import Audio, load_dataset
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model_id = "MosesJoshuaCoker/Novax_Music_Lyrics_Krio"
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id).to("cuda").eval()

ds = load_dataset("MosesJoshuaCoker/30_hours_krio_voice", split="train")
ds = ds.filter(lambda t: t is not None and len(t.strip()) > 0, input_columns=["transcriptions"])
ds = ds.cast_column("audio_path", Audio(sampling_rate=16000))
eval_ds = ds.train_test_split(test_size=0.05, seed=42)["test"]

wer = evaluate.load("wer")
preds, refs = [], []
for i in range(0, len(eval_ds), 8):
    batch = eval_ds[i : i + 8]
    audios = [a["array"] for a in batch["audio_path"]]
    features = processor(audios, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")
    with torch.no_grad():
        ids = model.generate(features, language="en", task="transcribe", max_new_tokens=225)
    preds += processor.batch_decode(ids, skip_special_tokens=True)
    refs += batch["transcriptions"]

print(f"WER: {100 * wer.compute(predictions=preds, references=refs):.2f}%")

Note: this split is only truly "unseen" for checkpoints trained with the same filter and seed as the training notebook.

Training Details

Training Data

Training Procedure

Preprocessing

Audio resampled to 16 kHz; log-mel features computed on the fly, padded/truncated to Whisper's 30 s window.
Labels tokenized with the Whisper tokenizer configured with language="en" (the Krio anchor) and task="transcribe", truncated to 448 tokens.
Trained with distributed data parallel (DDP) across 2 GPUs via torchrun.

Training Hyperparameters

Training regime: fp16 mixed precision
Effective batch size: 32 (16 per GPU × 2 GPUs)
Learning rate: 1e-5, linear schedule with 10% warmup
Epochs: up to 8, with early stopping on eval WER (patience 3, eval every 200 steps)
Gradient checkpointing: enabled
Selection: best checkpoint by lowest eval WER

Speeds, Sizes, Times

~242M parameters, ~967 MB checkpoint (fp32)
Roughly 1.5–2.5 h wall-clock on the hardware below

Evaluation

Testing Data, Factors & Metrics

Testing data: the held-out 5% split of the training dataset (see the reproduction snippet above).
Metrics: Word Error Rate (WER), plus a case/punctuation-normalized WER — WER is the standard ASR metric and Krio orthography varies, so the normalized number is often the fairer one.

Results

Table with columns: Metric, Score
Metric	Score
WER	TODO %
WER (normalized)	TODO %

Environmental Impact

Hardware Type: 2× NVIDIA T4
Hours used: ~2
Cloud Provider: Kaggle
Carbon Emitted: can be estimated with the ML Impact calculator

Technical Specifications

Model Architecture and Objective

Compute Infrastructure

Hardware: Kaggle GPU T4 ×2 instance
Software: transformers (Seq2SeqTrainer), datasets <4.0, evaluate/jiwer, PyTorch with torchrun DDP

Citation

If you use this model, please cite the dataset and the Whisper paper:

BibTeX:

bibtex
@misc{radford2022whisper,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  year={2022},
  eprint={2212.04356},
  archivePrefix={arXiv},
  primaryClass={eess.AS}
}

Model Card Contact

Moses Joshua Coker — via the Hugging Face profile