tonibirat

sagarmatha-v6-nepali-asr

README

License: apache-2.0

Overview

Sagarmatha V6 is a state-of-the-art Automatic Speech Recognition (ASR) foundation model specifically engineered for the Nepali language and its complex dialectal, conversational, and acoustic realities.

Built on OpenAI's whisper-large-v3-turbo architecture, Sagarmatha V6 was trained on 1,285.4 hours of multi-channel Nepali audio (280,376 time-aligned windows) spanning broadcast news, conversational podcasts, television interviews, educational lectures, and field recordings across Nepal. The model features:

Extended Devanagari Vocabulary (+2,000 tokens): Drastically reduces sequence lengths and byte-fallback rates on Devanagari script, mitigating repetition loops and boosting inference throughput.
Controllable Multi-Mode Decoding: Dedicated decoder prefix tokens allowing users to dynamically select Script (all Devanagari vs mixed Latin/Devanagari) and Style (clean editorial vs verbatim with vocables).
Weight Averaged Soup ( $η = 0.80$ ): Linearly interpolated with Sagarmatha V5 to prevent catastrophic forgetting and preserve robust out-of-distribution acoustic generalisation.
Dual Native Formats: Includes standard PyTorch weights (model.safetensors, 3.25 GB) and pre-converted float16 CTranslate2 weights (ct2-fp16/model.bin, 1.62 GB) delivering over 52× real-time throughput on modern GPUs.

Benchmark Results

All benchmarks are evaluated on Raw WER and Raw CER (standard Unicode NFC normalization with punctuation stripped, no artificial orthographic folding). 95% bootstrap confidence intervals are computed over clips (2,000 resamples).

Zero Leakage Guarantee: Rigorous n-gram cross-matching against the training corpus verifies that 0 benchmark sentences in FLEURS or Common Voice appear in the 1,285.4-hour training dataset.

1. Public Benchmarks (`deva+clean` mode)

Table with columns: Benchmark, Split, Clips, Duration, Raw WER [95% CI], Raw CER [95% CI], vs Sagarmatha V5
Benchmark	Split	Clips	Duration	Raw WER [95% CI]	Raw CER [95% CI]	vs Sagarmatha V5
Google FLEURS	`ne_np` test	724	2.27 h	23.84 [22.76, 25.07]	8.35 [7.78, 8.94]	-3.69 [-4.39, -2.98]
Common Voice 17	`ne-NP` test	217	0.23 h	23.40 [20.41, 26.43]	5.58 [4.71, 6.49]	-6.63 [-8.90, -4.43]
Common Voice 17	`ne-NP` validated	704	0.74 h	24.50 [22.76, 26.22]	6.12 [5.50, 6.78]	-5.04 [-6.28, -3.84]
In-Domain Channels	Unseen host / video	1,486	11.2 h	15.50 [14.88, 16.14]	4.92 [4.58, 5.28]	-3.12 [-3.80, -2.45]

FLEURS with digits spelled as words: 20.71 WER on the 588 digit-free reference subset.

2. Comparison with Contemporary Systems

Table with columns: System, Parameters, Training Recipe, FLEURS WER, Common Voice Test WER, Note
System	Parameters	Training Recipe	FLEURS WER	Common Voice Test WER	Note
Sagarmatha V6 (Ours)	809M	1,285 h Multi-channel + Soup	23.84	23.40	State of the Art on Conversational & Read Nepali
Sagarmatha V5	809M	193 h In-domain	27.53	30.03	Predecessor base model
Meta Omnilingual ASR CTC	1B	Massive Multilingual	31.56	39.42	Paired: V6 outperforms by 9.76 WER [8.75, 10.75]
Meta Omnilingual ASR LLM	7B	Massive Multilingual	20.59	45.87	V6 outperforms 7B by 22.46 WER on Common Voice test
Best OpenSLR Fine-tune	809M	OpenSLR-54 (arXiv:2608.12327)	39.56	48.35	V6 is 15.7 to 24.9 WER points better
Whisper Medium Fine-tune	769M	OpenSLR-54 (arXiv:2608.12327)	39.06	48.98	V6 is 15.2 to 25.5 WER points better
MMS-1B (Meta)	1B	Indic Fine-tune (OpenSLR-54)	39.83	58.65	V6 is 16.0 to 35.2 WER points better

Controllable Mode Tokens

Sagarmatha V6 introduces four steering prefix tokens placed immediately following <|startoftranscript|>:

text

[<|startoftranscript|>, <SCRIPT_TOKEN>, <STYLE_TOKEN>, <|ne|>, <|transcribe|>, <|notimestamps|>]

Table with columns: Token, ID, Role, Description
Token	ID	Role	Description
`<	deva	>`	`53866`
`<	mixed	>`	`53867`
`<	clean	>`	`53869`
`<	verbatim	>`	`53868`

Recommended Production Prefix

For general-purpose production, subtitles, voice typing, and business transcription, use deva+clean:

Token IDs: [53866, 53869, 50313, 50360, 52364]

Usage & Code Examples

1. Fast Inference with CTranslate2 (Recommended for Production)

CTranslate2 delivers 52× real-time processing with low memory consumption:

python

import ctranslate2from transformers import WhisperProcessorimport librosa # 1. Load processor and CTranslate2 modelprocessor = WhisperProcessor.from_pretrained("tonibirat/sagarmatha-v6-nepali-asr")model = ctranslate2.models.Whisper(    "tonibirat/sagarmatha-v6-nepali-asr",    subfolder="ct2-fp16",    device="cuda",  # or "cpu"    compute_type="float16"  # or "int8_float16" on CUDA, "int8" on CPU) # 2. Load audio (resampled to 16 kHz mono)audio, sr = librosa.load("nepali_speech.wav", sr=16000) # 3. Extract log-mel filterbank featuresinputs = processor(audio, sampling_rate=16000, return_tensors="np")features = ctranslate2.StorageView.from_array(inputs.input_features) # 4. Define mode prompt: <|deva|><|clean|><|ne|><|transcribe|><|notimestamps|># IDs: 53866 (<|deva|>), 53869 (<|clean|>), 50313 (<|ne|>), 50360 (<|transcribe|>), 52364 (<|notimestamps|>)prompt_tokens = [53866, 53869, 50313, 50360, 52364] # IMPORTANT: Always set max_length=896 to prevent Devanagari byte fallback cutoffresults = model.generate(    features,    [prompt_tokens],    max_length=896,    repetition_penalty=1.1,    no_repeat_ngram_size=6) # 5. Decode predicted token IDs to Nepali textoutput_ids = results[0].sequences_ids[0]transcript = processor.tokenizer.decode(output_ids, skip_special_tokens=True)print("Transcript:", transcript)

2. Standard Inference with Hugging Face Transformers

python

import torchfrom transformers import WhisperForConditionalGeneration, WhisperProcessorimport librosa device = "cuda" if torch.cuda.is_available() else "cpu"torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32 # 1. Load model and processorprocessor = WhisperProcessor.from_pretrained("tonibirat/sagarmatha-v6-nepali-asr")model = WhisperForConditionalGeneration.from_pretrained(    "tonibirat/sagarmatha-v6-nepali-asr",    torch_dtype=torch_dtype).to(device) # 2. Load and process audioaudio, sr = librosa.load("nepali_speech.wav", sr=16000)inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(device, dtype=torch_dtype) # 3. Force mode tokens in decoder: deva+clean# <|deva|> (53866), <|clean|> (53869), <|ne|> (50313), <|transcribe|> (50360), <|notimestamps|> (52364)forced_prefix_ids = [    (1, 53866),  # <|deva|>    (2, 53869),  # <|clean|>    (3, 50313),  # <|ne|>    (4, 50360),  # <|transcribe|>    (5, 52364),  # <|notimestamps|>] # 4. Generatewith torch.no_grad():    predicted_ids = model.generate(        inputs.input_features,        forced_decoder_ids=forced_prefix_ids,        max_length=448    ) transcript = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]print("Transcript:", transcript)

Critical Engineering Best Practices

1. Token Budget & Context Limit (`max_length = 896`)

In Whisper, each 30-second window is allotted a maximum context length of 448 positions. Under stock byte fallback or high-density Devanagari text, a 25-30 second Nepali audio window can generate up to 350+ tokens. When using CTranslate2, OpenAI's default implementation caps output at max_length // 2 (224 tokens), which cuts off 96.8% of long Nepali utterances. Always supply max_length = 896 in CTranslate2.

2. Voice Activity Detection (VAD) Gating

Like all sequence-to-sequence audio models, unconstrained decoding on long silent intervals or background music tracks can induce hallucinated text. For long-form audio or live streaming, always pair Sagarmatha V6 with a pre-segmentation VAD filter such as Silero VAD v5 (threshold=0.5). Gating removes over 95% of non-speech hallucination windows on challenging environmental audio (ESC-50).

3. Caution Regarding Whisper's `BasicTextNormalizer`

Do NOT use Whisper's stock BasicTextNormalizer when evaluating Devanagari ASR models. Its regex patterns treat Devanagari combining vowel signs (matras) and viramas as standalone whitespace/punctuation, exploding single Nepali words into 2.49× more sub-units and distorting WER results (spuriously inflating scores down to ~11.5%). Benchmark using standard Unicode NFC normalization with punctuation removal.

Training Details

Base Architecture: openai/whisper-large-v3-turbo (809M parameters, 4 decoder layers, 32 encoder layers).
Corpus: tonibirat/nepali-asr-v6 — 1,285.4 hours (280,376 audio windows) with word-level forced alignment.
Hardware: NVIDIA RTX 5090 (32 GB VRAM).
Optimizer & Schedule: AdamW, cosine learning rate decay with linear warmup, fp16 mixed precision.
Soup Interpolation: $θ final = 0.80 \times θ V6 + 0.20 \times θ V5 $ .

Citation & Contact

If you use Sagarmatha V6 in your research or application, please cite:

bibtex

@misc{sagarmatha2026v6,  author = {Basyal, Toni and contributors},  title = {Sagarmatha V6: Controllable State-of-the-Art Speech Recognition for Nepali},  year = {2026},  publisher = {Hugging Face},  howpublished = {\url{https://huggingface.co/tonibirat/sagarmatha-v6-nepali-asr}}}

License

This model is licensed under the Apache License 2.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

tonibirat

Model Tree

Base

openai/whisper-large-v3-turbo

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Overview

Extended Devanagari Vocabulary (+2,000 tokens): Drastically reduces sequence lengths and byte-fallback rates on Devanagari script, mitigating repetition loops and boosting inference throughput.
Controllable Multi-Mode Decoding: Dedicated decoder prefix tokens allowing users to dynamically select Script (all Devanagari vs mixed Latin/Devanagari) and Style (clean editorial vs verbatim with vocables).
Weight Averaged Soup ( $η = 0.80$ ): Linearly interpolated with Sagarmatha V5 to prevent catastrophic forgetting and preserve robust out-of-distribution acoustic generalisation.
Dual Native Formats: Includes standard PyTorch weights (model.safetensors, 3.25 GB) and pre-converted float16 CTranslate2 weights (ct2-fp16/model.bin, 1.62 GB) delivering over 52× real-time throughput on modern GPUs.

Benchmark Results

Zero Leakage Guarantee: Rigorous n-gram cross-matching against the training corpus verifies that 0 benchmark sentences in FLEURS or Common Voice appear in the 1,285.4-hour training dataset.

1. Public Benchmarks (`deva+clean` mode)

Table with columns: Benchmark, Split, Clips, Duration, Raw WER [95% CI], Raw CER [95% CI], vs Sagarmatha V5
Benchmark	Split	Clips	Duration	Raw WER [95% CI]	Raw CER [95% CI]	vs Sagarmatha V5
Google FLEURS	`ne_np` test	724	2.27 h	23.84 [22.76, 25.07]	8.35 [7.78, 8.94]	-3.69 [-4.39, -2.98]
Common Voice 17	`ne-NP` test	217	0.23 h	23.40 [20.41, 26.43]	5.58 [4.71, 6.49]	-6.63 [-8.90, -4.43]
Common Voice 17	`ne-NP` validated	704	0.74 h	24.50 [22.76, 26.22]	6.12 [5.50, 6.78]	-5.04 [-6.28, -3.84]
In-Domain Channels	Unseen host / video	1,486	11.2 h	15.50 [14.88, 16.14]	4.92 [4.58, 5.28]	-3.12 [-3.80, -2.45]

FLEURS with digits spelled as words: 20.71 WER on the 588 digit-free reference subset.

2. Comparison with Contemporary Systems

Table with columns: System, Parameters, Training Recipe, FLEURS WER, Common Voice Test WER, Note
System	Parameters	Training Recipe	FLEURS WER	Common Voice Test WER	Note
Sagarmatha V6 (Ours)	809M	1,285 h Multi-channel + Soup	23.84	23.40	State of the Art on Conversational & Read Nepali
Sagarmatha V5	809M	193 h In-domain	27.53	30.03	Predecessor base model
Meta Omnilingual ASR CTC	1B	Massive Multilingual	31.56	39.42	Paired: V6 outperforms by 9.76 WER [8.75, 10.75]
Meta Omnilingual ASR LLM	7B	Massive Multilingual	20.59	45.87	V6 outperforms 7B by 22.46 WER on Common Voice test
Best OpenSLR Fine-tune	809M	OpenSLR-54 (arXiv:2608.12327)	39.56	48.35	V6 is 15.7 to 24.9 WER points better
Whisper Medium Fine-tune	769M	OpenSLR-54 (arXiv:2608.12327)	39.06	48.98	V6 is 15.2 to 25.5 WER points better
MMS-1B (Meta)	1B	Indic Fine-tune (OpenSLR-54)	39.83	58.65	V6 is 16.0 to 35.2 WER points better

Controllable Mode Tokens

Sagarmatha V6 introduces four steering prefix tokens placed immediately following <|startoftranscript|>:

text

[<|startoftranscript|>, <SCRIPT_TOKEN>, <STYLE_TOKEN>, <|ne|>, <|transcribe|>, <|notimestamps|>]

Table with columns: Token, ID, Role, Description
Token	ID	Role	Description
`<	deva	>`	`53866`
`<	mixed	>`	`53867`
`<	clean	>`	`53869`
`<	verbatim	>`	`53868`

Recommended Production Prefix

For general-purpose production, subtitles, voice typing, and business transcription, use deva+clean:

Token IDs: [53866, 53869, 50313, 50360, 52364]

Usage & Code Examples

1. Fast Inference with CTranslate2 (Recommended for Production)

CTranslate2 delivers 52× real-time processing with low memory consumption:

python

import ctranslate2from transformers import WhisperProcessorimport librosa # 1. Load processor and CTranslate2 modelprocessor = WhisperProcessor.from_pretrained("tonibirat/sagarmatha-v6-nepali-asr")model = ctranslate2.models.Whisper(    "tonibirat/sagarmatha-v6-nepali-asr",    subfolder="ct2-fp16",    device="cuda",  # or "cpu"    compute_type="float16"  # or "int8_float16" on CUDA, "int8" on CPU) # 2. Load audio (resampled to 16 kHz mono)audio, sr = librosa.load("nepali_speech.wav", sr=16000) # 3. Extract log-mel filterbank featuresinputs = processor(audio, sampling_rate=16000, return_tensors="np")features = ctranslate2.StorageView.from_array(inputs.input_features) # 4. Define mode prompt: <|deva|><|clean|><|ne|><|transcribe|><|notimestamps|># IDs: 53866 (<|deva|>), 53869 (<|clean|>), 50313 (<|ne|>), 50360 (<|transcribe|>), 52364 (<|notimestamps|>)prompt_tokens = [53866, 53869, 50313, 50360, 52364] # IMPORTANT: Always set max_length=896 to prevent Devanagari byte fallback cutoffresults = model.generate(    features,    [prompt_tokens],    max_length=896,    repetition_penalty=1.1,    no_repeat_ngram_size=6) # 5. Decode predicted token IDs to Nepali textoutput_ids = results[0].sequences_ids[0]transcript = processor.tokenizer.decode(output_ids, skip_special_tokens=True)print("Transcript:", transcript)

import ctranslate2from transformers import WhisperProcessorimport librosa # 1. Load processor and CTranslate2 modelprocessor = WhisperProcessor.from_pretrained("tonibirat/sagarmatha-v6-nepali-asr")model = ctranslate2.models.Whisper(    "tonibirat/sagarmatha-v6-nepali-asr",    subfolder="ct2-fp16",    device="cuda",  # or "cpu"    compute_type="float16"  # or "int8_float16" on CUDA, "int8" on CPU) # 2. Load audio (resampled to 16 kHz mono)audio, sr = librosa.load("nepali_speech.wav", sr=16000) # 3. Extract log-mel filterbank featuresinputs = processor(audio, sampling_rate=16000, return_tensors="np")features = ctranslate2.StorageView.from_array(inputs.input_features) # 4. Define mode prompt: <|deva|><|clean|><|ne|><|transcribe|><|notimestamps|># IDs: 53866 (<|deva|>), 53869 (<|clean|>), 50313 (<|ne|>), 50360 (<|transcribe|>), 52364 (<|notimestamps|>)prompt_tokens = [53866, 53869, 50313, 50360, 52364] # IMPORTANT: Always set max_length=896 to prevent Devanagari byte fallback cutoffresults = model.generate(    features,    [prompt_tokens],    max_length=896,    repetition_penalty=1.1,    no_repeat_ngram_size=6) # 5. Decode predicted token IDs to Nepali textoutput_ids = results[0].sequences_ids[0]transcript = processor.tokenizer.decode(output_ids, skip_special_tokens=True)print("Transcript:", transcript)

2. Standard Inference with Hugging Face Transformers

python

import torchfrom transformers import WhisperForConditionalGeneration, WhisperProcessorimport librosa device = "cuda" if torch.cuda.is_available() else "cpu"torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32 # 1. Load model and processorprocessor = WhisperProcessor.from_pretrained("tonibirat/sagarmatha-v6-nepali-asr")model = WhisperForConditionalGeneration.from_pretrained(    "tonibirat/sagarmatha-v6-nepali-asr",    torch_dtype=torch_dtype).to(device) # 2. Load and process audioaudio, sr = librosa.load("nepali_speech.wav", sr=16000)inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(device, dtype=torch_dtype) # 3. Force mode tokens in decoder: deva+clean# <|deva|> (53866), <|clean|> (53869), <|ne|> (50313), <|transcribe|> (50360), <|notimestamps|> (52364)forced_prefix_ids = [    (1, 53866),  # <|deva|>    (2, 53869),  # <|clean|>    (3, 50313),  # <|ne|>    (4, 50360),  # <|transcribe|>    (5, 52364),  # <|notimestamps|>] # 4. Generatewith torch.no_grad():    predicted_ids = model.generate(        inputs.input_features,        forced_decoder_ids=forced_prefix_ids,        max_length=448    ) transcript = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]print("Transcript:", transcript)

import torchfrom transformers import WhisperForConditionalGeneration, WhisperProcessorimport librosa device = "cuda" if torch.cuda.is_available() else "cpu"torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32 # 1. Load model and processorprocessor = WhisperProcessor.from_pretrained("tonibirat/sagarmatha-v6-nepali-asr")model = WhisperForConditionalGeneration.from_pretrained(    "tonibirat/sagarmatha-v6-nepali-asr",    torch_dtype=torch_dtype).to(device) # 2. Load and process audioaudio, sr = librosa.load("nepali_speech.wav", sr=16000)inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(device, dtype=torch_dtype) # 3. Force mode tokens in decoder: deva+clean# <|deva|> (53866), <|clean|> (53869), <|ne|> (50313), <|transcribe|> (50360), <|notimestamps|> (52364)forced_prefix_ids = [    (1, 53866),  # <|deva|>    (2, 53869),  # <|clean|>    (3, 50313),  # <|ne|>    (4, 50360),  # <|transcribe|>    (5, 52364),  # <|notimestamps|>] # 4. Generatewith torch.no_grad():    predicted_ids = model.generate(        inputs.input_features,        forced_decoder_ids=forced_prefix_ids,        max_length=448    ) transcript = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]print("Transcript:", transcript)

Critical Engineering Best Practices

1. Token Budget & Context Limit (`max_length = 896`)

2. Voice Activity Detection (VAD) Gating

3. Caution Regarding Whisper's `BasicTextNormalizer`

Training Details

Base Architecture: openai/whisper-large-v3-turbo (809M parameters, 4 decoder layers, 32 encoder layers).
Corpus: tonibirat/nepali-asr-v6 — 1,285.4 hours (280,376 audio windows) with word-level forced alignment.
Hardware: NVIDIA RTX 5090 (32 GB VRAM).
Optimizer & Schedule: AdamW, cosine learning rate decay with linear warmup, fp16 mixed precision.
Soup Interpolation: $θ final = 0.80 \times θ V6 + 0.20 \times θ V5 $ .

Citation & Contact

If you use Sagarmatha V6 in your research or application, please cite:

bibtex

@misc{sagarmatha2026v6,  author = {Basyal, Toni and contributors},  title = {Sagarmatha V6: Controllable State-of-the-Art Speech Recognition for Nepali},  year = {2026},  publisher = {Hugging Face},  howpublished = {\url{https://huggingface.co/tonibirat/sagarmatha-v6-nepali-asr}}}

@misc{sagarmatha2026v6,  author = {Basyal, Toni and contributors},  title = {Sagarmatha V6: Controllable State-of-the-Art Speech Recognition for Nepali},  year = {2026},  publisher = {Hugging Face},  howpublished = {\url{https://huggingface.co/tonibirat/sagarmatha-v6-nepali-asr}}}

License

This model is licensed under the Apache License 2.0.

sagarmatha-v6-nepali-asr

README

Overview

Benchmark Results

1. Public Benchmarks (deva+clean mode)

2. Comparison with Contemporary Systems

Controllable Mode Tokens

Recommended Production Prefix

Usage & Code Examples

1. Fast Inference with CTranslate2 (Recommended for Production)

2. Standard Inference with Hugging Face Transformers

Critical Engineering Best Practices

1. Token Budget & Context Limit (max_length = 896)

2. Voice Activity Detection (VAD) Gating

3. Caution Regarding Whisper's BasicTextNormalizer

Training Details

Citation & Contact

License

Explore FriendliAI today

README

Overview

Benchmark Results

1. Public Benchmarks (deva+clean mode)

2. Comparison with Contemporary Systems

Controllable Mode Tokens

Recommended Production Prefix

Usage & Code Examples

1. Fast Inference with CTranslate2 (Recommended for Production)

2. Standard Inference with Hugging Face Transformers

Critical Engineering Best Practices

1. Token Budget & Context Limit (max_length = 896)

2. Voice Activity Detection (VAD) Gating

3. Caution Regarding Whisper's BasicTextNormalizer

Training Details

Citation & Contact

License

1. Public Benchmarks (`deva+clean` mode)

1. Token Budget & Context Limit (`max_length = 896`)

3. Caution Regarding Whisper's `BasicTextNormalizer`

1. Public Benchmarks (`deva+clean` mode)

1. Token Budget & Context Limit (`max_length = 896`)

3. Caution Regarding Whisper's `BasicTextNormalizer`