GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
huwenjie333
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
huwenjie333
Model Tree
huwenjie333/whisper-v3-ft-af51
Input Modalities
Output Modalities
Supported Functionality
The model is used in a similar way to the base Whisper model. To perform the transcription for certain language, for better accuracy, please include the language token mapping SALT_LANGUAGE_TOKENS_WHISPER, and specify the language code in the forced_decoder_ids during generation. See the example below:
python
import transformersimport datasetsimport torch model_id = "huwenjie333/whisper-v3-ft-af51-0703"processor = transformers.WhisperProcessor.from_pretrained(model_id)model = transformers.WhisperForConditionalGeneration.from_pretrained(model_id) SALT_LANGUAGE_TOKENS_WHISPER = { # Existing languges codes from Whisper "eng": 50259, "fra": 50265, "swa": 50318, "sna": 50324, "yor": 50325, "som": 50326, "afr": 50327, "amh": 50334, "mlg": 50349, "lin": 50353, "hau": 50354, # Overwrite unused language tokens "ach": 50357, "aka": 50356, "bam": 50355, "bem": 50352, "ber": 50351, "cgg": 50350, "dag": 50348, "dga": 50347, "ewe": 50346, "ful": 50345, "ibo": 50344, "kab": 50343, "kau": 50342, "kik": 50341, "kin": 50340, "kln": 50339, "koo": 50338, "kpo": 50337, "led": 50336, "lgg": 50335, "lth": 50333, "lug": 50332, "luo": 50331, "luy": 50330, "myx": 50329, "nbl": 50328, "nya": 50323, "nyn": 50322, "orm": 50321, "pcm": 50320, "ruc": 50319, "rwm": 50317, "sot": 50316, "teo": 50315, "tsn": 50314, "ttj": 50313, "wol": 50312, "xho": 50311, "xog": 50310, "zul": 50309,} # Get some test audiods = datasets.load_dataset('Sunbird/salt', 'multispeaker-lug', split='test')audio = ds[0]['audio']sample_rate = ds[0]['sample_rate'] # Specify a language from one of the above.lang = 'lug' # Apply the modeldevice = torch.device("cuda" if torch.cuda.is_available() else "cpu")input_features = processor( audio, sampling_rate=sample_rate, return_tensors="pt").input_featuresinput_features = input_features.to(device) lang_tok = SALT_LANGUAGE_TOKENS_WHISPER[lang_code]transcribe_tok = processor.tokenizer.convert_tokens_to_ids("<|transcribe|>")notimestamps_tok = processor.tokenizer.convert_tokens_to_ids("<|notimestamps|>")forced_decoder_ids = [ (1, lang_tok), (2, transcribe_tok), (3, notimestamps_tok),] predicted_ids = model.to(device).generate( input_features, forced_decoder_ids=forced_decoder_ids, num_beams=1, do_sample=False,)transcription = processor.batch_decode( predicted_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) print(transcription)# Ekikoola kya kasooli kya kyenvu wabula langi yaakyo etera okuba eya kitaka wansi.The Word Error Rate (WER) and Character Error Rate (CER) are measured for each language on the evaluation datasets. For ach, eng, lgg, lug, nyn, teo, the dev split of multispeaker-* subsets in Sunbird/salt dataset were used for evaluation. For myx and xog, the last 100 examples of train split of makerere-yogera-* subsets in Sunbird/external-speech-data dataset were used. For the rest languages, we picked first 50 examples in the dev split of each subset in huwenjie333/speech dataset.
text
code language WER CER n_examplesafr Afrikaans 0.084 0.028 170aka Akan 0.323 0.129 50amh Amharic 0.401 0.265 250bam Bambara 0.361 0.170 149bem Bemba 0.408 0.091 100ber Berber 0.147 0.041 50cgg Rukiga 0.242 0.057 49dag Dagbani 0.321 0.109 100dga Dagaare 0.294 0.113 50ewe Ewe 0.291 0.097 78fra French 0.032 0.019 50ful Fulani 0.395 0.115 149hau Hausa 0.168 0.047 287ibo Igbo 0.340 0.117 247kab Kabyle 0.447 0.219 50kau Kanuri 0.389 0.101 50kik Kikuyu 0.209 0.099 50kin Kinyarwanda 0.307 0.084 100kln Kalenjin 0.471 0.112 100koo Rukonjo 0.527 0.093 47kpo Ikposo 0.617 0.231 50led Lendu 0.273 0.093 50lin Lingala 0.243 0.104 193lth Thur 0.148 0.059 41luo Luo 0.226 0.051 149luy Luhya 0.203 0.038 50mlg Malagasy 0.143 0.060 49nbl Ndebele 0.263 0.051 10nya Chichewa 0.273 0.058 98orm Oromo 0.241 0.064 119pcm Nigerian Pidgin 0.167 0.065 100ruc Ruruuli 0.480 0.105 38rwm Kwamba 0.584 0.234 49sna Shona 0.196 0.050 147som Somali 0.381 0.127 149sot Sotho 0.229 0.117 94swa Swahili 0.087 0.022 99tsn Tswana 0.067 0.018 98ttj Rutooro 0.222 0.063 49wol Wolof 0.319 0.122 135xho Xhosa 0.262 0.054 126yor Yoruba 0.384 0.136 249zul Zulu 0.207 0.042 143ach Acholi 0.197 0.044 101eng English 0.041 0.016 101lgg Lugbara 0.216 0.052 101lug Luganda 0.109 0.022 103nyn Runyankole 0.262 0.051 103teo Ateso 0.275 0.068 101myx Lumasaba 0.347 0.071 100xog Lusoga 0.301 0.056 100
import transformersimport datasetsimport torch model_id = "huwenjie333/whisper-v3-ft-af51-0703"processor = transformers.WhisperProcessor.from_pretrained(model_id)model = transformers.WhisperForConditionalGeneration.from_pretrained(model_id) SALT_LANGUAGE_TOKENS_WHISPER = { # Existing languges codes from Whisper "eng": 50259, "fra": 50265, "swa": 50318, "sna": 50324, "yor": 50325, "som": 50326, "afr": 50327, "amh": 50334, "mlg": 50349, "lin": 50353, "hau": 50354, # Overwrite unused language tokens "ach": 50357, "aka": 50356, "bam": 50355, "bem": 50352, "ber": 50351, "cgg": 50350, "dag": 50348, "dga": 50347, "ewe": 50346, "ful": 50345, "ibo": 50344, "kab": 50343, "kau": 50342, "kik": 50341, "kin": 50340, "kln": 50339, "koo": 50338, "kpo": 50337, "led": 50336, "lgg": 50335, "lth": 50333, "lug": 50332, "luo": 50331, "luy": 50330, "myx": 50329, "nbl": 50328, "nya": 50323, "nyn": 50322, "orm": 50321, "pcm": 50320, "ruc": 50319, "rwm": 50317, "sot": 50316, "teo": 50315, "tsn": 50314, "ttj": 50313, "wol": 50312, "xho": 50311, "xog": 50310, "zul": 50309,} # Get some test audiods = datasets.load_dataset('Sunbird/salt', 'multispeaker-lug', split='test')audio = ds[0]['audio']sample_rate = ds[0]['sample_rate'] # Specify a language from one of the above.lang = 'lug' # Apply the modeldevice = torch.device("cuda" if torch.cuda.is_available() else "cpu")input_features = processor( audio, sampling_rate=sample_rate, return_tensors="pt").input_featuresinput_features = input_features.to(device) lang_tok = SALT_LANGUAGE_TOKENS_WHISPER[lang_code]transcribe_tok = processor.tokenizer.convert_tokens_to_ids("<|transcribe|>")notimestamps_tok = processor.tokenizer.convert_tokens_to_ids("<|notimestamps|>")forced_decoder_ids = [ (1, lang_tok), (2, transcribe_tok), (3, notimestamps_tok),] predicted_ids = model.to(device).generate( input_features, forced_decoder_ids=forced_decoder_ids, num_beams=1, do_sample=False,)transcription = processor.batch_decode( predicted_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) print(transcription)# Ekikoola kya kasooli kya kyenvu wabula langi yaakyo etera okuba eya kitaka wansi.code language WER CER n_examplesafr Afrikaans 0.084 0.028 170aka Akan 0.323 0.129 50amh Amharic 0.401 0.265 250bam Bambara 0.361 0.170 149bem Bemba 0.408 0.091 100ber Berber 0.147 0.041 50cgg Rukiga 0.242 0.057 49dag Dagbani 0.321 0.109 100dga Dagaare 0.294 0.113 50ewe Ewe 0.291 0.097 78fra French 0.032 0.019 50ful Fulani 0.395 0.115 149hau Hausa 0.168 0.047 287ibo Igbo 0.340 0.117 247kab Kabyle 0.447 0.219 50kau Kanuri 0.389 0.101 50kik Kikuyu 0.209 0.099 50kin Kinyarwanda 0.307 0.084 100kln Kalenjin 0.471 0.112 100koo Rukonjo 0.527 0.093 47kpo Ikposo 0.617 0.231 50led Lendu 0.273 0.093 50lin Lingala 0.243 0.104 193lth Thur 0.148 0.059 41luo Luo 0.226 0.051 149luy Luhya 0.203 0.038 50mlg Malagasy 0.143 0.060 49nbl Ndebele 0.263 0.051 10nya Chichewa 0.273 0.058 98orm Oromo 0.241 0.064 119pcm Nigerian Pidgin 0.167 0.065 100ruc Ruruuli 0.480 0.105 38rwm Kwamba 0.584 0.234 49sna Shona 0.196 0.050 147som Somali 0.381 0.127 149sot Sotho 0.229 0.117 94swa Swahili 0.087 0.022 99tsn Tswana 0.067 0.018 98ttj Rutooro 0.222 0.063 49wol Wolof 0.319 0.122 135xho Xhosa 0.262 0.054 126yor Yoruba 0.384 0.136 249zul Zulu 0.207 0.042 143ach Acholi 0.197 0.044 101eng English 0.041 0.016 101lgg Lugbara 0.216 0.052 101lug Luganda 0.109 0.022 103nyn Runyankole 0.262 0.051 103teo Ateso 0.275 0.068 101myx Lumasaba 0.347 0.071 100xog Lusoga 0.301 0.056 100