GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
fixie-ai
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
fixie-ai
Model Tree
Input Modalities
Output Modalities
Supported Functionality
Ultravox is a multimodal model that can consume both speech and text as input (e.g., a text system prompt and voice user message).
The input to the model is given as a text prompt with a special <|audio|> pseudo-token, and the model processor will replace this magic token with embeddings derived from the input audio.
Using the merged embeddings as input, the model will then generate output text as usual.
In a future revision of Ultravox, we plan to expand the token vocabulary to support generation of semantic and acoustic audio tokens, which can then be fed to a vocoder to produce voice output. No preference tuning has been applied to this revision of the model.
Think of the model as an LLM that can also hear and understand speech. As such, it can be used as a voice agent, and also to do speech-to-speech translation, analysis of spoken audio, etc.
To use the model, try the following:
python
# pip install transformers peft librosa import transformersimport numpy as npimport librosa pipe = transformers.pipeline(model='fixie-ai/ultravox-v0_2', trust_remote_code=True) path = "<path-to-input-audio>" # TODO: pass the audio hereaudio, sr = librosa.load(path, sr=16000) turns = [ { "role": "system", "content": "You are a friendly and helpful character. You love to answer questions for people." },]pipe({'audio': audio, 'turns': turns, 'sampling_rate': sr}, max_new_tokens=30)The model uses a pre-trained Llama3-8B-Instruct backbone as well as the encoder part of Whisper-small.
The multi-modal projector is first trained (while keeping backbones frozen) in stage 1 and then in stage 2, Llama3 is also fine-tuned using LoRA.
Training dataset is a mix of ASR datasets (Gigaspeech), instruction-following and QA data (AnyInstruct and an extended version of BoolQ), and conversational data (SODA with alternative generations for last two turns).
Supervised speech to audio finetuning. For more info, see training code in Ultravox repo.
The current version of Ultravox, when invoked with audio content, has a time-to-first-token (TTFT) of approximately 200ms, and a tokens-per-second rate of ~50-100 when using an A100-40GB GPU, all using a Llama 3 8B backbone.
Check out the audio tab on TheFastest.ai for daily benchmarks and a comparison with other existing models.
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
# pip install transformers peft librosa import transformersimport numpy as npimport librosa pipe = transformers.pipeline(model='fixie-ai/ultravox-v0_2', trust_remote_code=True) path = "<path-to-input-audio>" # TODO: pass the audio hereaudio, sr = librosa.load(path, sr=16000) turns = [ { "role": "system", "content": "You are a friendly and helpful character. You love to answer questions for people." },]pipe({'audio': audio, 'turns': turns, 'sampling_rate': sr}, max_new_tokens=30)