ultravox-v0_4

Model Description

Ultravox is a multimodal model that can consume both speech and text as input (e.g., a text system prompt and voice user message). The input to the model is given as a text prompt with a special <|audio|> pseudo-token, and the model processor will replace this magic token with embeddings derived from the input audio. Using the merged embeddings as input, the model will then generate output text as usual.

In a future revision of Ultravox, they plan to expand the token vocabulary to support generation of semantic and acoustic audio tokens, which can then be fed to a vocoder to produce voice output. No preference tuning has been applied to this revision of the model.

Usage

Think of the model as an LLM that can also hear and understand speech. As such, it can be used as a voice agent, and also to do speech-to-speech translation, analysis of spoken audio, etc.

Training Details

The model uses a pre-trained Llama3.1-8B-Instruct backbone as well as the encoder part of Whisper-medium.

Only the multi-modal adapter is trained, while Whisper encoder and Llama are kept frozen.

They use a knowledge-distillation loss where Ultravox is trying to match the logits of the text-based Llama backbone.

Training Data

Training dataset is a mix of ASR datasets, extended by adding a "continuation" generated by Llama 3.1 8B.

Training Procedure

Supervised speech to audio finetuning. For more info, see training code in Ultravox repo.

Training Hyperparameters

• Training regime: BF16 mixed precision training
• Hardware used: 8x H100 GPUs

Speeds, Sizes, Times

The current version of Ultravox, when invoked with audio content, has a time-to-first-token (TTFT) of approximately 150ms, and a tokens-per-second rate of ~50-100 when using an A100-40GB GPU, all using a Llama 3.1 8B backbone.

License

MIT