Benchmarks (native inference, greedy, n=100 per set)
Table with columns: Metric, v2, ultravox-v0_6-gemma-3-27b (same rig)| Metric | v2 | ultravox-v0_6-gemma-3-27b (same rig) |
|---|
| LibriSpeech test-clean WER | 2.08% | 1.90% |
| Switchboard (real phone calls) WER | 9.67% | 21.2% |
| Phone-number digit accuracy (phone-quality audio) | 97.0% | 98.5% |
| Person-name accuracy | 78.8% | 60.6% |
| Spelled-email accuracy | 53.2% | 50.6% |
| Business-name accuracy | 91.2% | 73.5% |
On conversational phone audio — the target domain — this model roughly halves the word
error rate of general-purpose Ultravox models, while matching digit accuracy and
substantially exceeding name recognition.
Conversational use
v2 responds natively to audio (not transcription-only). It is trained on this exact user
turn shape:
[
{"role": "system", "content": "<your agent persona>"},
{"role": "user", "content": [
{"type": "text", "text": "Respond conversationally to what the user just said. Keep replies short — one or two natural sentences. Never use markdown, code, symbols, or headers. Never invent quotes. If you didn't catch what they said, ask them to repeat. "},
{"type": "input_audio", "input_audio": {"data": "<b64 wav>", "format": "wav"}}
]}
]
For transcription use: "Repeat the following text, without any explanation: " + audio.
A persona system prompt is strongly recommended for conversational use.
Training
Two phases from scratch on 2×H200:
- Phase 1B (~35K steps, eff. batch 16): English ASR corpora + real telephone
conversations, with telephony augmentation (8kHz resample, AMR/μ-law compression,
band-pass, pink noise, gain).
- Phase 2 (15K steps): 150K synthetic entity-dense utterances (phone numbers, names,
spelled emails, addresses, dates/times across 31 TTS voices) + 40K spoken-dialogue
response pairs + anti-forgetting garnish from phase-1 corpora.
Serving
Works with vLLM (--trust-remote-code). Note: Qwen 3.6's hybrid linear attention
currently loses some accuracy under vLLM relative to native transformers inference; for
maximum quality use HF transformers, or vLLM with --no-enable-chunked-prefill and
default (fp32) mamba cache dtypes.
Apache 2.0. Built with the Ultravox training framework by Fixie.ai.