GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
0x3
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Talk with our engineer to get a quote for reserved GPU instances with discounts.
Model provider
0x3
Model tree
Base
this model
Modalities
Input
Audio, Text
Output
Text
Pricing
Dedicated Endpoints
View detailsSupported Functionality
Model APIs
Dedicated Endpoints
Container
More information
What it predicts. UltraVAD computes the probability P(<eot_id> | context, user_audio)
Use UltraVAD as a turn-taking oracle in voice agents. Run it alongside a lightweight streaming VAD; when short silences are detected, call UltraVAD and trigger your agent’s response once the <eot> probability crosses your threshold.
python
import transformersimport torchimport librosaimport ospipe = transformers.pipeline(model='fixie-ai/ultraVAD', trust_remote_code=True, device="cpu")sr = 16000wav_path = os.path.join(os.path.dirname(__file__), "sample.wav")audio, sr = librosa.load(wav_path, sr=sr)turns = [{"role": "assistant", "content": "Hi, how are you?"},]# Build model inputs via pipeline preprocessinputs = {"audio": audio, "turns": turns, "sampling_rate": sr}model_inputs = pipe.preprocess(inputs)# Move tensors to model devicedevice = next(pipe.model.parameters()).devicemodel_inputs = {k: (v.to(device) if hasattr(v, "to") else v) for k, v in model_inputs.items()}# Forward pass (no generation)with torch.inference_mode():output = pipe.model.forward(**model_inputs, return_dict=True)# Compute last-audio token positionlogits = output.logits # (1, seq_len, vocab)audio_pos = int(model_inputs["audio_token_start_idx"].item() +model_inputs["audio_token_len"].item() - 1)# Resolve <|eot_id|> token id and compute probability at last-audio indextoken_id = pipe.tokenizer.convert_tokens_to_ids("<|eot_id|>")if token_id is None or token_id == pipe.tokenizer.unk_token_id:raise RuntimeError("<|eot_id|> not found in tokenizer.")audio_logits = logits[0, audio_pos, :]audio_probs = torch.softmax(audio_logits.float(), dim=-1)eot_prob_audio = audio_probs[token_id].item()print(f"P(<|eot_id|>) = {eot_prob_audio:.6f}")threshold = 0.1if eot_prob_audio > threshold:print("Is End of Turn")else:print("Is Not End of Turn")
Text-only post-training (LLM): Post-train the backbone to predict in dialog, yielding a probability over likely stop points rather than brittle binary labels. Data: Synthetic conversational corpora with inserted tokens, translation-augmented across 26 languages.
Audio-native fusion (Ultravox projector): Attach and fine-tune the Ultravox audio projector so the model conditions jointly on audio embeddings and text, aligning prosodic cues with the objective. Data: Robust to real-world noise, device/mic variance, overlapping speech.
Calibration: Choose a decision threshold to balance precision vs. recall per language/domain. Recommended starting threshold: 0.1. Raise the threshold if you find the model interrupting too eagerly, and lower the threshold if you find the model not responding when its supposed to.
Latency (forward pass): ~65-110 ms on an A6000.
Common pattern: Pair with a streaming VAD (e.g., Silero). Invoke UltraVAD on short silences; its latency is often hidden under TTS time-to-first-token.
UltraVAD is evaluated on both context-dependent and single-turn datasets.
Contextual benchmark: 400 held-out samples requiring dialog history (fixie-ai/turntaking-contextual-tts).
Single-turn sets: Smart-Turn V2’s Orpheus synthetic datasets (aggregate).
Results
Context-dependent turn-taking (400 held-out samples)
| Metric | UltraVAD | Smart-Turn V2 |
|---|---|---|
| Accuracy | 77.5% | 63.0% |
| Precision | 69.6% | 59.8% |
| Recall | 97.5% | 79.0% |
| F1-Score | 81.3% | 68.1% |
| AUC | 89.6% | 70.0% |
Single-turn datasets (Orpheus aggregate)
| Dataset | UltraVAD | Smart-Turn V2 |
|---|---|---|
| orpheus-aggregate-test | 93.7% | 94.3% |