Model Details
Table | |
|---|
| Base LLM | Qwen/Qwen3.6-27B (27B params, hybrid GDN/DeltaNet attention) |
| Audio encoder | openai/whisper-large-v3-turbo |
| Projection adapter | Swiglu MLP, stack_factor=8, hidden_size=5120 |
| Trainable params | ~100 M (projector + Whisper LoRA r=8) |
| Frozen params | 27 B (LLM never gradient-updated) |
| Training precision | bfloat16 |
| Recommended serve precision | Base LLM in FP8, projector + LoRA in bf16 |
| Language | English (base checkpoint) |
| Context length | 131 072 tokens |
Training
- Loss: KL divergence to teacher (temperature 2.0)
- Steps: 10 000 (loss converged near step ~500, remaining steps refined the projection)
- Effective batch: 16 (8 × 2 gradient accumulation)
- LR: 5e-4 cosine → 5e-5, 500-step warmup
- Optimizer: AdamW
- Hardware: 1 × NVIDIA B200 (180 GB HBM3e), single-GPU, no FSDP
- Wall-clock: ~6 h 30 min
- Whisper adaptation: LoRA rank 8 on Whisper
k_proj, q_proj, linear_k, linear_q
Training data
English audio-transcription and audio-continuation mix (~10 datasets):
fixie-ai/librispeech_asr (clean + other, continuation + transcription)fixie-ai/peoplespeech-cleanfixie-ai/gigaspeech-xlfixie-ai/commonvoice-en
This is an English-only base checkpoint. Multilingual and phone-quality audio training were deliberately excluded to accelerate iteration and reduce cost — Phase 1B (broader dataset mix) may be released later.
Evaluation
Training-time metrics on fixie-ai/librispeech-clean-transcription (validation, 256 samples):
Table with columns: Step, Eval loss| Step | Eval loss |
|---|
| 500 | 0.442 |
| 5 000 | 0.383 |
| 9 000 | 0.381 |
| 10 000 (final) | 0.372 |
WER benchmark is being computed against librispeech-clean-test and will be added to this card as soon as it lands. Fixie's ultravox-v0_6-qwen-3-32b achieves 2.88 WER on librispeech (multilingual training, 32B base). Our English-only + 27B target is 3.5-5.0.
Usage
With vLLM (recommended for serving)
vllm serve QuantumDesk-AI/ultravox-qwen3.6-27b-base \
--served-model-name ultravox \
--max-model-len 32768 \
--gpu-memory-utilization 0.85 \
--port 8000
from transformers import AutoModel, AutoProcessor
import torchaudio
processor = AutoProcessor.from_pretrained(
"QuantumDesk-AI/ultravox-qwen3.6-27b-base",
trust_remote_code=True,
)
model = AutoModel.from_pretrained(
"QuantumDesk-AI/ultravox-qwen3.6-27b-base",
trust_remote_code=True,
torch_dtype="bfloat16",
).to("cuda")
audio, sr = torchaudio.load("hello.wav")
messages = [
{"role": "user", "content": [
{"type": "audio", "audio": audio[0].numpy(), "sampling_rate": sr},
{"type": "text", "text": "What did I say?"},
]},
]
inputs = processor.apply_chat_template(messages, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(outputs[0], skip_special_tokens=True))
Serving hardware
FP8 base + bf16 adapter fits on any Blackwell (RTX PRO 6000, B200) or Hopper (H100 80/94GB) GPU:
Table with columns: Card, VRAM, Concurrent voice sessions*| Card | VRAM | Concurrent voice sessions* |
|---|
| RTX PRO 6000 Blackwell | 96 GB | ~30-40 |
| H100 94 GB | 94 GB | ~30-40 |
| B200 180 GB | 180 GB | ~80-100 |
*Compute-bound (KV cache is not the limit). Assumes ~2000 tokens context per session and voice-typical 5-6 tok/s per session sustained rate.
Limitations
- English only — do not use for other languages; expect degraded quality even for accented English.
- Clean audio only — trained without phone-codec, noise, or music-background data. Real-world telephony will underperform relative to studio audio.
- No chat / instruction tuning — this is a base checkpoint. For interactive voice agents, fine-tune on your domain (see [Phase 2 discussion in repo]).
- Reasoning under audio input — some capacity is spent decoding audio; expect small (2-5 %) degradation vs pure-text prompting on reasoning-heavy tasks. Use text cascade for complex queries.
License
Apache 2.0. Model weights are Quantum Desk LTD's contribution. See individual base-model licenses for Qwen 3.6-27B (Apache 2.0) and Whisper-large-v3-turbo (MIT).
Acknowledgements
Built on:
Citation
@misc{quantumdesk_ultravox_qwen36_2026,
author = {Quantum Desk LTD},
title = {Ultravox-Qwen3.6-27B-Base},
year = {2026},
publisher = {HuggingFace},
url = {https://huggingface.co/QuantumDesk-AI/ultravox-qwen3.6-27b-base},
}