GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
switchboardllm
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
switchboardllm
Model Tree
Qwen/Qwen3-32B
Input Modalities
Output Modalities
Supported Functionality
Serve it with vLLM. In FP8 it fits on one 48 GB GPU:
text
vllm serve switchboardllm/switchboard-router-32b --quantization fp8 --max-model-len 4096Then ask it about a prompt, with any OpenAI client:
python
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")reply = client.chat.completions.create( model="switchboardllm/switchboard-router-32b", messages=[{"role": "user", "content": "Unscramble the following words: eglar, ngareo, kobo"}], max_tokens=24, temperature=0,)print(reply.choices[0].message.content)json
{"tier":"frontier","domain":"general"}Or let the switchboard-router package do the routing:
python
from switchboard_router import Router router = Router(backend="openai", base_url="http://localhost:8000/v1", backend_model="switchboardllm/switchboard-router-32b")router.pick_model("Unscramble the following words: eglar, ngareo, kobo")switchboard serve puts the same router in front of your app as an OpenAI-compatible proxy: send model="auto".
Use greedy decoding (temperature 0).
| Field | Values |
|---|---|
tier | small, mid or frontier: the cheapest class of model expected to answer the prompt well |
domain | code, math, writing or general: a hint, from keyword labels |
If the output isn't valid JSON, send the prompt to frontier. A failed routing call should cost money, never quality.
All three models on the same 10,000 held-out prompts they never saw in training, in their natural mix:
| 0.5B | 4B | 32B | |
|---|---|---|---|
| Cost saved vs. always using Claude Opus 5.5 | 64.5% | 67.0% | 64.3% |
| Hard prompts sent to frontier | 59.1% | 63.2% | 67.5% |
| Sent too low (needed frontier, sent cheaper) | 5.5% | 5.0% | 4.4% |
| Prompts served well (proxy) | 94.5% | 95.0% | 95.6% |
| Ranking AUC (hard vs. rest) | 0.787 | 0.827 | 0.840 |
| Tier accuracy | 60.3% | 64.0% | 63.2% |
| Invalid JSON outputs | 0 | 0 | 0 |
32B minus 4B, paired bootstrap over the 10,000 prompts (2,000 resamples, 95% intervals):
| 32B minus 4B | |
|---|---|
| Cost saved | -2.8 pts (-3.3 to -2.1) |
| Hard prompts sent to frontier | +4.4 pts (+2.3 to +6.4) |
| Sent too low | -0.6 pts (-0.9 to -0.3) |
A perfect router on this data would save 76.2%. The labels set that ceiling, not the model size.
How it's measured. Each prompt in routellm/gpt4_dataset comes with a 2023 Mixtral-8x7B answer scored 1-5 by a GPT-4 judge. A prompt counts as served well if Switchboard sent it to frontier, or sent it lower and that answer scored at least 4. Cost uses Surplus output prices per 1M tokens: Claude Haiku 5.5 (0.50)forsmall,ClaudeSonnet5.5(10) for mid, Claude Opus 5.5 ($20) for frontier.
In FP8 the 32B fits on one 48 GB GPU. We ran 1,347 prompts through vLLM in BF16 and in FP8: the routing decision was the same on 1,285 of them (95.4%).
The 62 that changed go both ways: FP8 picked a higher tier 32 times and a lower one 30 times. On the 1,000 labelled prompts in that set, FP8 routes as well as BF16:
| BF16 | FP8 | |
|---|---|---|
| Sent too low | 4.2% | 4.2% |
| Cost saved | 65.1% | 64.5% |
| Base | Qwen3-32B (Apache-2.0), 32.8B parameters |
| Method | QLoRA: rank 32 adapters trained on a 4-bit NF4 base, then merged into the BF16 base weights |
| Training data | routellm/gpt4_dataset (Apache-2.0, LMSYS), 63,749 rows, the same set as the 0.5B and 4B |
| Labels | Mixtral score 5 → small, 4 → mid, 1-3 → frontier; domain from keywords |
| Training | 1 epoch in 4.1 hours on one H100 80GB. Best checkpoint by eval loss (0.0829). Loss on the JSON answer only. |
| Prompt format | The same chat template as the 0.5B and 4B, so the three are drop-in replacements for each other |
| Prompt length | Trained on prompts cut to 512 tokens |
domain comes from keyword labels and is only a hint.Apache-2.0, inherited from Qwen3-32B and routellm/gpt4_dataset.
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")reply = client.chat.completions.create( model="switchboardllm/switchboard-router-32b", messages=[{"role": "user", "content": "Unscramble the following words: eglar, ngareo, kobo"}], max_tokens=24, temperature=0,)print(reply.choices[0].message.content){"tier":"frontier","domain":"general"}from switchboard_router import Router router = Router(backend="openai", base_url="http://localhost:8000/v1", backend_model="switchboardllm/switchboard-router-32b")router.pick_model("Unscramble the following words: eglar, ngareo, kobo")