GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
mbehr90
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
mbehr90
Model Tree
openai/whisper-large-v3-turbo
Input Modalities
Output Modalities
Supported Functionality
FP8 E4M3 weights with a per-output-channel scale; activations quantized per token at
runtime. proj_out (the output head, tied to the token embeddings) and the two encoder Conv1d
layers are untouched.
WER-neutral: 2.229 % against the fp16 source's 2.303 % under transformers, 2.137 % against 2.146 % under vLLM — both differences well inside the noise of 500 utterances. But read the throughput column before deploying this on Hopper.
| build | size | WER (transformers) | WER (vLLM) | RTFx |
|---|---|---|---|---|
| openai/whisper-large-v3-turbo (source) | 1.51 GiB | 2.303 % | 2.146 % | 437.8 |
whisper-large-v3-turbo-fp8 ← this repo | 0.83 GiB | 2.229 % | 2.137 % | 317.1 |
| NVFP4 encoder + FP8 decoder (not published — see below) | 0.57 GiB | 2.422 % | cannot load | — |
whisper-large-v3-turbo-nvfp4 | 0.53 GiB | 2.561 % | 2.330 % | 380.5 |
WER on the first 500 utterances of LibriSpeech test-clean, scored after Whisper's
EnglishTextNormalizer, identical corpus and identical WER implementation for both runtimes.
transformers loads the checkpoint with run_compressed=False (decompressed to bf16), so that column
isolates the quantization from the kernel. RTFx = seconds of audio transcribed per second of wall
clock, 16 concurrent requests.
Quantization makes this model slower, not faster. RTFx measured on the same 500 utterances at 16 concurrent requests: fp16 437.8, FP8 317.1 (−28 %), NVFP4 380.5 (−13 %).
The reason is size. At 1.6 GB there is no memory-bandwidth pressure to relieve, the workload is
encoder-bound and compute-bound, and with d_model = 1280 the GEMMs are small enough that
per-matmul dequantization overhead outweighs what the FP8 tensor cores give back. The whole model
comfortably fits in HBM either way; the absolute saving is under one gigabyte.
Use this build if you are packing many Whisper instances onto one card, or targeting Blackwell where NVFP4 runs as true W4A4. For a single-model H100 deployment, the fp16 original is faster and at least as accurate.
whisper-large-v3-turbo is lopsided: 32 encoder layers (192 Linear, ~630M params) against 4
decoder layers (40 Linear, ~105M). A third build, NVFP4 on the encoder and FP8 on the decoder,
scores 2.422 % — meaning the 4-bit decoder, at 13 % of the weights, costs roughly as much accuracy
as the entire 4-bit encoder does. Four layers that run autoregressively per token are simply more
sensitive than 32 that run once per 30 s window.
That mixed build is not published: no vLLM available here can load it. Whisper's
packed_modules_mapping fuses cross-attention K and V into a two-way MergedColumnParallelLinear,
and the mixed-precision weight loader dies on it with
AttributeError: 'MergedColumnParallelLinear' object has no attribute 'data' — in 0.26.0 and in a
0.1.dev20073 nightly alike. The same mixed-precision format loads fine for decoder-only models.
bash
vllm serve mbehr90/whisper-large-v3-turbo-fp8 --max-model-len 448curl http://localhost:8000/v1/audio/transcriptions \-F file=@audio.wav -F model=mbehr90/whisper-large-v3-turbo-fp8 -F language=en
The stock vllm/vllm-openai image ships without audio extras — every transcription request comes
back 400 with "Please install vllm[audio] for audio support". Install soundfile and librosa
into the image, or into a directory on PYTHONPATH.
recipe.yaml is the exact llm-compressor recipe. Calibration for NVFP4: 256 LibriSpeech utterances
as log-mel input_features plus decoder_input_ids. Two things that do not work out of the box —
llm-compressor's dataset path assumes text (data["labels"] = data["input_ids"].copy(), and Whisper
has no input_ids), so pass a ready DataLoader; and the sequential pipeline cannot torch.fx-trace
an encoder-decoder, so use pipeline="basic".