GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
dkhokhlov
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
dkhokhlov
Model Tree
openai/whisper-tiny
Input Modalities
Output Modalities
Supported Functionality
dkhokhlov/whisper-base-hqq-4bit — HQQ 4-bit, whisper-base (CPU eval)dkhokhlov/whisper-small-hqq-4bit — HQQ 4-bit, whisper-small (A10 GPU eval)openai/whisper-tiny (fp32)dkhokhlov/whisper-cascadeopenai/whisper-tiny quantized
with HQQ 4-bit
grouped quantization for CPU inference. Resident weight RAM (fp16 compute) is
57.53 MB, 23.8% smaller than the unquantized fp16 model
(75.52 MB). fp16 compute is WER-neutral; the published WER benchmark uses
fp32 compute for cross-model comparability. The key setting is mixed
precision: the whole encoder stack and fc1 are kept at 8-bit, the remaining
decoder linears are 4-bit (see the repo README for the full config and the
config-sweep ablation).
English (fleurs en_us, n=100) WER is 0.1367 vs 0.1381 fp32 (-1.0%),
within n=100 noise. HQQ is within 5% relative of fp32 on every tested config
(5 fleurs + 4 talkbank).
English (fleurs en_us, n=100, fp32 compute):
| Metric | unquantized fp32 | HQQ 4-bit | Delta % |
|---|---|---|---|
| WER | 0.1381 | 0.1367 | -1.0% |
| Resident RAM (fp16) | 75.52 MB | 57.53 MB | -23.8% |
| Samples succeeded | 100 / 100 | 100 / 100 | - |
HQQ is within 5% relative of fp32 on every tested config. The full
multilingual and telephone WER tables, the cross-reference against
whisper-base/whisper-small, and the size-by-component breakdown are in
the repo README.
The model auto-detects the spoken language and transcribes (multilingual
Whisper behavior). Pass language to force a language when it is known.
python
import hqq_asrpipe = hqq_asr.build_pipeline("dkhokhlov/whisper-tiny-hqq-4bit", quant="hqq")text = pipe({"array": audio, "sampling_rate": 16000})["text"] # auto-detecttext = pipe({"array": audio, "sampling_rate": 16000},generate_kwargs={"language": "spanish", "task": "transcribe"})["text"] # force
Command line (this repository):
markdown
make asr MODEL_ASR=dkhokhlov/whisper-tiny-hqq-4bit QUANT=hqq AUDIO=clip.wav
This repo ships two ONNX exports of the same HQQ model, differing only in compute dtype:
encoder_model.onnx + decoder_model_merged.onnx —
a fp16-only graph (zero fp32 ops). It uses eager attention so the attention
scale stays a fp16 Mul (SDPA would decompose it to Sqrt→Div in fp32).
On CPU ONNX Runtime it runs ~4.4× slower than the fp32 export (ORT-CPU
upcasts fp16→fp32 internally — fp16 is not a primary CPU compute format, so
the slowdown is expected) but loads ~29% less RAM.encoder_model-fp32.onnx + decoder_model_merged-fp32.onnx — the
recommended CPU compute and the benchmark compute. Faster on CPU ORT; matches
the published fp32 WER benchmark.Both keep the packed uint8 W_q and the per-group scale/zero as ONNX
initializers and emit the unpack + dequant as standard ONNX ops (opset 18), so
each graph carries the exact HQQ weights, not a re-dequantized dense copy.
Whisper is an encoder-decoder model, so each export is two ONNX graphs; the
autoregressive generation loop (argmax, KV-cache, EOS stop) runs in Python in
ORTModelForSpeechSeq2Seq, calling the encoder once and the decoder once per
token:
| File (fp16 / fp32) | Role | Input | Output | Runs |
|---|---|---|---|---|
encoder_model.onnx / encoder_model-fp32.onnx | encoder | audio mel-spectrogram | hidden states | once per utterance |
decoder_model_merged.onnx / decoder_model_merged-fp32.onnx | decoder | encoder hidden states + KV cache | next text token | once per token (loop) |
The merged decoder carries the no-past (first step) and with-past (cached steps) branches behind one control-flow switch, so one session handles the whole generation; the separate un-merged decoder files optimum emits are not shipped.
Both exports reproduce the HQQ WER (0.1367). The fp32 export exact-matches the HQQ manifest (0 mismatches); the fp16 export matches the WER and differs from the fp32 manifest by one case-only token (a fp16-vs-fp32 rounding effect, WER-neutral).
Load via ONNX Runtime (the no-suffix files are the fp16 default):
python
import hqq_asrpipe = hqq_asr.build_pipeline("dkhokhlov/whisper-tiny-hqq-4bit", quant="onnx")text = pipe({"array": audio, "sampling_rate": 16000})["text"]
Reproduce the fp16 export and the gate (set HQQ_COMPUTE_DTYPE=fp32 for the
fp32 export):
markdown
make onnx HQQ_REPO=dkhokhlov/whisper-tiny-hqq-4bitmake hqq-reference HQQ_REPO=dkhokhlov/whisper-tiny-hqq-4bit EVAL_OUT=build/hqq_reference_tiny_fp16.jsonmake eval-onnx \HQQ_REFERENCE_MANIFEST=build/hqq_reference_tiny_fp16.json EVAL_OUT=build/eval_onnx_tiny_fp16.json
The export spec and the two validation gates are in docs/onnx.md in the repo.
markdown
# 1. Quantize locally (writes whisper-tiny-hqq-4bit/).python quantize.py# 2. Measure baseline WER (fp32).EVAL_LIMIT=100 MODEL_ASR=openai/whisper-tiny EVAL_CONFIG=en_us \EVAL_OUT=eval_baseline.json python eval_wer.py# 3. Measure HQQ WER.EVAL_LIMIT=100 QUANT=hqq MODEL_ASR=./whisper-tiny-hqq-4bit EVAL_CONFIG=en_us \EVAL_OUT=eval_hqq.json python eval_wer.py# 4. Telephone benchmark (talkbank segment split).EVAL_DATASET=diabolocom/talkbank_4_stt EVAL_CONFIG=en EVAL_SPLIT=segment EVAL_LIMIT=100 \MODEL_ASR=openai/whisper-tiny EVAL_OUT=talkbank_en_fp32.json python eval_wer.py# 5. Publish (needs a Hugging Face write token).PUSH=1 HQQ_REPO=dkhokhlov/whisper-tiny-hqq-4bit python quantize.py
MIT. Derived from openai/whisper-tiny
(Apache-2.0) and HQQ. The quantized
weights inherit the openai/whisper license terms.
See the repo README for the BibTeX entry.
Quantization config, config-sweep ablation, safetensors format, the full WER
tables (multilingual fleurs, talkbank telephone, cross-reference), and the
resident-RAM-by-component breakdown are in the repo
README. Per-config WER
evidence JSONs are committed under eval_multilingual/ and eval_telephone/
in dkhokhlov/whisper-cascade.