armanibadboy

whisper-small-ternary

Files

Table with columns: file, what it is
file	what it is
`ggml-small-wal-ternary-q2_0.bin`	140 MiB — run this. ggml Q2_0 for whisper.cpp, CPU inference
`model.safetensors`	922 MiB — the same weights materialised dense in fp32, for verification in PyTorch

The dense file is not a different model. It is the ternary codes multiplied by their learned scales, written out as ordinary weights so you can reproduce the WER numbers with transformers and confirm they match.

Run on CPU

Q2_0 exists in ggml but had never been exercised end to end, so whisper.cpp needs a patch — it is four small changes and it is in the repo:

bash
git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp
git apply ../whisper-ternary/deploy/whisper_cpp_q2_0.patch
cmake -B build && cmake --build build -j

./build/bin/whisper-cli -m ggml-small-wal-ternary-q2_0.bin -f audio.wav -ng

As a service:

bash
./build/bin/whisper-server -m ggml-small-wal-ternary-q2_0.bin \
    --host 127.0.0.1 --port 8178 -t 16 -ng -ml 100000
curl 127.0.0.1:8178/inference -F file=@audio.wav -F response_format=json

Pass -ml 100000. Without it the server forces 60-character segments and splits words in half (fl uttered).

Verify in PyTorch

python
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained("armanibadboy/whisper-small-ternary")
proc  = WhisperProcessor.from_pretrained("armanibadboy/whisper-small-ternary")

Every quantised matrix holds at most 3 distinct values per group of 128 — check it yourself:

python
import torch
W = model.model.encoder.layers[0].fc1.weight
g = W[:, :128]
print(torch.unique(g / g.abs()[g != 0].min()).round())   # -> tensor([-1., 0., 1.])

Speed

Same weights, 66 s of audio, 16 threads, batch 1, Xeon:

Table with columns: fp16, ternary Q2_0
	fp16	ternary Q2_0
file	465 MiB	140 MiB
encoder / 30 s window	578 ms	473 ms
decode / step	6.8 ms	6.0 ms

Against an H200 running the same weights in PyTorch, a single decoder step is faster on the CPU — 6.0 ms against 8.3 ms — because at batch 1 the GPU is launch-bound (439 kernel launches per decoder pass, ~15 µs each) rather than compute-bound. But end to end the H200 still wins by 2.3× (1.17 s against 2.7 s for the same 66 s of audio): the encoder is a dense matmul over 1500 frames and there the GPU is 77× ahead. Note this compares runtimes as well as hardware — hand-written C/AVX2 against PyTorch.

Limitations

Stated plainly, because a quantisation result without a failure list has not been examined hard enough.

No punctuation. 0.0–0.7 % of utterances contain a comma depending on mode; stock whisper-small manages 63.3 %. Capitals appear on 2 % against 98 %. In timestamp mode a sentence-final period survives on 15 % of utterances — residue from the original model, not learned punctuation. Measured on 300 utterances.
English only. Russian degrades by 12.6×. An fp16 control trained identically degrades the same way, so the cause is monolingual fine-tuning, not ternarisation.
Out of domain, English is roughly twice as bad as on LibriSpeech.
The tied embedding head stays fp16 — 18 % of parameters, 86 of the 140 MiB. A one-shot ternary projection of it, with no dedicated QAT, takes WER from 2.71 % to 1152 %. That shows this projection fails on that tensor, not that the head is incompressible; a learned Q4 head would plausibly bring the file to ~78 MiB.
Load takes 950 ms against 447 ms for fp16 — the encoder expansion at load is an unresolved 826 ms. Three hypotheses were tested and all three were wrong.
Trained and evaluated on read speech (LibriSpeech). Conversational and noisy audio were not measured.

What ternarisation cost

Beating stock whisper-small answers "is this model good", not "what did quantisation cost" — this model was also fine-tuned. So an fp16 control was trained on the same code path, same data, same schedule, quantisation off:

the gap holds at ≈0.37 pp absolute on test-clean and does not narrow;
neither branch reached a floor — both still falling nearly linearly when the runs ended (−0.173 and −0.186 pp per 10k steps);
so "ternary gets to the same place, just later" is unproven. It needs a run to plateau, which was not done.

Training

Table

base	`openai/whisper-small`
steps	80 000 (70k packed LibriSpeech 951 h, then 10k repair with timestamps)
quantised	192 matrices, 198 180 864 weights, alphabet {−1, 0, +1}, group 128
bits per weight	2.125
method	quantisation-aware distillation, straight-through estimator
loss	KL against bf16 teacher + cross-entropy + feature matching

Codes are initialised by an activation-weighted projection, not by rounding: per group of 128 inputs we solve for the codes and scale minimising the error weighted by how large the activations multiplying them actually are.

One trap worth naming, specific to this design: in classical QAT the high-precision master weights are genuinely the source the deployed checkpoint is rebuilt from. Here they are not — the group scales are separately learned parameters, so re-projecting from the latent tensor recomputes scales that were never trained, and the latent tensor is only a gradient surrogate free to drift outside its quantisation cells. Re-projecting from it produced 1000.56 % WER. The deployment representation is the trained (codes, scales) pair, from the start.

Evaluation protocol

markdown
batch_size     = 16
max_new_tokens = 440
greedy, num_beams = 1

Batch size shifts WER by 0.013 pp on identical code (3.4298 % / 3.4224 % / 3.4168 % at batch 16 / 8 / 1) — systematic implementation sensitivity, not random noise. Batch size is therefore pinned and differences under 0.02 pp are not treated as signal. No confidence intervals are reported yet; the test-other gap in particular has not been shown to exceed its uncertainty.

test-clean and test-other are selection sets, not sealed tests. They took no part in gradient training, but were measured 21 times across four runs, and this checkpoint was chosen over its own step-7500 sibling by looking at them. A publication-grade claim needs a fresh one-shot set; that has not been done.

The CPU path was verified independently: all 2620 utterances of test-other through patched whisper.cpp give 7.3754 %, against 7.3817 % in the PyTorch harness — within the noise floor.

Citation

bibtex
@software{aubakirov2026ternarywhisper,
  author = {Aubakirov, Arman},
  title  = {Ternary Whisper: whisper-small at 2.125 bits per weight},
  year   = {2026},
  url    = {https://github.com/AubakirovArman/whisper-ternary}
}

MIT.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

armanibadboy

Model Tree

Base

openai/whisper-small

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Files

Table with columns: file, what it is
file	what it is
`ggml-small-wal-ternary-q2_0.bin`	140 MiB — run this. ggml Q2_0 for whisper.cpp, CPU inference
`model.safetensors`	922 MiB — the same weights materialised dense in fp32, for verification in PyTorch

Run on CPU

Q2_0 exists in ggml but had never been exercised end to end, so whisper.cpp needs a patch — it is four small changes and it is in the repo:

bash
git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp
git apply ../whisper-ternary/deploy/whisper_cpp_q2_0.patch
cmake -B build && cmake --build build -j

./build/bin/whisper-cli -m ggml-small-wal-ternary-q2_0.bin -f audio.wav -ng

As a service:

bash
./build/bin/whisper-server -m ggml-small-wal-ternary-q2_0.bin \
    --host 127.0.0.1 --port 8178 -t 16 -ng -ml 100000
curl 127.0.0.1:8178/inference -F file=@audio.wav -F response_format=json

Pass -ml 100000. Without it the server forces 60-character segments and splits words in half (fl uttered).

Verify in PyTorch

python
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained("armanibadboy/whisper-small-ternary")
proc  = WhisperProcessor.from_pretrained("armanibadboy/whisper-small-ternary")

Every quantised matrix holds at most 3 distinct values per group of 128 — check it yourself:

python
import torch
W = model.model.encoder.layers[0].fc1.weight
g = W[:, :128]
print(torch.unique(g / g.abs()[g != 0].min()).round())   # -> tensor([-1., 0., 1.])

Speed

Same weights, 66 s of audio, 16 threads, batch 1, Xeon:

Table with columns: fp16, ternary Q2_0
	fp16	ternary Q2_0
file	465 MiB	140 MiB
encoder / 30 s window	578 ms	473 ms
decode / step	6.8 ms	6.0 ms

Limitations

Stated plainly, because a quantisation result without a failure list has not been examined hard enough.

No punctuation. 0.0–0.7 % of utterances contain a comma depending on mode; stock whisper-small manages 63.3 %. Capitals appear on 2 % against 98 %. In timestamp mode a sentence-final period survives on 15 % of utterances — residue from the original model, not learned punctuation. Measured on 300 utterances.
English only. Russian degrades by 12.6×. An fp16 control trained identically degrades the same way, so the cause is monolingual fine-tuning, not ternarisation.
Out of domain, English is roughly twice as bad as on LibriSpeech.
The tied embedding head stays fp16 — 18 % of parameters, 86 of the 140 MiB. A one-shot ternary projection of it, with no dedicated QAT, takes WER from 2.71 % to 1152 %. That shows this projection fails on that tensor, not that the head is incompressible; a learned Q4 head would plausibly bring the file to ~78 MiB.
Load takes 950 ms against 447 ms for fp16 — the encoder expansion at load is an unresolved 826 ms. Three hypotheses were tested and all three were wrong.
Trained and evaluated on read speech (LibriSpeech). Conversational and noisy audio were not measured.

What ternarisation cost

the gap holds at ≈0.37 pp absolute on test-clean and does not narrow;
neither branch reached a floor — both still falling nearly linearly when the runs ended (−0.173 and −0.186 pp per 10k steps);
so "ternary gets to the same place, just later" is unproven. It needs a run to plateau, which was not done.

Training

Table

base	`openai/whisper-small`
steps	80 000 (70k packed LibriSpeech 951 h, then 10k repair with timestamps)
quantised	192 matrices, 198 180 864 weights, alphabet {−1, 0, +1}, group 128
bits per weight	2.125
method	quantisation-aware distillation, straight-through estimator
loss	KL against bf16 teacher + cross-entropy + feature matching

Evaluation protocol

markdown
batch_size     = 16
max_new_tokens = 440
greedy, num_beams = 1

The CPU path was verified independently: all 2620 utterances of test-other through patched whisper.cpp give 7.3754 %, against 7.3817 % in the PyTorch harness — within the noise floor.

Citation

bibtex
@software{aubakirov2026ternarywhisper,
  author = {Aubakirov, Arman},
  title  = {Ternary Whisper: whisper-small at 2.125 bits per weight},
  year   = {2026},
  url    = {https://github.com/AubakirovArman/whisper-ternary}
}

MIT.

whisper-small-ternary

README

Files

Run on CPU

Verify in PyTorch

Speed

Limitations

What ternarisation cost

Training

Evaluation protocol

Citation

Explore FriendliAI today

README

Files

Run on CPU

Verify in PyTorch

Speed

Limitations

What ternarisation cost

Training

Evaluation protocol

Citation