thomaseibner

whisper-large-v3-turbo-us-atc-v3

What is actually new here

This is the first model in this project trained against a verified holdout. Every earlier release, including …-us-atc-v2, was trained on corpora that overlapped their own evaluation sets by 33.7–41.4%.

⚠ Do not read v2's published numbers as a baseline for these. They are not comparable, in three independent ways: different eval sets, a train/eval overlap of 96 of 285 clips, and a scorer whose number-normalisation chain changed on 2026-09-20 (worth ~2.4 WER on its own).

Results

Eval: 285 clips, held out with verified 0% train overlap, stratified by annotator-judged noise.

Table with columns: WER, identifier recall
	WER	identifier recall
v3	20.60	89.48

Table with columns: stratum, clips, WER, ident
stratum	clips	WER	ident
clean	84	15.88	94.85
noisy	130	19.01	89.10
severe	71	27.37	83.16

⚠ 20.60 is the mean of two seeds (42, 43). The published weights are seed 43, which alone scores 20.01. Quote 20.60: taking the better of two draws is an upward-biased estimate of what the checkpoint will do.

Table with columns: seed, pooled WER, ident, clean, noisy, severe
seed	pooled WER	ident		clean	noisy	severe
42	21.19	89.88		17.03	19.25	27.92
43 — published	20.01	89.09		14.72	18.77	26.82

⚠ The two seeds differ by 1.18 pooled WER — about double the spread on the previous corpus (0.49). Adding satellite-field clips made the result noisier across seeds, so a single-seed number from this data is less trustworthy than one from the earlier corpus. Differences below ~1.0 WER here are not readable, and on severe the two seeds differ by 3.06 identifier-recall points on their own.

On a second, fully independent holdout — 35 clips from satellite/GA fields that no model in this project has ever trained on, decoded through the production CTranslate2 path:

Table with columns: WER, ident
	WER	ident
v3	15.23	86.42
v2 (previous release)	19.75	88.89

⚠ v2 still wins identifier recall on that set. If callsign accuracy matters more to you than word accuracy, v2 may remain the better choice. At n=35 neither gap is statistically resolvable.

⚠ What this model outputs, and what a pipeline adds afterwards

Three live transmissions through the deployed model, showing every layer. Everything in capitals below was produced after the model, not by it.

zmp-darwin, 125.500 MHz

Table with columns: layer, text
layer	text
this model	`Descend via the torgy arrival 12 right transition Delta 2191.`
+ text cleaner	`Descend via the TORGY arrival 12R transition Delta 2191.`
+ identifier resolver	airline `DAL` · fix `TORGY` · flight `DAL2191` (0.90) · procedure `TORGY4`

kmsp-dep-se, 124.700 MHz

Table with columns: layer, text
layer	text
this model	`Direct zumbro, FedEx 1618 heavy.`
+ text cleaner	`direct ZMBRO, FedEx 1618 heavy.`
+ identifier resolver	airline `FDX` · fix `ZMBRO` · flight `FDX1618` (0.90)

kmsp-dep-se, 124.700 MHz — a frequency handoff, and a self-correction

Table with columns: layer, text
layer	text
this model	`13485, FedEx 17, er, FedEx 1618 Heavy.`
+ text cleaner	unchanged
+ identifier resolver	frequency `134.850` → ZMP Swinns Valley LOW, flagged `handoff` · flight `FDX1618`

Here the model wrote 13485 — five bare digits, exactly as a controller says a frequency. Turning that into 134.850, recognising which en-route sector owns it, and marking the transmission as a handoff are all downstream work. The model also kept the speaker's self-correction verbatim — "FedEx 17, er, FedEx 1618" — which is correct: the target is what was said, disfluencies included, not a tidied version.

The model heard "zumbro" and "torgy" — the spoken words. ZMBRO and TORGY are FAA waypoint idents, vowel-compressed spellings that are never pronounced as written; 12R is a runway code where "12 right" was said; TORGY4 is a procedure revision number that appears nowhere in the audio at all. A separate resolver produced each, using ADS-B and flight-plan data the model never sees.

This is deliberate, and it is why the numbers above are measured before resolution. A model trained to emit ZMBRO would emit it in San Diego, where the waypoint does not exist. Local nouns belong in a resolver or in decode-time context, not in the weights — so the references this model is scored against record "killer", "coult", "dwight", never KKILR, COULT, DWIYT.

That this is the right target, and not merely our convention, shows up in the live stream: on two separate transmissions this model and the unrelated Qwen model, which share no weights, produced the same spoken spelling as each other —

Table with columns: both models wrote, the chart says
both models wrote	the chart says
`Direct zumbro, FedEx 1618 heavy.`	`ZMBRO`	a waypoint
`…descending via the muscle 4, information delta`	`MUSCL4`	an arrival procedure

Neither model emitted the ident, and neither should: "zumbro" and "muscle" are the sounds in the air, while ZMBRO and MUSCL are vowel-compressed spellings that are never pronounced as written. Two independent models agreeing on the spoken form is the corpus telling you where the boundary belongs — mapping one to the other is a lookup, and a lookup belongs outside the weights.

⚠ These three are illustrations of the layering, not a sample of accuracy. All three happen to be correct. The model's word error rate is ~20%, so roughly one word in five across the eval set is not.

The same transmissions in the application

One departure, five consecutive transmissions, as the live scanner renders them

⚠ This is the application, not the model's output. The sentences are this model's; everything else on those rows was added downstream. ZMBRO, FDX1618, ↑17000, hdg 120°,

text

134.850 ZMP Swinns Valley LOW (handoff)

and ✓ heading matches track all come from the resolver and from ADS-B — compare the tables above to see what the model actually emitted. The badges reading fine-tuned-v3 mark which model transcribed each clip, and the millisecond figures beside them are its decode latency.

Two details worth noticing. The bottom row is greyed and marked suffix at 42% — the pilot said only "1618 heavy" with no airline word, so the pipeline could match a flight number but not confirm an operator, and it says so rather than guessing. And the top row shows the self-correction from the table above surviving all the way to the display: the model heard the stumble, and nothing downstream tidied it.

The same audio, two models

This project runs a second, unrelated model — qwen3-asr-0.6b-us-atc-v1, a full fine-tune of Qwen3-ASR-0.6B — as a shadow lane on the identical live stream. Over one 4-minute window, 27 airline transmissions, both models on the same audio: 14 agreed word for word.

Table with columns: Whisper v3, Qwen shadow
	Whisper v3	Qwen shadow
02:36:59	`Departure, FedEx 1618 heavy, passing 2000 for 7000, heading 120.`	identical
02:37:48	`Direct zumbro, FedEx 1618 heavy.`	identical
02:38:25	`Up to 17000, FedEx 1618 heavy.`	identical
02:39:58	`13485, FedEx` `17, er, FedEx`

⚠ There is no human reference for these clips, so "differs" does not mean "one is wrong." Two of the rows can be settled from outside the audio, and they do not both go the same way:

02:39:58 — Whisper v3 is right. The pilot really did stumble: "FedEx 17, er, FedEx 1618." v3 kept the self-correction; Qwen silently tidied it away. If you are building references or training data, that difference matters more than a WER point — the target is what was said, disfluencies included, and a model that quietly repairs speech produces labels that no longer describe the audio.
02:38:47 — Qwen is right. Delta 2929 appears in the next two transmissions and both models agree on it there, so v3's 2979 is the error. Consistency across nearby transmissions catches things a single clip cannot.

02:40:05 is the failure class worth fearing: not a digit but a different airline — Southwest 1670 against Delta 1170. A downstream resolver would attribute that transmission to the wrong aircraft, and nothing in the sentence looks wrong.

Training

Table

base	`openai/whisper-large-v3-turbo`
method	LoRA, r=16, α=32, dropout 0.05
target modules	`q_proj, k_proj, v_proj, out_proj, fc1, fc2` — all linear, not the usual `q,v`
learning rate	3e-4
schedule	2.711842 epochs = 2,061 optimizer steps, batch 2 × grad-accum 4, bf16
data	1,216 clips × 5 (1 original + 4 augmented) = 6,080 files
hardware

Two recipe choices were measured rather than assumed, each against a step-matched control:

lr 1e-3 → 3e-4 is worth −1.53 WER, confirmed across two seeds. The widely-copied HF PEFT Whisper recipe uses 1e-3; on this corpus that is roughly 100× the full-fine-tune rate and materially too high.
Adapting all linear layers rather than q,v is worth +0.88 identifier recall at no readable WER cost — but only at the corrected learning rate. At lr 1e-3 the same change costs +5.74 WER through overfitting.

⚠ Limitations

One airport's terminal airspace. KMSP plus nearby satellite fields. Phraseology, accents, facility names and traffic mix are local; cross-corpus ATC transfer is known to be poor (published examples run 9.82% in-domain → 57.35% out).
Channel generalisation is largely untested. The main eval has only 1 of 285 clips on a frequency absent from training. A 35-clip channel-disjoint check exists and is included above; it is small.
Adding satellite-field clips traded accuracy between strata. At fixed compute they readably cost clean-audio accuracy (+1.52 WER) while buying noisy-audio accuracy (−0.67). Appending domain data to a small corpus is not neutral.
Not comparable to published ATC WERs. The same model scores 13.46 and 37.62 on "ATCO2" in two papers, differing only by split and normalisation.
Scored on what was said, never on resolved identifiers. References record "killer", "coult", "dwight" — not KKILR, COULT, DWIYT. Identifier resolution is a separate post-processing step and is not part of these numbers.

Scoring

Numbers come from one pinned chain: normalise → fold_numbers → thousands-collapse, reported per noise stratum, with a measured seed spread so that small differences are declared unreadable rather than claimed. Decoding used max_new_tokens=96, repetition_penalty=1.1, greedy, language="en".

Every figure above was re-derived from the two seeds' stored prediction files on 2026-09-21 immediately before publishing, rather than copied forward from a results table. The two runs were first checked to share identical references on all 285 clips, so the per-seed comparison is not confounded by a reference difference.

⚠ That re-derivation disagrees with this project's own results table by 0.04 pooled WER (20.60 vs a recorded 20.64) and 0.13 on severe (27.37 vs 27.50); clean, noisy and identifier recall reproduce exactly. The cause is not established. Both figures are far below the ~1.0 WER readable threshold on this corpus, so nothing in the conclusions moves either way — but the number published here is the one that can be reproduced today from the stored predictions, not the one that could not be.

Intended use

Transcribing live VHF air-traffic-control audio for situational awareness, logging and research.

⚠ Not for operational use. This is a research model on safety-critical speech with a ~20% word error rate. Do not use it for anything a controller or pilot would rely on.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

thomaseibner

Model Tree

Base

openai/whisper-large-v3-turbo

Adapter

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

What is actually new here

Results

Eval: 285 clips, held out with verified 0% train overlap, stratified by annotator-judged noise.

Table with columns: WER, identifier recall
	WER	identifier recall
v3	20.60	89.48

Table with columns: stratum, clips, WER, ident
stratum	clips	WER	ident
clean	84	15.88	94.85
noisy	130	19.01	89.10
severe	71	27.37	83.16

Table with columns: seed, pooled WER, ident, clean, noisy, severe
seed	pooled WER	ident		clean	noisy	severe
42	21.19	89.88		17.03	19.25	27.92
43 — published	20.01	89.09		14.72	18.77	26.82

On a second, fully independent holdout — 35 clips from satellite/GA fields that no model in this project has ever trained on, decoded through the production CTranslate2 path:

Table with columns: WER, ident
	WER	ident
v3	15.23	86.42
v2 (previous release)	19.75	88.89

⚠ v2 still wins identifier recall on that set. If callsign accuracy matters more to you than word accuracy, v2 may remain the better choice. At n=35 neither gap is statistically resolvable.

⚠ What this model outputs, and what a pipeline adds afterwards

Three live transmissions through the deployed model, showing every layer. Everything in capitals below was produced after the model, not by it.

zmp-darwin, 125.500 MHz

Table with columns: layer, text
layer	text
this model	`Descend via the torgy arrival 12 right transition Delta 2191.`
+ text cleaner	`Descend via the TORGY arrival 12R transition Delta 2191.`
+ identifier resolver	airline `DAL` · fix `TORGY` · flight `DAL2191` (0.90) · procedure `TORGY4`

kmsp-dep-se, 124.700 MHz

Table with columns: layer, text
layer	text
this model	`Direct zumbro, FedEx 1618 heavy.`
+ text cleaner	`direct ZMBRO, FedEx 1618 heavy.`
+ identifier resolver	airline `FDX` · fix `ZMBRO` · flight `FDX1618` (0.90)

kmsp-dep-se, 124.700 MHz — a frequency handoff, and a self-correction

Table with columns: layer, text
layer	text
this model	`13485, FedEx 17, er, FedEx 1618 Heavy.`
+ text cleaner	unchanged
+ identifier resolver	frequency `134.850` → ZMP Swinns Valley LOW, flagged `handoff` · flight `FDX1618`

Table with columns: both models wrote, the chart says
both models wrote	the chart says
`Direct zumbro, FedEx 1618 heavy.`	`ZMBRO`	a waypoint
`…descending via the muscle 4, information delta`	`MUSCL4`	an arrival procedure

The same transmissions in the application

One departure, five consecutive transmissions, as the live scanner renders them

⚠ This is the application, not the model's output. The sentences are this model's; everything else on those rows was added downstream. ZMBRO, FDX1618, ↑17000, hdg 120°,

text

134.850 ZMP Swinns Valley LOW (handoff)

The same audio, two models

Table with columns: Whisper v3, Qwen shadow
	Whisper v3	Qwen shadow
02:36:59	`Departure, FedEx 1618 heavy, passing 2000 for 7000, heading 120.`	identical
02:37:48	`Direct zumbro, FedEx 1618 heavy.`	identical
02:38:25	`Up to 17000, FedEx 1618 heavy.`	identical
02:39:58	`13485, FedEx` `17, er, FedEx`

⚠ There is no human reference for these clips, so "differs" does not mean "one is wrong." Two of the rows can be settled from outside the audio, and they do not both go the same way:

02:39:58 — Whisper v3 is right. The pilot really did stumble: "FedEx 17, er, FedEx 1618." v3 kept the self-correction; Qwen silently tidied it away. If you are building references or training data, that difference matters more than a WER point — the target is what was said, disfluencies included, and a model that quietly repairs speech produces labels that no longer describe the audio.
02:38:47 — Qwen is right. Delta 2929 appears in the next two transmissions and both models agree on it there, so v3's 2979 is the error. Consistency across nearby transmissions catches things a single clip cannot.

Training

Table

base	`openai/whisper-large-v3-turbo`
method	LoRA, r=16, α=32, dropout 0.05
target modules	`q_proj, k_proj, v_proj, out_proj, fc1, fc2` — all linear, not the usual `q,v`
learning rate	3e-4
schedule	2.711842 epochs = 2,061 optimizer steps, batch 2 × grad-accum 4, bf16
data	1,216 clips × 5 (1 original + 4 augmented) = 6,080 files
hardware

Two recipe choices were measured rather than assumed, each against a step-matched control:

lr 1e-3 → 3e-4 is worth −1.53 WER, confirmed across two seeds. The widely-copied HF PEFT Whisper recipe uses 1e-3; on this corpus that is roughly 100× the full-fine-tune rate and materially too high.
Adapting all linear layers rather than q,v is worth +0.88 identifier recall at no readable WER cost — but only at the corrected learning rate. At lr 1e-3 the same change costs +5.74 WER through overfitting.

⚠ Limitations

One airport's terminal airspace. KMSP plus nearby satellite fields. Phraseology, accents, facility names and traffic mix are local; cross-corpus ATC transfer is known to be poor (published examples run 9.82% in-domain → 57.35% out).
Channel generalisation is largely untested. The main eval has only 1 of 285 clips on a frequency absent from training. A 35-clip channel-disjoint check exists and is included above; it is small.
Adding satellite-field clips traded accuracy between strata. At fixed compute they readably cost clean-audio accuracy (+1.52 WER) while buying noisy-audio accuracy (−0.67). Appending domain data to a small corpus is not neutral.
Not comparable to published ATC WERs. The same model scores 13.46 and 37.62 on "ATCO2" in two papers, differing only by split and normalisation.
Scored on what was said, never on resolved identifiers. References record "killer", "coult", "dwight" — not KKILR, COULT, DWIYT. Identifier resolution is a separate post-processing step and is not part of these numbers.

Scoring

Intended use

Transcribing live VHF air-traffic-control audio for situational awareness, logging and research.

⚠ Not for operational use. This is a research model on safety-critical speech with a ~20% word error rate. Do not use it for anything a controller or pilot would rely on.

whisper-large-v3-turbo-us-atc-v3

README

What is actually new here

Results

⚠ What this model outputs, and what a pipeline adds afterwards

The same transmissions in the application

The same audio, two models

Training

⚠ Limitations

Scoring

Intended use

Explore FriendliAI today

README

What is actually new here

Results

⚠ What this model outputs, and what a pipeline adds afterwards

The same transmissions in the application

The same audio, two models

Training

⚠ Limitations

Scoring

Intended use