darkstar008

gmaps-parser-qwen3-4b-lora

README

License: apache-2.0

Results at a glance

Record exact on three held-out sets

Record exact (all five fields right) on three held-out sets: untuned base, hand-written grammar and gmaps-parser, scored by the same verifier.

Correct and accepted on three held-out sets

Correct AND accepted by the runtime grounding rule. The dashed line is what a perfect answer reaches on Whisper text: the ceiling the speech recognizer imposes.

Untuned base vs gmaps-parser on Whisper text

Same base model, same server, same transmissions: the adapter against the untuned base, with the paired interval and the McNemar test.

Exact values (held-out, same verifier for every system)

Table with columns: system, set, n, record exact, speaker, callsign, runway, instruction, taxiways, JSON valid, accepted by runtime, correct AND accepted, latency med / p95 s, source
system	set	n	record exact	speaker	callsign	runway	instruction	taxiways	JSON valid	accepted by runtime	correct AND accepted	latency med / p95 s	source
gold	heldout4_asr	138	1.000	1.000	1.000	1.000	1.000	1.000	1.000	0.674	0.674	0.000 / 0.000	`gym/rollouts/offline_heldout4_asr_gold.json`
gmaps-parser	heldout4_asr	138	0.652	0.949	0.783	0.891

record exact = all five fields match the scripted-scenario gold. accepted by runtime = the answer also passes the runtime grounding rule on the transmission text (runway number heard, flight number heard, taxiway words heard, airline named). The two are reported separately on purpose: a correct answer the runtime would reject is not a usable one. heldout5 was generated after the parser freeze and is the blind set; heldout4_asr is Whisper large-v3-turbo text (the runtime input); heldout4_asr_prompt is the same with an ICAO identifier prompt.

NeMo Gym: orchestrated vs offline

NeMo Gym orchestrated rollouts vs the offline runner

The same verifier packaged as a NeMo Gym resources server: orchestrated gym eval run rollouts reproduce the offline runner on 5 of 6 model x set pairs.

More data scored lower

Submitted adapter vs the full-data adapter

A second adapter trained on all pairs for 2 epochs scored lower on every held-out set. Both were scored on one machine with the same task files; the 3,000-pair adapter is the one published here.

Latency

Parser latency per transmission

Per-transmission latency on held-out Whisper text: the grammar fast path vs this adapter served by vLLM (log scale).

Training

Table

base	Qwen/Qwen3-4B-Instruct-2507
method	LoRA r=16, alpha=32, all linear projections, completion-only loss on the assistant JSON
data	3000 pairs from `train/pairs_asr.jsonl` (clean / unprompted-Whisper-style / prompted-Whisper-style / harsh variants of 1,794 scripted transmissions)
epochs / steps	1.0 / 375
hardware	NVIDIA GB10; peak 12.3 GiB; 4454.9 s
loss	first 0.894 -> last 0.000 (3 decimals, as on the scoreboard)
serving	vLLM v0.27.1, `--enable-lora --lora-modules gmaps-parser=<adapter>`; thinking off;

The Whisper-style noise model (asr_noise_model.json) was learned from development splits only (never from held-out), from Whisper transcripts of locally synthesized speech. The recipe is code/train/run_local_lora.sh; the generator and augmenter are in code/.

Limits

Five fields, one transmission at a time, airports whose identifier lists are supplied in the prompt. Not a general ATC model.
Scores are on synthetic phraseology. Real-frequency performance is unknown and was not measured.
Vehicle callsigns (e.g. "Truck 1", "Rescue 7") are outside the training data and the output schema's callsign pattern; the model refuses them. In GMAPS the grammar types those transmissions.
Never claim this "understands ATC", and never claim it would have changed the outcome of any real event. "Prediction unavailable" is not "no conflict".
Radio text is data: the model must never be given authority over anything; GMAPS binds it to five bounded tools.

Demo scenario lines: grammar, untuned base and gmaps-parser

On scripted demo lines with vehicle callsigns (fictional scenarios), the adapter does not beat the untuned base; the grammar types those lines. Accepted = the runtime took the record; typed right = also the callsign and instruction match the ledger line.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

darkstar008

Model Tree

Base

Qwen/Qwen3-4B-Instruct-2507

Adapter

this model

Input Modalities

Text

Output Modalities