Results — head-to-head vs the teacher (our measurement)
This is the code-switch-optimized checkpoint (stacked KD: CE + logit-KL + hidden-state KD, then code-switch
upweighting). All rows below are our own runs of both models with one identical protocol (greedy,
fp16, no language hint) so the comparison is fair — absolute numbers may differ from Qwen's official figures
because we can't reproduce their exact text normaliser.
Table with columns: metric (set), this 0.3B-CS (562M), teacher 0.6B (938M)| metric (set) | this 0.3B-CS (562M) | teacher 0.6B (938M) |
|---|
| zh-CER — CommonVoice-zh, 500 clips (strip punct, CJK+alnum, t2s) | 0.0610 | 0.0445 |
| CS-MER — Taiwan code-switch, 60 clips (mixed CJK-char/en-word)¹ | 0.0765 | ref² |
| en-WER — CommonVoice-en, 200 clips | ~0.080 | 0.0233 |
| total params | 562 M | 938 M |
| GGUF q8_0 size | 735 MB | 961 MB |
| Jetson Nano gen1 RTF (q8_0, 13.5 s clip) | 0.79 (real-time) | 1.3 |
| on-device decode | 6.7 s (14 layers) | 13.8 s (28 layers) |
¹ Before code-switch upweighting this checkpoint's CS-MER was 0.128; upweighting in-corpus CS clips to 35 %
cut it to 0.0765 (−40 %) at no pure-zh cost (0.0615 → 0.0610) but raised pure-en WER 0.048 → ~0.080
— a deliberate code-switch/English trade. ² CS-MER references are teacher-generated, so the teacher is the
ceiling by construction (not a fair teacher number).
Honest gap: on clean read-speech (CommonVoice-zh) the 0.3B is ~37 % relatively behind the 0.6B teacher
(0.061 vs 0.044) — the cost of 40 % fewer params and ~197 h of training data vs the teacher's vastly larger
corpus. The cleaner official sets (Fleurs-zh, AISHELL-2, WenetSpeech) weren't runnable from our box
(download-blocked / gated). The 0.3B's wins are size, real-time-on-Nano, and code-switch — not raw zh accuracy.
CER/WER are on different sets and must not be compared across non-comparable benchmarks. The
deployed streaming X-ASR transducer is still lower on pure zh-CER (~0.068); this model's value is being a
bilingual LLM-ASR with code-switch, native Traditional output, 40 % smaller, and real-time on the Nano.
Files
qwen3-asr-0.3b-q8_0.gguf — GGUF (q8_0) for rapidspeech.cpp / qwen3-asr.cpp (Nano / llama.cpp-style engines)model.safetensors, config.json, tokenizer — PyTorch (transformers) checkpointscripts/ — the full reproduction pipeline (see below)
Usage
GGUF (edge, recommended on Nano): the block_count=14 flows through the GGUF, so the existing Qwen3-ASR
engine builds the right decoder with no rebuild:
rs-asr-offline -m qwen3-asr-0.3b-q8_0.gguf -w clip_16k.wav # rapidspeech.cpp
qwen3-asr-cli -m qwen3-asr-0.3b-q8_0.gguf -f clip_16k.wav -t 4 # qwen3-asr.cpp (Nano, RTF ~0.8)
Engines: vieenrose/qwen3-asr.cpp (standalone)
· vieenrose/RapidSpeech.cpp (integrated).
PyTorch (transformers from source — Qwen3-ASR is only in main):
pip install git+https://github.com/huggingface/transformers.git
import torch, soundfile as sf
from transformers import Qwen3ASRForConditionalGeneration, AutoProcessor
proc = AutoProcessor.from_pretrained("Luigi/qwen3-asr-0.3b")
model = Qwen3ASRForConditionalGeneration.from_pretrained("Luigi/qwen3-asr-0.3b", dtype=torch.float16).to("cuda").eval()
a, sr = sf.read("clip_16k.wav")
conv = [{"role":"user","content":[{"type":"audio","audio":a},{"type":"text","text":""}]}]
inp = proc.apply_chat_template(conv, add_generation_prompt=True, tokenize=True, return_dict=True,
return_tensors="pt", sampling_rate=16000)
inp = {k:(v.to("cuda").half() if v.is_floating_point() else v.to("cuda")) for k,v in inp.items()}
out = model.generate(**inp, max_new_tokens=96, do_sample=False)
print(proc.batch_decode(out[:, inp["input_ids"].shape[1]:], skip_special_tokens=True)[0])
Full reproduction process
All scripts are in scripts/ here. The training data is NOT published (the zh-TW audio is YouTube content
— copyright / platform ToS); the pipeline below reconstructs an equivalent corpus from your own sources.
0. Environment
1. Data — zh-TW (teacher-labeled, the hard part)
The published 0.6B is the teacher; it labels raw audio, which also gives native-Traditional targets.
- Re-download the source Taiwan-Mandarin audio from YouTube (
scripts/scale_corpus.py, IDs in
data/youtube_ids.txt): yt-dlp (itag-140) → 16 kHz mono → fixed 16 s windows.
- Teacher-label each window with Qwen3-ASR-0.6B (greedy), then apply OpenCC
s2twp → Traditional
Chinese targets (the teacher's raw script is inconsistent; s2twp is a deterministic post-step).
- ~135 videos → ~145 h, 36 105 segments. Neither the audio nor the labels are published (copyright);
supply your own Taiwan-Mandarin source list to
scale_corpus.py to reconstruct an equivalent corpus.
2. Data — English balance (anti-forgetting)
scripts/english_prep3.py: LibriSpeech train-clean-100 via HF datasets streaming with
Audio(decode=False) + soundfile (bypasses torchcodec) → 16 kHz wav + gold transcripts, ~30 h.- A 30 % English mix during training prevents catastrophic forgetting of English.
3. Student construction
- Start from the teacher; prune the 28-layer Qwen3 decoder to 14 layers, "drop-middle" (keep first 7 +
last 7 — naive first-N or last-N truncation collapses the model). Re-index
self_attn.layer_idx.
- Freeze the audio encoder + projector + embeddings; train only the 14 decoder layers + final norm
(220 M trainable). The encoder is one-shot and already good; the decoder is what we shrink.
- Note: a fresh-pruned decoder has CER ≈ 1.0 — recovery requires distillation (it is not a "prune + heal").
4. KD training (scripts/train_student_full.py)
- Loss: cross-entropy on the teacher's Traditional labels (sequence-level KD). Label masking: build the prompt
(
add_generation_prompt=True) then manually append tokenizer(text)+eos and mask the prompt with -100
(the chat template does not append assistant text).
- Optimizer: AdamW lr 2e-5, 100-step linear warmup, grad-clip 0.5, nan-guard (skip non-finite loss),
grad-accum 4, EN_PROB 0.30, 8000 steps (~23 min on a single GB10). Save best-by-zh-CER checkpoint.
- Curve (v1, CE-only): zh-CER 0.997 → 0.201 (1k) → 0.0865 (200-clip eval).
- Refinement (this checkpoint),
scripts/train_03b_kd.py + scripts/train_03b_hskd.py: warm-start the v1
student and add logit-KL (KL of student vs teacher softened logits, T=2) then hidden-state KD
(normalised MSE of student decoder layer l vs teacher layer 2l). Loss 0.4·CE + 0.4·KL + 0.2·HS, cosine
LR 1e-5→1e-6. This took zh-CER 0.0865 → 0.0813 → 0.0722, then scaling the zh corpus 145 h → ~197 h reached 0.0702 (teacher 0.065). Both teacher+student in fp32
(mixed fp16/fp32 causes conv dtype errors); eval only on the 200-clip set (the 30-clip train-eval is noisy).
5. Export to GGUF (scripts/remap_student_to_thinker.py + convert_qwen3_asr_to_gguf.py)
- Remap the transformers checkpoint (
model.*) back to Qwen thinker.* (inverse of the HF converter), dup the
tied lm_head. Fix config.json layer_types to length 14.
convert_qwen3_asr_to_gguf.py --hf-dir … --output … --quant q8_0. Full 151 705 vocab kept.
6. Deploy on Jetson Nano gen1
- The engine (cuFFT batched mel + sm_53 CUDA-10.2 build) reads
qwen3.block_count=14 and builds the 14-layer
decoder automatically — no rebuild. Measured RTF 0.957 (q8_0, 13.5 s clip).
Training hyper-parameters
Table | |
|---|
| teacher | Qwen/Qwen3-ASR-0.6B |
| student | 14-layer decoder (drop-middle of 28), encoder frozen, full vocab |
| data | ~145 h zh-TW (YouTube, teacher+s2twp) + 30 h en (LibriSpeech) |
| objective | CE + logit-KL + hidden-state KD (stacked) |
| lr / sched / steps | 2e-5 / linear-warmup-100 / 8000 |
| batch | 1 × grad-accum 4, 30 % English |
| precision | fp32 train, q8_0 GGUF |
Limitations
- Small-student gap: zh-CER 0.0610 vs teacher 0.0445 on CommonVoice-zh (~37 % relative) — a real gap (40 % fewer params, ~197 h training vs the teacher's far larger corpus).
- Code-switch/English trade: this checkpoint is tuned for code-switch (CS-MER 0.128→0.0765); pure-English regressed (en-WER ~0.048→~0.080). If you need max pure-English, the pre-CS checkpoint is in the git history.
- Couldn't benchmark on Fleurs-zh / AISHELL-2 / WenetSpeech (download-blocked / gated from our box); numbers above are CommonVoice-zh + an internal Taiwan code-switch set.
- English is only ~30 h (anti-forgetting, not parity); scale it for stronger English.
- Trained on broadcast/news/lecture Taiwan YouTube; conversational/attendant domains under-represented.
- zh-TW labels via teacher +
s2twp, not human gold.
Ethics / data
The training corpus is not published. The zh-TW audio is YouTube content; neither the audio nor its
derived labels are redistributed (copyright / platform ToS). Only the reproduction scripts are shared, so
others can build an equivalent corpus from their own sources. English is public LibriSpeech (referenced, not
re-hosted).
License
Apache-2.0 (inherits the Qwen3-ASR-0.6B base license).
