LayerNorm

Qwen3-ASR-0.6B-hf-hqq-int4

README

License: apache-2.0

Usage

bash
# runtime deps for loading/transcribing (this model only needs transformers + torch + hqq)
pip install transformers==5.15.1 torch==2.13.0 hqq==0.2.8.post1

# analysis dep (compare_baseline.py computes CER):
pip install jiwer

Transcribe a clip (recommended path — reconstructs the HQQ layers correctly):

python
import sys
sys.path.insert(0, "scripts")
from infer_hqq_int4 import reconstruct_model
import torch
from transformers import AutoProcessor

checkpoint_dir = "."   # this repo
processor = AutoProcessor.from_pretrained(checkpoint_dir)
model = reconstruct_model(checkpoint_dir)   # restores the HQQLinear layers
model.eval()

inputs = processor.apply_transcription_request(
    audio="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav",
).to(model.device, model.dtype)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=256)
gen = out[:, inputs["input_ids"].shape[1]:]
print(processor.decode(gen, return_format="transcription_only")[0])

Evaluation vs. baseline (bf16)

Measured on an RTX 3060 12GB (CUDA 13.3, Ampere, compute 8.6) on a real Chinese corpus (AISHELL-1 short clips + a ~47 s long clip + an 8 dB-noise clip + the official demo clip, 10 samples total). CER = character error rate (CJK/latin alphanumerics only); latencies are the median of 3 generate calls.

Table with columns: Metric, bf16 (baseline), HQQ-INT4, Δ
Metric	bf16 (baseline)	HQQ-INT4	Δ
Checkpoint size (MB)	1503.3	650.7	−56.7%
GPU memory (MB)	1492.5	643.1	−56.9%
Mean CER vs. bf16	—	0.0063	−
CER vs. AISHELL ground truth	0.0231	0.0157	−
Mean latency (s)	0.90	1.76	×1.9
Short-clip median latency (s)	0.60	1.07	×1.8
Long-clip (~47 s) latency (s)	3.69	7.23	×2.0

Takeaways

Nearly lossless accuracy. Across all samples the quantization-degradation CER vs. the bf16 model is ~0.006 (a single character difference on the noisy clip). Absolute accuracy on these clean Chinese clips is actually slightly better than bf16 (CER 0.0157 vs 0.0231).
~57% smaller and ~57% less GPU memory — the main win. Latency grows (~1.8–2×) on consumer Ampere GPUs, a typical quantized-decode trade-off.

Reproduce the numbers with scripts/compare_baseline.py (needs the local Chinese test set described in scripts/compare_baseline.py).

Reproduction

bash
# 1) Quantize the base model to HQQ-INT4 (default output = this repo root)
python scripts/quantize_hqq_int4.py --base Qwen/Qwen3-ASR-0.6B-hf

# 2) Verify it reloads and transcribes correctly
python scripts/infer_hqq_int4.py --audio <some.wav>

# 3) Compare against the bf16 baseline (Chinese test set)
python scripts/compare_baseline.py

Citation

bibtex
@article{Qwen3-ASR,
  title={Qwen3-ASR Technical Report},
  author={Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo,
          Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, Junyang Lin},
  journal={arXiv preprint arXiv:2601.21337},
  year={2026}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

LayerNorm

Model Tree

Base

Qwen/Qwen3-ASR-0.6B-hf

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality