Evaluation
The candidate was evaluated with the public Open ASR Leaderboard code pinned
at commit d1e99b25524814332d6868a5645e568670834cfb. On a deterministic paired
screen using the first 200 examples from each of the seven public English
short-form datasets, the macro WER was 5.0100, compared with 5.0600 for
greedy decoding of the same base checkpoint.
Table with columns: Decoding, Paired 7-set macro WER| Decoding | Paired 7-set macro WER |
|---|
| Greedy | 5.0600 |
| Beam 2 | 5.0286 |
| Beam 4 | 5.0100 |
| Beam 6 | 5.0229 |
| Beam 8 | 5.0214 |
The screen is a candidate-selection measurement, not a substitute for the
leaderboard's full public and hidden evaluation. Full public evaluation and
leaderboard submission are in progress.
Usage
Load the checkpoint with Transformers and call generate normally. Its saved
generation configuration selects four-beam decoding automatically. Callers
that explicitly pass generation arguments can override the saved setting.
Provenance
- Base revision:
bcd2b5b7f32b480ab5790554cfa8347f246a14f3
- Base license: Apache-2.0
- Weight changes: none
- Processor changes: none
- Generation change:
num_beams=4