# Evidence for checking transcript scores

The examples below use invented text or aggregate results from public benchmark submissions. Word error rate (WER) divides total insertions, deletions and substitutions by the number of reference words. Multiply the result by 100 to express it as a percentage. [JiWER scoring and alignment](https://jitsi.github.io/jiwer/usage/#word-error-rate)

## Segment example

The reference segments are `turn left` and `then stop`. The delivered segments are `turn left then` and `stop`. Across the separate segments, the scorer counts an insertion and a deletion for `then`. That is two errors over four reference words, or 50%. Joining the segments gives `turn left then stop` on both sides, with zero errors over four words. This compares text in the supplied segments. Verifying when the words occur in the recording requires checking the audio and timestamps. [JiWER alignment](https://jitsi.github.io/jiwer/usage/#alignment)

## Silent-clip example

These are pooled rates for invented clips. The two speech clips supply eight reference words. The third clip is verified silence and contributes zero reference words.

| Clips scored | System A word errors | System B word errors | Reference words | A WER | B WER |
| --- | ---: | ---: | ---: | ---: | ---: |
| Two speech clips | 0 | 1 | 8 | 0% | 12.5% |
| Two speech clips and one silent clip | 2 | 1 | 8 | 25% | 12.5% |

A produced two words on the silent clip. B left it blank. Report verified silent clips with unwanted words separately from the pooled word error rate. Give clips with missing reference text their own status. [JiWER 4 empty-reference rules](https://jitsi.github.io/jiwer/#a-note-on-empty-references)

## Punctuation test

JiWER 4.0.0 was run on two invented text pairs with optional punctuation removal applied to both sides. `offset -5 meters` and `offset 5 meters` become identical. So do `offset 1.5 meters` and `offset 15 meters`. Both comparisons give 0% WER after the transform. The original numeric fields differ in both pairs. [JiWER punctuation removal](https://jitsi.github.io/jiwer/reference/transforms/#transforms.RemovePunctuation)

## Whisper test

In a separate fictional example, Whisper’s pinned English normalizer transforms `offset negative one point five meters` to `offset -one.5 meters`. The fixed prediction after one cleanup is `offset one.5 meters`. The texts differ by one word out of three, or 33.3% WER. A second pass transforms the saved reference to `offset one.5 meters`. Against the same prediction the error count becomes zero. This replay tests text scoring. [Whisper release](https://github.com/openai/whisper/releases/tag/v20250625) · [Pinned English normalizer](https://github.com/openai/whisper/blob/31243bad24cc746f07d4c8bfdd2d974872cb1803/whisper/normalizers/english.py)

## Score comparison

Match the two models’ saved outputs and the cleaned GigaSpeech split by clip ID. The two model submissions contain the same clip IDs and agree on the original reference for each clip. The intersection with the cleaned split contains **18,760 clips**. Score that same set with the original and cleaned references. Keep each model’s saved prediction text fixed across scorers. Each revision supplies its English normalization. Alignment uses Kaldialign 0.12.0 with compound merging enabled. Add errors and reference words across the matched clips before dividing.

The scoring revisions are `edd68424736537b6d0191be325ae3605c33de284` and `f58d58a3f98eb9d5c2dd284ce85954b3cd801759`. [Earlier scorer](https://github.com/huggingface/open_asr_leaderboard/tree/edd68424736537b6d0191be325ae3605c33de284) · [Updated scorer](https://github.com/huggingface/open_asr_leaderboard/tree/f58d58a3f98eb9d5c2dd284ce85954b3cd801759)

| References | Model | Earlier errors / words | Earlier WER | Updated errors / words | Updated WER |
| --- | --- | ---: | ---: | ---: | ---: |
| Original | ARK-ASR-3B | 27,504 / 397,141 | 6.93% | 65,917 / 436,320 | 15.11% |
| Original | ARK-ASR-0.6B | 29,339 / 397,141 | 7.39% | 67,608 / 436,320 | 15.50% |
| Cleaned | ARK-ASR-3B | 27,506 / 397,143 | 6.93% | 27,506 / 397,143 | 6.93% |
| Cleaned | ARK-ASR-0.6B | 29,341 / 397,143 | 7.39% | 29,336 / 397,143 | 7.39% |

On the original 3B GigaSpeech manifest, removing GigaSpeech’s four documented punctuation marker types before the newer normalization restores the earlier reference and prediction tokens across all 19,898 rows. The cleaned 0.6B rates round to the same percentage, with five fewer errors under the updated rules. These are replays of saved submissions from one publisher. They establish the effect on these outputs under the pinned scoring settings. [GigaSpeech text preprocessing](https://github.com/SpeechColab/GigaSpeech/blob/44289e4bb6d65291456cf94de495ea83668d1df1/README.md#text-pre-processing)

[Pinned 3B submission](https://huggingface.co/datasets/Edge0/ark-asr-3b-open-asr-leaderboard-results/blob/b98599d8b525251956c36fd237b0c30af18fea90/MODEL_AutoArk-AI-ARK-ASR-3B_DATASET_hf-audio-open-asr-leaderboard_gigaspeech_test.jsonl) · [Pinned 0.6B submission](https://huggingface.co/datasets/Edge0/ark-asr-open-asr-leaderboard-results/blob/1419570c86ca7599cb25efedee2361de38a77ff4/MODEL_AutoArk-AI-ARK-ASR-0.6B_DATASET_hf-audio-open-asr-leaderboard_gigaspeech_test.jsonl) · [Pinned cleaned references](https://huggingface.co/datasets/hf-audio/open-asr-leaderboard/tree/de79e57adf9983993bcc72586d7313460825f5eb/gigaspeech_cleaned)

## Saved-score test

The fictional reference is `we [need to leave] now`. The prediction is `we now`. The earlier scorer removes the three bracketed words and scores the pair at 0% WER. With the score file retained, the updated scorer returns 0% and aligns zero pairs. When recalculation is requested, it aligns one pair, keeps the bracketed words as reference words and returns three deletions over five words, or 60%. The merged change documents the recalculation option. [Open ASR scoring change](https://github.com/huggingface/open_asr_leaderboard/pull/246)

## Recording test

Group the 18,760 matched clips by the original recording encoded in the GigaSpeech clip ID. For each of the 131 groups, omit all its clips from both systems and recalculate the gap using the remaining reference words. The gap is 0.6B WER minus 3B WER. With cleaned references and the newer scorer, the full-set gap is `100 × (29,336 − 27,506) / 397,143`, or **0.460791 percentage points**. The 131 omission results range from **0.430797 to 0.478149 percentage points**. Every omission retains the same order. This checks sensitivity to one recording at a time in the matched set. [GigaSpeech recording and segment identifiers](https://github.com/SpeechColab/GigaSpeech/blob/44289e4bb6d65291456cf94de495ea83668d1df1/README.md#metadata-walkthrough)
