Read the article · Download the evidence (Markdown)

Evidence for checking transcript scores

The examples below use invented text or aggregate results from public benchmark submissions. Word error rate (WER) divides total insertions, deletions and substitutions by the number of reference words. Multiply the result by 100 to express it as a percentage. JiWER scoring and alignment

Segment example

The reference segments are turn left and then stop. The delivered segments are turn left then and stop. Across the separate segments, the scorer counts an insertion and a deletion for then. That is two errors over four reference words, or 50%. Joining the segments gives turn left then stop on both sides, with zero errors over four words. This compares text in the supplied segments. Verifying when the words occur in the recording requires checking the audio and timestamps. JiWER alignment

Silent-clip example

These are pooled rates for invented clips. The two speech clips supply eight reference words. The third clip is verified silence and contributes zero reference words.

Clips scoredSystem A word errorsSystem B word errorsReference wordsA WERB WER
Two speech clips0180%12.5%
Two speech clips and one silent clip21825%12.5%

A produced two words on the silent clip. B left it blank. Report verified silent clips with unwanted words separately from the pooled word error rate. Give clips with missing reference text their own status. JiWER 4 empty-reference rules

Punctuation test

JiWER 4.0.0 was run on two invented text pairs with optional punctuation removal applied to both sides. offset -5 meters and offset 5 meters become identical. So do offset 1.5 meters and offset 15 meters. Both comparisons give 0% WER after the transform. The original numeric fields differ in both pairs. JiWER punctuation removal

Whisper test

In a separate fictional example, Whisper’s pinned English normalizer transforms offset negative one point five meters to offset -one.5 meters. The fixed prediction after one cleanup is offset one.5 meters. The texts differ by one word out of three, or 33.3% WER. A second pass transforms the saved reference to offset one.5 meters. Against the same prediction the error count becomes zero. This replay tests text scoring. Whisper release · Pinned English normalizer

Score comparison

Match the two models’ saved outputs and the cleaned GigaSpeech split by clip ID. The two model submissions contain the same clip IDs and agree on the original reference for each clip. The intersection with the cleaned split contains 18,760 clips. Score that same set with the original and cleaned references. Keep each model’s saved prediction text fixed across scorers. Each revision supplies its English normalization. Alignment uses Kaldialign 0.12.0 with compound merging enabled. Add errors and reference words across the matched clips before dividing.

The scoring revisions are edd68424736537b6d0191be325ae3605c33de284 and f58d58a3f98eb9d5c2dd284ce85954b3cd801759. Earlier scorer · Updated scorer

ReferencesModelEarlier errors / wordsEarlier WERUpdated errors / wordsUpdated WER
OriginalARK-ASR-3B27,504 / 397,1416.93%65,917 / 436,32015.11%
OriginalARK-ASR-0.6B29,339 / 397,1417.39%67,608 / 436,32015.50%
CleanedARK-ASR-3B27,506 / 397,1436.93%27,506 / 397,1436.93%
CleanedARK-ASR-0.6B29,341 / 397,1437.39%29,336 / 397,1437.39%

On the original 3B GigaSpeech manifest, removing GigaSpeech’s four documented punctuation marker types before the newer normalization restores the earlier reference and prediction tokens across all 19,898 rows. The cleaned 0.6B rates round to the same percentage, with five fewer errors under the updated rules. These are replays of saved submissions from one publisher. They establish the effect on these outputs under the pinned scoring settings. GigaSpeech text preprocessing

Pinned 3B submission · Pinned 0.6B submission · Pinned cleaned references

Saved-score test

The fictional reference is we [need to leave] now. The prediction is we now. The earlier scorer removes the three bracketed words and scores the pair at 0% WER. With the score file retained, the updated scorer returns 0% and aligns zero pairs. When recalculation is requested, it aligns one pair, keeps the bracketed words as reference words and returns three deletions over five words, or 60%. The merged change documents the recalculation option. Open ASR scoring change

Recording test

Group the 18,760 matched clips by the original recording encoded in the GigaSpeech clip ID. For each of the 131 groups, omit all its clips from both systems and recalculate the gap using the remaining reference words. The gap is 0.6B WER minus 3B WER. With cleaned references and the newer scorer, the full-set gap is 100 × (29,336 − 27,506) / 397,143, or 0.460791 percentage points. The 131 omission results range from 0.430797 to 0.478149 percentage points. Every omission retains the same order. This checks sensitivity to one recording at a time in the matched set. GigaSpeech recording and segment identifiers