Research
How to check a transcript score before accepting a delivery
Ask for the clip list, the reference text you’ll score against, the delivered transcripts and the scoring settings. Word error rate counts words changed, missed or added per 100 words of reference text. A low score can still hide mistakes that matter for the job, such as a missed minus sign or a word placed in the wrong part of a recording. Check those details before accepting the delivery. How word error rate works
Check whether words are in the right part of the recording
Say a reference has two segments, turn left and then stop. The delivery says turn left then and stop. Joining the text gives 0% word error because all four words are in order. Scoring the segments separately gives 50% because then is added to the first segment and missing from the second. Score each segment on its own if the boundaries matter for the work. Segment example
Keep silent clips in the test
In a fictional test, system A had 0% word error on two speech clips and system B had 12.5%. We added one verified silent clip. A produced two words on that clip and its score rose to 25%. B stayed at 12.5%. When the delivery includes silence, report how many verified silent clips contain unwanted text. Mark missing references separately so they don’t get counted as silence. Silent-clip example
Read numbers in the original text
In one test, the scoring rules treated offset -5 meters and offset 5 meters as identical. They also treated offset 1.5 meters and offset 15 meters as identical. In a separate Whisper test, the first cleanup kept the minus sign from “negative one point five.” A second cleanup dropped it and the same unsigned answer went from 33.3% word error to 0%. When numbers matter, read them in the original texts and record how many times each text is cleaned. Punctuation test · Whisper test
Score the same clips again when rules change
We scored two ARK-ASR models’ saved transcripts on the same 18,760 GigaSpeech clips. With the original reference transcripts, an Open ASR scoring update raised the larger model’s word error rate from 6.93% to 15.11% and the smaller model’s from 7.39% to 15.50%. The new rules counted punctuation markers such as <COMMA> as words. With cleaned reference transcripts for those same clips, the models scored 6.93% and 7.39% to two decimal places under both sets of rules. Score comparison
Recalculate both scores on the same clips and references when the scoring rules change. In a separate fictional example, an updated scorer returned the previous result from a saved file until we explicitly told it to recalculate. Check that the scorer ran again before comparing the results. Saved-score test
See how much one recording changes a close result
Some recordings supply many clips. In the GigaSpeech test, we left out one whole recording at a time and compared the models on the clips that remained. With cleaned references and the newer rules, the larger model led by 0.46 percentage points over all 131 recordings. Its lead ranged from 0.43 to 0.48 points when we left one recording out. Use this check if you need to know whether a single recording decides a close comparison. Recording test