# How to check a transcript score before accepting a delivery

Ask for the clip list, the reference text you’ll score against, the delivered transcripts and the scoring settings. Word error rate counts words changed, missed or added per 100 words of reference text. A low score can still hide mistakes that matter for the job, such as a missed minus sign or a word placed in the wrong part of a recording. Check those details before accepting the delivery. [How word error rate works](https://jitsi.github.io/jiwer/usage/#word-error-rate)

## Check whether words are in the right part of the recording

Say a reference has two segments, `turn left` and `then stop`. The delivery says `turn left then` and `stop`. Joining the text gives 0% word error because all four words are in order. Scoring the segments separately gives 50% because `then` is added to the first segment and missing from the second. Score each segment on its own if the boundaries matter for the work. [Segment example](https://sapien1.com/research/transcript-score-acceptance/evidence.html#segment-example)

## Keep silent clips in the test

In a fictional test, system A had 0% word error on two speech clips and system B had 12.5%. We added one verified silent clip. A produced two words on that clip and its score rose to 25%. B stayed at 12.5%. When the delivery includes silence, report how many verified silent clips contain unwanted text. Mark missing references separately so they don’t get counted as silence. [Silent-clip example](https://sapien1.com/research/transcript-score-acceptance/evidence.html#silent-clip-example)

## Read numbers in the original text

In one test, the scoring rules treated `offset -5 meters` and `offset 5 meters` as identical. They also treated `offset 1.5 meters` and `offset 15 meters` as identical. In a separate Whisper test, the first cleanup kept the minus sign from “negative one point five.” A second cleanup dropped it and the same unsigned answer went from 33.3% word error to 0%. When numbers matter, read them in the original texts and record how many times each text is cleaned. [Punctuation test](https://sapien1.com/research/transcript-score-acceptance/evidence.html#punctuation-test) · [Whisper test](https://sapien1.com/research/transcript-score-acceptance/evidence.html#whisper-test)

## Score the same clips again when rules change

We scored two ARK-ASR models’ saved transcripts on the same 18,760 GigaSpeech clips. With the original reference transcripts, an Open ASR scoring update raised the larger model’s word error rate from 6.93% to 15.11% and the smaller model’s from 7.39% to 15.50%. The new rules counted punctuation markers such as `<COMMA>` as words. With cleaned reference transcripts for those same clips, the models scored 6.93% and 7.39% to two decimal places under both sets of rules. [Score comparison](https://sapien1.com/research/transcript-score-acceptance/evidence.html#score-comparison)

Recalculate both scores on the same clips and references when the scoring rules change. In a separate fictional example, an updated scorer returned the previous result from a saved file until we explicitly told it to recalculate. Check that the scorer ran again before comparing the results. [Saved-score test](https://sapien1.com/research/transcript-score-acceptance/evidence.html#saved-score-test)

## See how much one recording changes a close result

Some recordings supply many clips. In the GigaSpeech test, we left out one whole recording at a time and compared the models on the clips that remained. With cleaned references and the newer rules, the larger model led by 0.46 percentage points over all 131 recordings. Its lead ranged from 0.43 to 0.48 points when we left one recording out. Use this check if you need to know whether a single recording decides a close comparison. [Recording test](https://sapien1.com/research/transcript-score-acceptance/evidence.html#recording-test)
