Skip to main content

Menu

Research

Reproduce and compare corpus chrF scores

Language: English

To reproduce a corpus chrF score, retain the metric name, version, options, and exact inputs. Save the prediction file and the ordered list of reference files with their checksums. Keep stable segment IDs or an equivalent record of the alignment between predictions and every reference file.

The examples below use invented strings, two segments, and SacreBLEU 2.6.0's default chrF2 settings.

Reproduce the reported order

SacreBLEU selects a reference separately for each prediction. It takes the counts from the reference with the highest sentence score and keeps the first reference on an exact tie. Corpus chrF2 sums the selected n-gram counts before computing the score.

In the first example, both references tie at 6.25 for system A's first prediction. The two references have different n-gram counts. Swapping the files changes A's corpus score from 79.03 to 39.48. System B stays at 64.31, so the winner reverses. The API and CLI give the same result, and all four runs have an identical metric signature. Use the ordered files to verify the reported value.

Compare both systems under each plausible order

If the original order is uncertain, record which orders are plausible and score both systems under each one. Calculate the difference within each run before summarizing the range of differences.

In a separate example with three reference files, A's scores range from 22.67 to 79.03 and B's from 16.33 to 43.33. The individual ranges overlap. Across all six orders shared by the systems, A leads by 6.33 to 35.69 points, calculated from unrounded scores. If plausible orders give different winners, report that dependence. If the winner stays the same, report which shared orders support that conclusion. Recover the original order to verify the original numerical result.

Treat a reference revision as a new evaluation input

A new reference can improve a sentence score while lowering corpus chrF2. In the third example, adding one reference document raises A's first sentence score from 6.25 to 6.94 and leaves the second at 100. A's sentence average rises, while its corpus score drops from 79.03 to 43.60. B stays at 64.31. The added first-segment reference has the strictly higher sentence score, and both orders lead to the same corpus result. Its selected counts change both matches and denominators for the corpus calculation.

When references change, score both systems on each version's exact ordered files. Keep the four results together so a reader can see each head-to-head comparison and how the reference revision changes it. Label the aggregation as corpus chrF2 or an average of sentence scores as applicable.

SacreBLEU 2.6.0 reference selection and corpus aggregation