# Evidence for the reading benchmark guides

These checks use the original Belebele and FLORES-200 archives and the public MLQA development files. Belebele has four answer choices per question. The 122 language and script files in the original ZIP share 900 question IDs. In these files, each ID pairs a source URL with a question number. [Belebele paper](https://aclanthology.org/2024.acl-long.44/) · [Belebele download and format](https://github.com/facebookresearch/belebele/blob/918890beb2290a8d3ef2d7a90369925959e1bacf/README.md)

## Match answers to questions

We copied the marked answer numbers from each Belebele file and arranged them in English question order. Fifty-two files list the questions in a different order from English. Matching the copied answers to each file by row number gave 73.55%. Matching by question ID gave 100% across 109,800 rows. When matched by row number, 38,876 replies went to a different question. Of those, 9,838 still received credit because the answer numbers matched.

## Check for repeated answer text

In 148 rows, another choice has exactly the same text as the marked answer. We copied the marked answer text from every row. Assigning the first choice with that text lost 16 credits. Assigning the last lost 133. This check compares text exactly as it is stored in the original ZIP.

## Keep the same questions

Removing the 148 affected rows only where they occur leaves between 894 and 899 questions per file. The affected rows span 23 distinct question IDs. Removing all 23 IDs from every file leaves the same 877 questions in each one.

## Check where passages came from

In the English file, 513 questions point to pages in the original FLORES-200 development split. For 417 of those questions, joining the original FLORES sentences in order exactly reproduces the Belebele passage. The number of exact matches varies across the 116 other language and script files the archives share.

We also checked 45,279 Belebele rows from FLORES-200 `devtest` URLs across the 117 shared language and script files, including English. Nine distinct question IDs each had a passage that contained a complete development sentence in at least one file. The check matches source URLs and exact text in the source archives. [FLORES-200 documentation](https://github.com/facebookresearch/flores/blob/a6c830c6e1051fb4ac1a44b32358f00463f332bd/flores200/README.md) · [Original FLORES-200 archive](https://dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz)

## Check the shortest answer rule

Always choosing the shortest answer scored 27.03% across the full Belebele test. The rule counts characters after standardizing equivalent Unicode characters and whitespace. If choices tie for shortest, it splits one point evenly among them and counts only the marked answer’s share. A uniform guess among four choices has an expected score of 25%. The original Belebele authors used word overlap and answer length to check for clues. They also reported a 28% result on 900 questions from a model that used the words without their order. [Belebele paper, section 3.2](https://aclanthology.org/2024.acl-long.44.pdf#page=4) · [Appendix A.6](https://aclanthology.org/2024.acl-long.44.pdf#page=20)

## Compare passage scores

We used the same 900 question IDs in English and Vietnamese. The rule picks the answer choice with the highest share of distinct words appearing in the question or supplied passage. With no passage, it uses only the question. Tied choices split one point evenly. Each replacement run takes the passage from another page and uses the same replacement source page in both languages.

| Language | No passage | First replacement | Intended passage | Range across 16 replacements |
| --- | ---: | ---: | ---: | ---: |
| English | 24.60% | 26.38% | 28.72% | 24.60% to 27.89% |
| Vietnamese | 25.98% | 24.80% | 27.17% | 23.58% to 27.84% |

The intended passage scored above all 16 replacements in English and 15 in Vietnamese. Compared with no passage, the intended passage increased credit on 311 questions in English and 313 in Vietnamese. It reduced credit on 218 questions in English and 234 in Vietnamese. [Word-matching code](./check-passage-word-matches.py) · [Aggregate results](./passage-word-matches.json)

## Change the question language

In the English and Vietnamese MLQA development files, 511 question IDs appear in all four combinations of question and passage language. Among those IDs, changing the question language from English to Vietnamese with the English passage held fixed leaves the passage and accepted answer list exactly the same. A comparison using that setup scores both runs against the same English answers. [MLQA paper](https://aclanthology.org/2020.acl-main.653/) · [MLQA file layout](https://github.com/facebookresearch/MLQA/blob/24bf470b2e68f2bfb93b2fd31c370bb8ee18d9ed/README.md#data-download)

## Repeat the word matching check

Download the [original Belebele ZIP](https://dl.fbaipublicfiles.com/belebele/Belebele.zip), the [word-matching script](./check-passage-word-matches.py), its [Python helper](./check.py), and the [saved aggregate results](./passage-word-matches.json). Put them in the same folder. The script verifies the archive hash and reads the English and Vietnamese files. It reports scores as credit units, with 12 units equal to one correct answer. Use Python 3.12 to repeat the comparison.

```bash
set -e -o pipefail
python3.12 -B check-passage-word-matches.py Belebele.zip | cmp - passage-word-matches.json
```
