# How to compare multiple-choice reading scores across languages

Use the same questions and scoring rules in every language. Check for repeated answer choices and reading passages your model may have seen before. On Belebele, a multilingual reading test, we copied the correct answer numbers from each file and put them in English question order. Lining them up with each file by row gave 73.55%. Matching them by question ID gave 100%. [Belebele scoring replay](https://sapien1.com/research/reading-benchmark-evidence/evidence.html#match-answers-to-questions)

## Match each answer to the right question

Save the language, question ID and chosen answer number or letter with each reply. Belebele’s 122 files share the same 900 question IDs, but 52 files list them in a different order from English. In the row-number test, 9,838 replies paired with the wrong question still received credit because the answer numbers happened to match. [Question order check](https://sapien1.com/research/reading-benchmark-evidence/evidence.html#match-answers-to-questions)

## Check whether two choices have the same text

Imagine that choices B and D both say `12`, and the file marks B as correct. A reply that says `12` gives you the answer text. You need the choice letter to score it by position. In Belebele, 148 rows repeat their marked answer text. When we copied those exact answer strings, picking the first matching choice lost 16 credits and picking the last lost 133. Decide whether your test scores choice labels or answer text, then check the choices using the same rules. [Repeated answer choices](https://sapien1.com/research/reading-benchmark-evidence/evidence.html#check-for-repeated-answer-text)

## Compare scores on the same questions

If you set aside the 148 repeated-answer rows only where they appear, each language keeps between 894 and 899 questions. If you need a comparison with the same questions in every language, set aside all 23 affected question IDs in every file. That leaves a shared set of 877 questions. Report the original count and the count you scored. [Question filtering comparison](https://sapien1.com/research/reading-benchmark-evidence/evidence.html#keep-the-same-questions)

## Check which passages the model may have seen before

If you used the FLORES-200 development set while tuning a model, check whether those same passages appear in Belebele. In the English file, 513 questions point to pages used in that development set. For 417 of those questions, putting the FLORES sentences together gives you the exact Belebele passage. The number of exact matches varies across the 116 other language and script files the two datasets share. Check the text in the language you are testing. [English passage reuse](https://sapien1.com/research/reading-benchmark-evidence/evidence.html#check-where-passages-came-from) · [Translated passage reuse](https://sapien1.com/research/reading-benchmark-evidence/evidence.html#check-where-passages-came-from)
