Research
How to check whether a model uses the passage
Ask the model the same questions twice, once with the passage and once without it. Keep the questions, answer choices and scoring rules the same. Compare the two scores to see how much the supplied passage helped on those questions. For multiple choice, run one more check using just the answer choices. On Belebele, a rule that always chooses the shortest answer earned 27.03% across the full test. Uniform guessing has an expected score of 25%. Belebele answer-length check
Change one input at a time
| What the model sees | What to compare |
|---|---|
| Passage, question and choices | How often the model answers correctly with everything supplied |
| Question and choices | How often it answers correctly without the passage |
| Choices alone | How often it answers correctly from the choices alone |
Use the same model and settings each time. Count how often adding the passage turns a wrong answer into a right one, and how often it does the reverse. Those counts explain a small score change when the two runs get different questions right. For a written-answer test, compare the first two runs without choices and use the same accepted answers in both.
Look at which questions changed
Suppose the model gets 70 out of 100 questions right in both runs. When you match the replies by question ID, 62 are right in both runs and 22 are wrong in both. Adding the passage turns eight wrong answers into right ones and eight right answers into wrong ones. The score stays at 70%, but you have 16 questions to inspect to see where the passage helped or hurt.
Try a simple word-matching rule
The Belebele authors checked for clues from word overlap and answer length. They also reported a score of 28% on 900 questions for a model that looked at which words appeared, regardless of their order. Belebele paper, section 3.2 · Appendix A.6
We ran a fixed word-matching rule on the same 900 Belebele questions in English and Vietnamese. We used no passage, the intended passage, or a passage from a different page. For each choice, we count what share of its distinct words also appear in the question or in the supplied passage. With no passage, we use only the question. The rule picks the choice with the highest share. When choices tie for first, we split one point evenly among them and count only the marked answer’s share. For each question, the English and Vietnamese replacement passages come from the same other page, in their own languages. Code · Results
| Language | No passage | Passage from another page | Intended passage |
|---|---|---|---|
| English | 24.60% | 26.38% | 28.72% |
| Vietnamese | 25.98% | 24.80% | 27.17% |
In the first comparison, the intended passage gave the highest score in both languages. We then tried 15 more ways to swap the passages. Across all 16 swaps, scores ranged from 24.60% to 27.89% in English and from 23.58% to 27.84% in Vietnamese. The intended passage beat all 16 swaps in English and 15 in Vietnamese. Try several replacements so your conclusion depends less on which other page you pick. Results for all swaps
Compared with no passage, adding the intended passage gave more credit on 311 questions in English and 313 in Vietnamese. It gave less credit on 218 in English and 234 in Vietnamese. If you compare this rule with a model, score both on the same questions.
Keep the questions and scoring rules fixed
Save a question ID with every reply. Belebele’s 122 files share 900 question IDs, but 52 files list them in a different order from English. If you exclude questions with repeated answer text, remove the same IDs from every run you compare. Keep the choice letter so the scorer can tell two choices apart when they display the same words. Question order replay · Question filter
Check what the model may have seen before
A correct answer without the passage can come from prior knowledge, clues in the question or the choices, or a guess. If you used FLORES-200 development to tune a model that you now want to test on Belebele, you can exclude questions from those development URLs. Check the text too. In the original files, nine questions from other URLs still contained a sentence from development in at least one checked language or script. Belebele passage check
Keep the passage fixed when you change the question language
To test whether asking in a different language changes the result, run that comparison on its own. In the English and Vietnamese MLQA files we checked, 511 question IDs let you switch the question language while keeping the English passage and accepted answers exactly the same. Score both runs against the same English answers. MLQA matched questions