# How to choose a multilingual reading test

Choose a test that asks your model to do what people need it to do. Finding the right passage, answering from a passage you give it and handling a question in a different language are separate skills. Test them separately to see which step needs a closer look.

| What you want to check | Where to start | What the model does |
| --- | --- | --- |
| Can it find the right part of an article? | TyDiQA-SelectP | Picks a passage from one article or indicates that none answers the question. |
| Can it find the words that answer the question? | TyDiQA-GoldP | Returns the answer words from a passage that contains them. |
| Does it answer the same questions across languages? | XQuAD or Belebele | Answers from translated passages in XQuAD or picks from four choices in Belebele. |
| Can it answer when the question and passage are in different languages? | MLQA | Finds the answer in a passage written in another language. |

## Find out whether the model can locate the answer

Use TyDiQA-SelectP if the model needs to find the right passage within an article. Use TyDiQA-MinSpan if it also needs to give a short answer, answer yes or no, or indicate that the article has no answer. People wrote the TyDi QA questions directly in each language. [TyDi QA task descriptions](https://github.com/google-research-datasets/tydiqa/blob/43cde6d598c1cf88c1a8b9ed32e89263ffb5e03b/README.md#the-tasks)

To see how much a shorter passage helps, ask the same model for a written answer twice. Give it the full article once and an excerpt copied exactly from that article once. Keep the settings, question, answer format, accepted answers and scoring rule the same. Check that each accepted answer appears in both inputs. When using a Gold Passage entry as supplied, check the excerpt and its answer list. The files can share a question even when the passage or accepted answers differ. The TyDi QA development files can help you build that comparison. All 5,077 Gold Passage questions also appear in the 18,670-question primary set. Gold Passage leaves out Japanese and Thai and omits 1,285 questions with marked written answers across the nine languages the files share. Pair questions by language, article title and exact question wording. Use only pairs that match once. [TyDi QA question comparison](https://sapien1.com/research/choose-a-multilingual-reading-test-evidence/evidence.html#tydi-qa-matched-questions)

## Use the same questions when you compare languages

XQuAD gives the model translated versions of the same questions and passages. The 1,190 English questions come from the SQuAD 1.1 development set. The translated files we checked share those question IDs. If you used SQuAD development to tune the model, check for overlap before treating XQuAD as a test on new questions. [Original XQuAD paper](https://aclanthology.org/2020.acl-main.421/) · [Question reuse check](https://sapien1.com/research/choose-a-multilingual-reading-test-evidence/evidence.html#xquad-question-reuse)

Belebele works when you want the model to select one of four answers. Its 122 language and script files share 900 question IDs. Save the number or letter of the selected answer. Two choices can have identical text, so the text alone may leave you unsure which one the model picked. If you exclude affected questions, use the same remaining IDs in every language you compare. [How to compare reading scores across languages](https://sapien1.com/research/cross-language-reading-scores)

## Change the question language and keep the passage the same

Use MLQA if, for example, a Vietnamese speaker might ask about an English document. In the English and Vietnamese MLQA development files we checked, 511 shared questions let you ask in either language while keeping the English passage and accepted answers exactly the same. Score both runs against the same English answers. This lets you see what changed when you switched the question language. [Original MLQA paper](https://aclanthology.org/2020.acl-main.653/) · [English and Vietnamese file comparison](https://sapien1.com/research/choose-a-multilingual-reading-test-evidence/evidence.html#mlqa-fixed-english-passage)

For a separate check of how much the passage helps, ask the same questions with and without it. [How to check whether a model uses the passage](https://sapien1.com/research/does-the-model-use-the-passage)

## Check that your test covers the language you need

For Thai, use TyDiQA-SelectP to check whether a model can find the right passage, XQuAD to check whether it can find the answer words in a supplied passage, or Belebele to check whether it can choose the right answer. The TyDiQA-GoldP development file has no Thai questions. Arabic and English are the only two languages present in all four file sets we compared: TyDi QA primary, TyDi QA Gold Passage, XQuAD and same-language MLQA development. [Language coverage check](https://sapien1.com/research/choose-a-multilingual-reading-test-evidence/evidence.html#language-coverage) · [Belebele language files](https://sapien1.com/research/choose-a-multilingual-reading-test-evidence/evidence.html#belebele-thai)

When you report scores, name the task and list the languages and question counts that went into each score. For example, both official TyDi QA scorers leave English out of the overall average even though the files include it. [TyDi QA scoring documentation](https://github.com/google-research-datasets/tydiqa/blob/43cde6d598c1cf88c1a8b9ed32e89263ffb5e03b/README.md#evaluation)
