Evidence for choosing a multilingual reading test

These checks compare the questions and languages in specific releases of TyDi QA, XQuAD, MLQA and Belebele. The counts describe the files used in the guide.

Which questions do the TyDi QA files share?

We matched the development files by language, article title and exact question wording. Each of the 5,077 Gold Passage rows matched one of the 18,670 primary rows. In the nine shared languages, the Gold Passage file leaves out 1,285 primary rows where at least one annotator marked a written answer. The primary development file includes Japanese and Thai, and the Gold Passage file has no rows in those two languages.

To compare a whole article with a shorter input, ask the same model for a written answer twice. Copy the shorter passage exactly from the article. Keep the settings, question, answer format, accepted answers and scoring rule the same. Check that each accepted answer appears in both inputs. Check the excerpt and its answer list if you use Gold Passage as supplied. The match by question establishes which questions the files share. The passage text and accepted answers can differ.

The authors’ primary and Gold Passage evaluations each leave English out of their overall average.

Counts by language (JSON) · Comparison script (Python) · Pinned primary development file · Pinned Gold Passage development file · Official scoring rules

Save the two source files as primary.parquet and gold.parquet. Save the script and counts in the same directory, then run the comparison. The script verifies the two input digests before reading them.

uv run --python 3.12 --with pyarrow==26.0.1 python compare-tydi-qa-questions.py primary.parquet gold.parquet > replay.json
cmp replay.json tydi-qa-question-comparison.json

Which XQuAD questions come from SQuAD?

All 1,190 English XQuAD IDs match questions in the SQuAD 1.1 development file. For those IDs, the article title, passage and question text match exactly. All 12 files in the XQuAD revision we checked share the same IDs, including the Romanian file added after the original release.

Question counts and matches (JSON) · Original XQuAD paper · Pinned XQuAD revision · Pinned SQuAD development file

What stays the same when an MLQA question changes language?

The four English–Vietnamese development files share 511 question IDs. Switching from English to Vietnamese questions while keeping the English passage changed all 511 questions, zero passages and zero accepted-answer lists. The check compares each passage and the ordered list of accepted answers, including the saved offsets. Use the same English answers to score both runs.

File comparison counts (JSON) · Official MLQA downloads · Original MLQA paper

Which languages are in the four file sets?

We checked TyDi QA primary and Gold Passage development, the pinned XQuAD release, and the MLQA development files that use the same language for the question and passage. Arabic and English are the only languages in all four sets. Thai appears in TyDi QA primary and XQuAD, and it is absent from Gold Passage.

Language codes by file set (JSON)

Where is the Thai multiple-choice test?

Belebele has a Thai file named tha_Thai.jsonl. Use it for questions that ask the model to pick one of four answers after reading a passage.

Pinned Thai file from the authors · Original Belebele paper · How to compare multiple-choice reading scores across languages