Skip to main content

Menu

Research

How to reproduce a BM25 ranking after an index change

Language: English

Save the documents used to compute BM25 statistics, the IDs eligible for retrieval, and the IDs passed between ranking stages. Those inputs let you locate a change in term weighting, shard scoring, or candidate selection. The examples here use fictional pretokenized documents and rank-bm25 0.2.2. Scoring implementation

Check the collection behind the score

Replay the same query and eligible IDs against each scoring collection. Save the index or corpus revision, query analysis and scorer settings. Compare collection size, average document length and query-term document frequencies in the scope that supplies them.

Adding a document with no query matches can change length normalization. In one example, a nonmatching append puts the average just below, at, or above 18 tokens. A leads below 18, ties B at 18, and loses above 18. B leads throughout when length normalization is disabled. Average length replay

Edits can also change term weights at a fixed average length. In a nine-document example, two background edits raise the number of documents containing red from two to four while blue stays at three. A matches red and B matches blue. A leads initially, ties B after one edit, and falls behind after two. The two candidates and the four-token average stay fixed. Term frequency replay

Check which documents supply shard statistics

For a local scorer, save the shard assignment of background documents as well as candidates. Rebuilding the same corpus into a different layout can change a comparison of scores from separate shards. In one 18-document example, two background exchanges turn a local lead for A into a lead for B. One scorer over the unchanged full collection gives B the higher score in every tested layout. Shard layout replay

Check the configured scoring mode when reproducing a distributed run. Elasticsearch, for example, documents modes that use frequencies from the current shard or from all shards. The preceding scores come from the Python fixture. Elasticsearch search API

Check which candidates reach the reranker

Compare the IDs before and after the shortlist cutoff. Record the scoring method at each stage, the depth per shard, and the tie-break rule if equal scores can change selection. Use a full eligible ranking on a tractable test query as a reference for which candidates could win under the reranker's scoring rule.

In a separate 18-document example, full-collection scoring orders the eligible candidates B, C, A. The table shows how the first-pass selection affects the final order when the reranker always uses full-collection scores. Shortlist replay

First passOriginal final orderFinal order after two background exchanges
Local BM25, one per shardC, AB, C
Local BM25, up to two per shardB, C, AB, C, A
Full-collection BM25, one per shardB, CB, C

The first row loses B before reranking in the original layout. This comparison separates a change in who gets scored from a change in how the finalists are scored. If a candidate is missing from the reranker input, replay the first-pass selection. If the inputs match and the order differs, compare reranking scores and tie handling.