# How to reproduce a BM25 ranking after an index change

Save the documents used to compute BM25 statistics, the IDs eligible for retrieval, and the IDs passed between ranking stages. Those inputs let you locate a change in term weighting, shard scoring, or candidate selection. The examples here use fictional pretokenized documents and `rank-bm25` 0.2.2. [Scoring implementation](https://github.com/dorianbrown/rank_bm25/blob/2550648efdbdcc5ebadbc8e5c8b26f5eb94b2b36/rank_bm25.py#L30-L134)

## Check the collection behind the score

Replay the same query and eligible IDs against each scoring collection. Save the index or corpus revision, query analysis and scorer settings. Compare collection size, average document length and query-term document frequencies in the scope that supplies them.

Adding a document with no query matches can change length normalization. In one example, a nonmatching append puts the average just below, at, or above 18 tokens. A leads below 18, ties B at 18, and loses above 18. B leads throughout when length normalization is disabled. [Average length replay](https://sapien1.com/research/replay-bm25-ranking-evidence/evidence.html#collection-length)

Edits can also change term weights at a fixed average length. In a nine-document example, two background edits raise the number of documents containing `red` from two to four while `blue` stays at three. A matches `red` and B matches `blue`. A leads initially, ties B after one edit, and falls behind after two. The two candidates and the four-token average stay fixed. [Term frequency replay](https://sapien1.com/research/replay-bm25-ranking-evidence/evidence.html#term-frequency)

## Check which documents supply shard statistics

For a local scorer, save the shard assignment of background documents as well as candidates. Rebuilding the same corpus into a different layout can change a comparison of scores from separate shards. In one 18-document example, two background exchanges turn a local lead for A into a lead for B. One scorer over the unchanged full collection gives B the higher score in every tested layout. [Shard layout replay](https://sapien1.com/research/replay-bm25-ranking-evidence/evidence.html#shard-statistics)

Check the configured scoring mode when reproducing a distributed run. Elasticsearch, for example, documents modes that use frequencies from the current shard or from all shards. The preceding scores come from the Python fixture. [Elasticsearch search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-msearch)

## Check which candidates reach the reranker

Compare the IDs before and after the shortlist cutoff. Record the scoring method at each stage, the depth per shard, and the tie-break rule if equal scores can change selection. Use a full eligible ranking on a tractable test query as a reference for which candidates could win under the reranker's scoring rule.

In a separate 18-document example, full-collection scoring orders the eligible candidates B, C, A. The table shows how the first-pass selection affects the final order when the reranker always uses full-collection scores. [Shortlist replay](https://sapien1.com/research/replay-bm25-ranking-evidence/evidence.html#shard-shortlist)

| First pass | Original final order | Final order after two background exchanges |
| --- | --- | --- |
| Local BM25, one per shard | C, A | B, C |
| Local BM25, up to two per shard | B, C, A | B, C, A |
| Full-collection BM25, one per shard | B, C | B, C |

The first row loses B before reranking in the original layout. This comparison separates a change in who gets scored from a change in how the finalists are scored. If a candidate is missing from the reranker input, replay the first-pass selection. If the inputs match and the order differs, compare reranking scores and tie handling.
