Worked examples for replaying a BM25 ranking

These examples use invented documents and queries. They show how the same candidates can get different scores when the scoring collection changes, and how the first search stage can drop a candidate before the reranker sees it. The calculations use the pinned rank-bm25 implementation.

Download the five replay scripts and saved results (ZIP). The README explains how to install the pinned Python dependencies and compare the replay outputs.

How did the average document length change?

Adding a document that matches no query terms can still change the average length used to calculate the scores. In this example, A leads below an average of 18 tokens, ties B at 18 and falls behind above 18. B leads throughout the control with length normalization disabled.

Which documents changed the term weights?

In a collection of nine invented documents, A matches red and B matches blue. Two edits to other documents increase the count of documents containing red from two to four. The count for blue stays at three. A starts ahead, ties B after the first edit and falls behind after the second. The candidates and the four-token average stay fixed.

Which documents supplied the shard statistics?

In this 18-document example, two exchanges of background documents between the shards flip the ordering of A and B when each shard supplies its own scoring statistics. The scorer over the unchanged full collection gives B the higher score in every tested layout. The Elasticsearch search API describes different scopes for term statistics. The scores here come from the Python example.

Which documents reached the reranker?

In a separate 18-document example, scoring the full eligible set gives the order B, C, A. The final scorer continues to use those same full-collection scores. Changing the first-pass selection controls which candidates it receives.

First-pass selection and reranked results
First passOriginal final orderAfter two background exchanges
Local BM25, one per shardC, AB, C
Local BM25, up to two per shardB, C, AB, C, A
Full-collection BM25, one per shardB, CB, C