# Plan a sample review to detect labeling errors

Set the decision before choosing a sample size. If the batch contains a specified number of incorrect labels, how much risk can you accept that the review finds none? The answer depends on how you select rows, what errors reviewers can detect, and how you assign the work.

The examples below extend the Dataset Evaluation guide's fictional batch of 120 rows with six incorrect labels. Each example isolates a different part of the review plan.

## Use the selection rule in the probability calculation

First assume every reviewed error is detected. A uniform sample of 20 distinct rows misses all six errors about 32.63% of the time. If you instead take every sixth row from a random starting position, errors aligned at a single position in successive six-row blocks can be found from only one of the six starts. The miss probability in that arrangement is 83.33%.

Choosing one row independently from each block of six brings the probability for that arrangement to 33.49%. That is also the highest miss probability over all placements of the six errors for this independent block design. In the periodic example, reviewing additional distinct systematic starts gets below 5% only when all six starts are included. Uniform sampling reaches that threshold at 47 rows under perfect detection.

## Separate reviewer detection from row selection

At 47 uniformly sampled rows, the chance of finding no true error is 4.66% with perfect detection. It is 10.06% when each sampled error can be detected independently with probability 80%. Under that independent model it first drops below 5% at 59 rows.

The same 80% per-error rate can produce miss probabilities from 8.96% to 23.73% at 47 rows, depending on whether errors are detectable together. This calculation assumes the uniform draw is independent of which errors the review process can detect. In the least favorable case, the process can detect all six errors in 80% of runs and none in the other 20%. Even reviewing every row once leaves a 20% chance of finding none. Evidence from known errors should record which errors were caught together during each comparable review.

## Count distinct rows and independent opportunities

Consider a budget of 60 row reviews. In this example, each reviewer can detect all assigned errors in 80% of runs and none in the rest. The rows are assigned uniformly independently of these states.

| Assignment of the review budget | Distinct rows | Chance of finding no true error |
| --- | --- | --- |
| One reviewer checks 60 rows | 60 | 21.10% |
| Two independent reviewers check disjoint sets of 30 | 60 | 10.33% |
| Two reviewers share a detection state and check disjoint sets of 30 | 60 | 21.10% |
| Two independent reviewers both check the same 30 rows | 30 | 20.36% |

Record the sampled IDs, selection rule, which reviewer saw each row, and the assumptions about detection. Apply the risk target to the resulting design. A review aimed at finding at least one true error needs separate checks for false alarms, adjudication quality, and fitness for the intended use.

[Dataset Evaluation sampling example](https://sapien1.com/dataset-evaluation#review-labels-and-coverage) · [Statistics Canada on sampling designs](https://www150.statcan.gc.ca/n1/edu/power-pouvoir/ch13/prob/5214899-eng.htm) · [Research on imperfect inspection](https://pubsonline.informs.org/doi/10.1287/opre.32.3.575)
