Skip to main content

Menu

Research

How to evaluate a system that can decline cases

Language: English

Keep the same test roster for both workflows. For each case, record the primary prediction and confidence, whether the system accepted or declined it, any fallback result, and the final outcome. These records support three checks before you compare accuracy.

Verify how the workflow reaches its coverage. Coverage is the share of cases accepted by the primary system. In a six-case replay, a confidence threshold jumped from one accepted case to five because four scores tied. An exact quota of three could split the tie. Sorting by score with file order as the tie rule gave either zero or two errors among those three cases when only the row order changed. Report the realized coverage for a threshold and record the tie rule and accepted IDs for an exact quota. scikit-learn score thresholds

Count the fallback’s decisions on the cases it actually receives. In a separate 12-case replay, A and B each accepted six cases and shared three accepted IDs. A got six of its accepted cases right, while B got five. The same fixed per-case fallback got two of A’s declined cases right and five of B’s. The full workflow therefore put B ahead, 10 correct to 8. Report both accepted-case accuracy and full-roster accuracy. Keep the fallback version and the declined-case outcomes with the report. Combined predictor and fallback evaluation

Check which missing outcomes could change the lead. In the 12-case replay, hiding three results from cases both workflows deferred left B ahead by two under every possible completion, because each hidden result was shared. Hiding six outcomes where the routing differed instead gave a B-minus-A margin from −4 to +2. Use the shared-outcome calculation only when both deferrals would use the same per-case result. When the possible margins allow either workflow to lead, prioritize recovering the outcomes from cases where their routes differ.

Include the full roster size, accepted count, correct decisions on each path, and missing outcome count with the reported scores. If data are incomplete, state the assumptions behind any remaining range.