Research
Audit a cross-validation AUROC before using it to select a model
Start with the comparison you need. Does it cover every required test fold? Should each fold contribute equally, or should each comparable pair contribute equally? If scores come from separate fitted models, decide whether the evaluation supports comparing those scores across folds.
Verify the score exists for every required fold
For binary AUROC, check the positive and negative counts in every test fold. Preserve which validation IDs were scored, each candidate's fold scores, and the warnings from the run.
In a fictional ten-case grid search, two folds contain both classes and a third contains only negatives. Scikit-learn 1.7.2 warns and returns NaN for both candidates in the third fold. Both means are NaN and both candidates rank 1. The search refits whichever candidate appears first, even with error_score="raise". Reversing the candidate order reverses the selection.
For a separate diagnostic limited to the two defined folds, the means are 1.00 for A and 0.375 for B. A is selected either way. Record the reduced validation scope if you report that diagnostic. Resolve coverage for the original required folds before using that search to choose a winner.
Check how the fold scores are weighted
For unweighted binary AUROC, a test fold with p positives and n negatives has p × n positive-negative comparisons. The within-fold AUROC describes how often the positive scores higher, with half credit for ties. Equal-fold averaging gives each fold the same influence. Within-fold pair weighting gives each eligible pair equal influence.
In a separate fictional example, both folds have six rows. Fold 1 has one positive and five negatives, so it contributes five pairs. Fold 2 has three of each, so it contributes nine. A scores 5/5 and 4/9. B scores 2/5 and 8/9.
| Summary | A | B | Higher |
|---|---|---|---|
| Equal weight per fold | 13/18 | 29/45 | A |
| Equal weight per within-fold pair | 9/14 | 10/14 | B |
For this comparison, the scores tie when fold 1 gets weight 20/47. Equal fold weights put more weight than that on fold 1. Pair weights give it 5/14 and put B ahead. Both summaries count comparisons within folds.
Check what pooling assumes about the scores
Pooling the predictions first also compares positives and negatives from different folds. In a third example, A scores 3/4 within each fold and B scores 2/4 within each. The original pooled scores are 13/16 for A and 11/16 for B. Adding a constant to A's second-fold scores preserves both local orders and reduces A's pooled score to 10/16. B's score stays at 11/16 and the pooled comparison reverses.
If your question concerns rankings within each held-out fold, report the fold scores and the chosen weighting. If you pool, record what makes score scales comparable across fits. Keep the fold assignment, class counts, score columns, aggregation rule and selection outcome together so a second reviewer can reproduce the comparison.
The three examples use fictional scores. scikit-learn on cross-validated predictions · Research on cross-validation aggregation