Research
What to check before comparing macro-F1
Macro-F1 gives equal weight to the class scores that enter the mean. Before using it to choose a classifier, record the case IDs, label version and scoring settings. Three synthetic replays in scikit-learn 1.7.2 show how the same predictions can support different rankings. scikit-learn F1 definition and options
Compare scores at the same label detail. On nine support cases with refund, invoice and login labels, B led, 0.778 to 0.556. Merging refund and invoice into billing turned four of A’s errors into billing matches, while B’s two errors still crossed categories. A then led 1.000 to 0.750. If you need to compare scores across label revisions, save the mapping and replay both systems at the same level. Preserve the original labels and predictions when the distinction still matters to the task.
Check which classes enter each mean. In a separate slice with no fraud references, A predicted fraud once and received a defined class F1 of zero. B had neither fraud references nor fraud predictions. Inferring labels gave B 0.667 across two classes and A 0.600 across three. With an explicit three-class list and undefined F1 set to zero, B fell to 0.444. With the same list and undefined F1 excluded, B returned to 0.667. Record the intended class roster, the undefined-score rule, each class’s reference and prediction counts, and the number of classes actually averaged.
State whether you pooled cases or averaged slice scores. On two balanced eight-case slices, the mean of the two slice-level macro-F1 scores put A ahead 0.600 to 0.564. Scoring all 16 predictions together put B ahead 0.625 to 0.600. Equal slice sizes give the same result for an equal-weight or case-weighted mean of the slice scores. A pooled class score instead uses summed true-positive, false-positive and false-negative counts. For a pooled report, combine the counts before computing each class’s F1. For a slice report, show the slice scores and their weights.
Keep the versioned label map and per-class confusion counts with the score. When settings differ between reports, replay both systems with the same case roster, label detail, undefined-score policy and aggregation before interpreting the ranking.