Skip to main content

Menu

Research

Check class coverage before accepting a grouped evaluation split

Language: English

Keep related examples in the same group when they must stay together across train and test. Groups can be conversations, customers, or other units defined by the evaluation. Before choosing a splitter, write down which classes must appear in each test fold and any minimum counts. Class presence and exact balance are different requirements.

Check the emitted split

Save the group-to-fold mapping and count each class in each test fold. Check group isolation and class coverage independently. If a required class is absent, hold the split and check whether another intact-group assignment can meet the requirement.

In a 24-row fictional example, scikit-learn 1.7.2's StratifiedGroupKFold used three folds with shuffling disabled. One set of numeric group identifiers gave the three class-count pairs (4, 2), (4, 6) and (4, 4). Reversing the identifiers gave (4, 6), (4, 0) and (4, 6), with every row, label and group membership preserved. The first result passes class presence and misses exact balance. The second fails both. An exact (4, 4) assignment was feasible in this fixture.

Check whether the requirement is feasible

For two folds requiring every class, each class must occur in at least two independent groups. Use the class-to-group memberships to check whether all classes can appear in both folds at once.

In two separate 12-row examples, every class occurs in two groups and every group contains two rows. Each class therefore requires its two groups to land in opposite folds. When the six pairs form a cycle, alternating the groups between folds works. When pairs form two triangles, each triangle has three groups that would all have to be opposite one another. Each triangle forces a gap. Enumerating all 62 assignments to two named, nonempty folds gives two full-coverage assignments for the cycle and zero for the two triangles.

Choose a repair that respects what is fixed

In the two-triangle example, the emitted first fold lacks class 5 and the second lacks class 1. If existing group assignments stay fixed, two new independent groups can fill those gaps separately. A single intact group can enter only one test fold.

If existing groups may move, a single new group can suffice. In the fixture, nine of the 15 possible two-class profiles work. Each includes a class from each triangle. Four profiles require moving at least one original group, and the other five require moving at least three. Check the classes a proposed new group actually contains and the moves the evaluation permits before collecting it.

Record the accepted assignment, the per-fold counts and how group isolation was checked. Give any changed roster its own evaluation revision. A coverage check establishes the specified counts and membership. An evaluation also needs its other acceptance checks, including whether the cases represent the intended use.

scikit-learn on grouped cross-validation