Skip to main content

Menu

Research

A zero-shot score can change when NLI label indices move

Two invented sets of three logits produce opposite multi-label rankings when arranged using two real model configurations. The single-candidate path also changes the score. The two-candidate exclusive path gives the same ranking for both arrangements. This comparison tests the zero-shot scorer in Transformers 4.55.0 and 5.23.0 using public configuration metadata and no model inference.

The expected pair and the pair used

The zero-shot classification pipeline constructs a premise and hypothesis for each candidate label. Its multi-label scoring documentation describes a softmax over the entailment and contradiction logits for each candidate. In both inspected releases, the scorer finds entailment in label2id. It uses the last logit as the other input when entailment is at index 0, and index 0 otherwise. The same branch runs when there is only one candidate. Scoring documentation · 4.55.0 scorer · 5.23.0 scorer

Published configIndex 0Index 1Index 2Logit paired with entailment
aiface/ModernBERT-large_nliEntailmentContradictionNeutralNeutral
facebook/bart-large-mnliContradictionNeutralEntailmentContradiction

Both configs have matching forward and reverse label maps. They establish the declared mapping. The experiment does not measure what the respective models learned or how often their real logits would produce a rank change. ModernBERT config · BART config

The ranking changes in a controlled fixture

The same semantic logits were rearranged using each config. Label A has entailment 2, contradiction 0 and neutral 4. Label B has entailment 1, contradiction 2 and neutral −1. The reference calculation uses the two classes named in the documentation. Results agree across both releases.

Scoring pathLabel ALabel BHigher score
Entailment and declared contradiction0.8807970.268941A
Multi-label using the ModernBERT mapping0.1192030.880797B
Multi-label using the BART mapping0.8807970.268941A
Two-candidate exclusive, either mapping0.7310590.268941A

With only label A and the default multi_label=False, the same branch gives 0.119203 for the ModernBERT mapping and 0.880797 for BART. Across all six permutations of the three distinct labels, the multi-label branch gives scores consistent with the declared contradiction slot for three permutations and the neutral slot for the other three. These are six possible arrangements in a fixture. The number gives no estimate of the share of published models affected. Full output

For a zero-shot evaluation, preserve the checkpoint revision, class mapping, scoring library version, candidate count and multi-label setting. Check which logits the scoring path actually uses before comparing independent scores across checkpoints. A label map needs to describe the existing head’s output order. Editing the config alone does not move its output coordinates.

Reproduce the check

Run python3 check.py | cmp - observations.json with pipeline failure checking enabled and NumPy available. The checker verifies the bytes of two pipeline source files and two model config files, totaling 27,139 bytes. It executes only the pinned entailment-index and score postprocessing methods in a small class without their package dependencies. It supplies float32 arrays via the interface expected from the scorer’s tensor inputs. The independent reference uses the logistic form of the two-class softmax. The package exercises the score reduction and sorting. It does not run tokenizer, model, dataset, full library initialization or benchmark.

Transformers’ repository is Apache 2.0. The two model card labels were Apache 2.0 and MIT, respectively. This package uses the public label map facts and identifiers. The checker fetches the upstream files at runtime and bundles no upstream source, model weights, card prose or examples. Checker · Transformers licence