# A zero-shot score can change when NLI label indices move

Two invented sets of three logits produce opposite multi-label rankings when arranged using two real model configurations. The single-candidate path also changes the score. The two-candidate exclusive path gives the same ranking for both arrangements. This comparison tests the zero-shot scorer in Transformers 4.55.0 and 5.23.0 using public configuration metadata and no model inference.

## The expected pair and the pair used

The zero-shot classification pipeline constructs a premise and hypothesis for each candidate label. Its multi-label scoring documentation describes a softmax over the entailment and contradiction logits for each candidate. In both inspected releases, the scorer finds entailment in `label2id`. It uses the last logit as the other input when entailment is at index 0, and index 0 otherwise. The same branch runs when there is only one candidate. [Scoring documentation](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.ZeroShotClassificationPipeline.__call__) · [4.55.0 scorer](https://github.com/huggingface/transformers/blob/06f8004e5cd9d06cfbffc3f47afb6c2b43bcb3d2/src/transformers/pipelines/zero_shot_classification.py#L244-L271) · [5.23.0 scorer](https://github.com/huggingface/transformers/blob/52b080b6416a7897e2af90febeb4dfad22b84d19/src/transformers/pipelines/zero_shot_classification.py#L235-L261)

| Published config | Index 0 | Index 1 | Index 2 | Logit paired with entailment |
| --- | --- | --- | --- | --- |
| aiface/ModernBERT-large_nli | Entailment | Contradiction | Neutral | Neutral |
| facebook/bart-large-mnli | Contradiction | Neutral | Entailment | Contradiction |

Both configs have matching forward and reverse label maps. They establish the declared mapping. The experiment does not measure what the respective models learned or how often their real logits would produce a rank change. [ModernBERT config](https://huggingface.co/aiface/ModernBERT-large_nli/blob/4f4d21a8351871d2ab4b6bf466b3952d096955bf/config.json) · [BART config](https://huggingface.co/facebook/bart-large-mnli/blob/d7645e127eaf1aefc7862fd59a17a5aa8558b8ce/config.json)

## The ranking changes in a controlled fixture

The same semantic logits were rearranged using each config. Label A has entailment 2, contradiction 0 and neutral 4. Label B has entailment 1, contradiction 2 and neutral −1. The reference calculation uses the two classes named in the documentation. Results agree across both releases.

| Scoring path | Label A | Label B | Higher score |
| --- | ---: | ---: | --- |
| Entailment and declared contradiction | 0.880797 | 0.268941 | A |
| Multi-label using the ModernBERT mapping | 0.119203 | 0.880797 | B |
| Multi-label using the BART mapping | 0.880797 | 0.268941 | A |
| Two-candidate exclusive, either mapping | 0.731059 | 0.268941 | A |

With only label A and the default `multi_label=False`, the same branch gives 0.119203 for the ModernBERT mapping and 0.880797 for BART. Across all six permutations of the three distinct labels, the multi-label branch gives scores consistent with the declared contradiction slot for three permutations and the neutral slot for the other three. These are six possible arrangements in a fixture. The number gives no estimate of the share of published models affected. [Full output](observations.json)

For a zero-shot evaluation, preserve the checkpoint revision, class mapping, scoring library version, candidate count and multi-label setting. Check which logits the scoring path actually uses before comparing independent scores across checkpoints. A label map needs to describe the existing head’s output order. Editing the config alone does not move its output coordinates.

## Reproduce the check

Run `python3 check.py | cmp - observations.json` with pipeline failure checking enabled and NumPy available. The checker verifies the bytes of two pipeline source files and two model config files, totaling 27,139 bytes. It executes only the pinned entailment-index and score postprocessing methods in a small class without their package dependencies. It supplies float32 arrays via the interface expected from the scorer’s tensor inputs. The independent reference uses the logistic form of the two-class softmax. The package exercises the score reduction and sorting. It does not run tokenizer, model, dataset, full library initialization or benchmark.

Transformers’ repository is Apache 2.0. The two model card labels were Apache 2.0 and MIT, respectively. This package uses the public label map facts and identifiers. The checker fetches the upstream files at runtime and bundles no upstream source, model weights, card prose or examples. [Checker](check.py) · [Transformers licence](https://github.com/huggingface/transformers/blob/52b080b6416a7897e2af90febeb4dfad22b84d19/LICENSE)
