Research
Can you compare two benchmark scores?
Start with the decision the scores are supposed to support. To quote a historical result, identify the exact published table. To choose between two agents, check that they faced the same cases under the same rules, then use results from those cases to calculate the comparison. Four public examples show why these checks matter.
Find the table that the citation refers to
LiveBench used the same leaderboard label, 2025-05-30, for two different source tables. Across the two pinned tables, 30 model labels appear in both. For 28 of those labels, at least one of the two scores in the separate Coding category is higher in the later table. The maintainers describe rerunning Agentic Coding with a changed harness and step limit. The source tables leave the cause of the separate Coding changes open. Save the table revision or file hash with the quoted number, and keep the benchmark's date label as a separate field. Earlier LiveBench table · Later LiveBench table · Maintainer changelog
Match cases before comparing success rates
Two example resource ceilings each select 44 of the 66 tasks in Terminal-Bench v4.1.0. Only 36 task IDs appear in both selections. The ceilings are 4 CPUs, 8,192 MB of memory and 10,240 MB of storage, or 2 CPUs, 16,384 MB of memory and 20,480 MB of storage. The resource filter applies each ceiling to the declarations for both the task and verifier environments. The following outcomes are fictional. Terminal-Bench resource settings and filter
Suppose agent A solves 32 cases in the first selection and agent B solves 34 in the second. Each selection has eight cases outside the common set. A could therefore have solved anywhere from 24 to 32 of the 36 shared cases, and B anywhere from 26 to 34. If A solved 32 shared cases and B solved 26, A would lead by six on the common set. If A solved 24 and B solved 34, B would lead by ten. Both scenarios fit the reported totals.
| Fictional result | Solved in its own 44-case selection | Possible solved cases among the 36 shared |
|---|---|---|
| Agent A | 32 | 24–32 |
| Agent B | 34 | 26–34 |
In this example, the totals leave the ordering on the shared cases undetermined. Ask for per-case outcomes or grade both agents over the same verified case list. If the runs used different resource budgets, describe the result as a comparison of those configurations. To isolate the effect of changing the agent, hold the budget and other run conditions fixed. Set the common denominator and decide how missing, failed or ungraded runs count before ranking the agents. Report the resulting coverage.
Check that the grader used the same inputs
The February and August 2026 SWE-bench Verified test snapshots contain the same 500 case IDs. Two August commits removed one expected regression-test entry for an Astropy case and five for a Django case. The revisions establish what changed in the stored lists. To measure the effect on scores, grade the same candidate patches under both revisions while holding the other settings fixed. For a comparison between agents, use the same dataset revision, grader, execution environment and evaluator options for both. February test snapshot · August test snapshot · First SWE-bench change · Second SWE-bench change · Grader source
Verify the per-case file before using it
Every Eval Ever gives readers aggregate records and a collection index for finding per-sample results. At one pinned datastore revision, the two sources declare different SHA-256 values for associated sample files in all 14 entries in the MMLU-Pro index and all 11 in the gsm8k index. In each collection, the first listed sample file matched the index digest when downloaded. The aggregate checksum differed. This check identifies a file-level discrepancy. It leaves the scores' correctness and the cause of the discrepancy unassessed. MMLU-Pro index · GSM8K index · GSM8K example aggregate
Keep the immutable file location and the verified digest with the paired results you use. If the publisher's declarations conflict, resolve the discrepancy before relying on that file as evidence for the score.
Record the comparison
| Record | What to save |
|---|---|
| Published result | Source URL, benchmark label, table revision or hash, metric and unit |
| Cases and trials | Dataset revision and split, exact case and trial IDs, scoring denominator and exclusion rules |
| How the work ran | Agent and model revisions, tools, grader and configuration, budget, environment identifier and image digest where available |
| Evidence | Per-case outputs or outcomes, immutable file locations and verified digests |
Draw the performance conclusion at the scope your records support. For a historical citation, identify its exact source. For an agent comparison, report the paired case results and coverage under the stated conditions.