Skip to main content

Menu

Research

A reference-file screen changes GDPval’s public task mix

A benchmark score is easier to interpret when you know which tasks could enter it. In the public GDPval gold task table, each of 44 occupations starts with five tasks. But a comparison that requires a listed expert reference file cannot use every row in that table in the same way: 35 of 220 rows have an empty deliverable_files list.

That total is already documented by the gbench GDPval implementation. We counted how those empty lists are distributed across sectors and occupations in the same pinned GDPval snapshot. All 44 occupations still have at least one row with a listed file. But the remaining mix is not as evenly distributed as the starting table.

The first screen

These counts ask only whether the table lists at least one expert deliverable path. A listed path is not proof that the file is accessible or renderable. An empty list does not establish whether a reference exists somewhere outside this table.

SectorRows in public gold tableWith a listed deliverableWith an empty list
Finance and Insurance25178
Government25223
Health Care and Social Assistance25223
Information251411
Manufacturing25250
Professional, Scientific, and Technical Services25241
Real Estate and Rental and Leasing25205
Retail Trade20164
Wholesale Trade25250
Total22018535
Rows with a listed expert deliverable by sector
View full-size figure

The bars show list presence, not verified file access or renderability.

The table contains 44 occupation-sector groups of five tasks each. Of those, 22 retain all five rows at this screen and 22 retain between two and four. Every occupation is still present, but presence alone no longer means that each has the same number of tasks.

The sector contrast is useful when reading an aggregate. Information and Manufacturing each account for 25 of the original 220 rows, or 11.4%. In a hypothetical equally task-weighted summary over exactly the 185 list-present rows, Information would account for 14/185, or 7.6%, and Manufacturing for 25/185, or 13.5%. Those are shares of tasks, not model scores or estimates of the value of different occupations.

A later screen can narrow the set further

The 185-row calculation is not gbench’s final scoring denominator. Its documentation describes four more tasks whose listed deliverables do not render to text, leaving 181 reference-renderable tasks at that snapshot. We did not independently render those files or run gbench.

Even without knowing which sectors contain those four tasks, the counts place a useful bound on the documented reference-renderable set. If those are four distinct rows from the 185 list-present rows, at least 21 Manufacturing rows and at most 14 Information rows remain. Under equal task weights, those limits are 21/181 (11.6%) and 14/181 (7.7%), respectively. An actual run can have a different scored denominator because generation failures, judge failures and the number of attempts per task also matter. These counts do not describe OpenAI’s original human-graded results.

Other grading methods have different requirements. In NeMo Gym's documented rubric mode at commit 5d6429f, an LLM judge grades a candidate deliverable against the task rubric. Its preparation code does not copy the expert-output deliverable_files list, while its rubric verifier reports missing_rubric if both rubric fields are empty. Any listed task-input files must be staged, and a candidate output and judge are still needed; the preparation call does not pin a dataset revision. We did not run NeMo Gym or measure its coverage. Its rubric score and gbench's wins-or-ties rate are different measures.

What the judge can see

Reference availability is only one part of the comparison. The gbench documentation already reports that 32 of the 220 rubrics exceed its default 8,000-character rubric limit. Its formatter keeps whole criteria in their original order until the next one would exceed the limit, then adds a truncation marker; if the first criterion alone exceeds the limit, it shows only part of that criterion. Character counts are not token counts.

We applied that pinned formatter to the task table. Among the 185 rows with a listed expert deliverable, 26 rubrics would be clipped at the default, leaving out 247 whole criteria across those rows. A listed deliverable is not necessarily renderable, and failed or empty generations can prevent a judge call, so these are not counts of judged comparisons. For illustration, changing only the formatter's limit to 12,000 or 16,000 characters would clip two or zero of those 185 rubrics, respectively. We did not run those settings through an evaluator or judge. Counts of omitted criteria do not establish their importance or whether a verdict would change; a larger limit can leave less room for the rest of a fixed context window. Inspect the clipping information recorded by the actual run alongside its reference and scoring counts.

The rubric screen reproduces these counts with DuckDB 1.4.4. It temporarily downloads the complete pinned task table and processes rubric text in memory; its output retains aggregate counts, not task IDs or rubric text. It removes the temporary table after a normal run. It executes only the inspected, hash-verified formatter from gbench in a restricted namespace, which is not a security sandbox. The archive includes the upstream license and notice. After the DuckDB setup below, using Python 3.11 to create the environment, download the archive and run on macOS or Linux:

unzip rubric-screen.zip -d rubric-screen
.gdpval-venv/bin/python rubric-screen/rubric-screen.py > rubric-observations.json

Record the path from tasks to scores

When setting up or comparing an evaluation, record the dataset revision, the required reference type and each eligibility rule. Report counts by group at the starting table, after reference checks and after scoring. Keep separate reasons for an absent listed reference, an unreadable reference, a failed generation and an unusable judge verdict. A single aggregate can otherwise conceal which tasks stopped contributing.

Also state how the remaining tasks and attempts are weighted. Equal weight per task, equal weight per occupation and equal weight per attempt answer different questions when counts differ. None repairs missing evidence by itself; the appropriate summary depends on what the evaluation is meant to measure.

For a task set you are designing yourself, the agent evaluation guide shows how to define observable checks and interpret repeated trials.

Pin task inputs as well as the table

A pinned task table does not necessarily pin every file an evaluator downloads. In an earlier public GDPval table at revision 4bf6426, 122 of 220 task rows list input files. All 262 listed input URLs point to the repository's moving main branch. The provider's metadata reports that 62 of those same paths had different sizes and blob IDs by revision 11e7900; those 62 paths appear in 20 of the earlier table's rows. These are task inputs, distinct from the expert reference outputs in deliverable_files counted above.

This does not show that a particular evaluation used a changed input, that a file is corrupt or that a score changed. It does show why following a saved table's moving URLs later can select a different version of an input. We did not analyze input-file contents.

At Inspect Evals commit 5c092d5, the GDPval loader pins its task table to revision a3848a2. That table is byte-identical to the earlier table above, and the provider's input-file metadata matches too. The loader maps paired table URLs into sample file mappings; Inspect's documentation says those mappings can use URLs. This source path can therefore carry moving URLs alongside a fixed table. We did not test Inspect's file downloads or caching, or establish what any run used.

An evaluator can instead resolve dataset-relative paths at the same immutable revision as its table. The gbench implementation inspected here explicitly supplies its pinned revision when fetching attachments. Record the revision and, where practical, verify the files actually used. Hugging Face's download documentation explains how to resolve a moving revision to a commit for consistent downloads.

The input-URL analysis script reproduces the counts using DuckDB 1.4.4. It downloads the earlier task table temporarily, removes it after a normal run and retains no task text, individual IDs, input paths or input contents in its output. An optional flag checks one path's response headers and the current main revision; its result can change as main moves. For that check, the script sends HEAD requests only, including through redirects, and leaves unavailable results unobserved. The Python standard-library checks compare the Inspect-pinned table and metadata and inspect the pinned loader source without running it. The table-hash check reads the complete table into memory but does not parse or retain its task content. The scripts do not request input-file bodies.

After the DuckDB setup below, run the checks on macOS or Linux with:

.gdpval-venv/bin/python input-url-revisions.py > input-url-observations.json
.gdpval-venv/bin/python inspect-dataset-check.py
.gdpval-venv/bin/python inspect-source-check.py

How we counted

For the reference-availability census, we inspected the train Parquet table at GDPval revision 11e7900cdcac61bc4daf59e65feb238acda98fbf, verified 220 distinct task IDs, and grouped only sector, occupation and the length of each deliverable_files list. The analysis script checks the source hash and reproduces the counts using DuckDB 1.4.4. It downloads the full table temporarily and removes it after a normal run; its output retains no prompts, rubrics, individual task IDs, deliverable paths or deliverables. No model, benchmark or reference renderer was run.

To reproduce locally with Python 3.9 or later on macOS or Linux, make sure python3 selects that version:

python3 -m venv .gdpval-venv
.gdpval-venv/bin/python -m pip install duckdb==1.4.4
.gdpval-venv/bin/python analyze.py > observations.json

Sapien1 makes these scripts available to download and run locally; no additional license to modify or redistribute them is stated. The source dataset's terms are separate.

OpenAI describes the 220-task gold subset as open-sourced. The pinned dataset card contains a benchmark canary and no license field; check the terms applicable to your own use rather than inferring them from a card label or from this count. The sector and occupation labels and benchmark remain attributed to their source.