# A reference-file screen changes GDPval’s public task mix


A benchmark score is easier to interpret when you know which tasks could enter it. In the public GDPval gold task table, each of 44 occupations starts with five tasks. But a comparison that requires a listed expert reference file cannot use every row in that table in the same way: 35 of 220 rows have an empty `deliverable_files` list.

That total is already documented by the [gbench GDPval implementation](https://github.com/google-deepmind/gbench/blob/d3e4283162d6c34353fa933284135d2a7ef3b9fe/docs/evals/gdpval.md). We counted how those empty lists are distributed across sectors and occupations in the same [pinned GDPval snapshot](https://huggingface.co/datasets/openai/gdpval/tree/11e7900cdcac61bc4daf59e65feb238acda98fbf). All 44 occupations still have at least one row with a listed file. But the remaining mix is not as evenly distributed as the starting table.

## The first screen

These counts ask only whether the table lists at least one expert deliverable path. A listed path is not proof that the file is accessible or renderable. An empty list does not establish whether a reference exists somewhere outside this table.

| Sector | Rows in public gold table | With a listed deliverable | With an empty list |
| --- | ---: | ---: | ---: |
| Finance and Insurance | 25 | 17 | 8 |
| Government | 25 | 22 | 3 |
| Health Care and Social Assistance | 25 | 22 | 3 |
| Information | 25 | 14 | 11 |
| Manufacturing | 25 | 25 | 0 |
| Professional, Scientific, and Technical Services | 25 | 24 | 1 |
| Real Estate and Rental and Leasing | 25 | 20 | 5 |
| Retail Trade | 20 | 16 | 4 |
| Wholesale Trade | 25 | 25 | 0 |
| **Total** | **220** | **185** | **35** |

![Rows with a listed expert deliverable by sector](sector-coverage.svg)

The bars show list presence, not verified file access or renderability.

The table contains 44 occupation-sector groups of five tasks each. Of those, 22 retain all five rows at this screen and 22 retain between two and four. Every occupation is still present, but presence alone no longer means that each has the same number of tasks.

The sector contrast is useful when reading an aggregate. Information and Manufacturing each account for 25 of the original 220 rows, or 11.4%. In a hypothetical equally task-weighted summary over exactly the 185 list-present rows, Information would account for 14/185, or 7.6%, and Manufacturing for 25/185, or 13.5%. Those are shares of tasks, not model scores or estimates of the value of different occupations.

## A later screen can narrow the set further

The 185-row calculation is not gbench’s final scoring denominator. Its documentation describes four more tasks whose listed deliverables do not render to text, leaving 181 reference-renderable tasks at that snapshot. We did not independently render those files or run gbench.

Even without knowing which sectors contain those four tasks, the counts place a useful bound on the documented reference-renderable set. If those are four distinct rows from the 185 list-present rows, at least 21 Manufacturing rows and at most 14 Information rows remain. Under equal task weights, those limits are 21/181 (11.6%) and 14/181 (7.7%), respectively. An actual run can have a different scored denominator because generation failures, judge failures and the number of attempts per task also matter. These counts do not describe OpenAI’s original human-graded results.

Other grading methods have different requirements. In [NeMo Gym's documented rubric mode at commit `5d6429f`](https://github.com/NVIDIA-NeMo/Gym/blob/5d6429f5b1362af47954d8cf72f03cbe443ff941/benchmarks/gdpval/README.md), an LLM judge grades a candidate deliverable against the task rubric. Its [preparation code](https://github.com/NVIDIA-NeMo/Gym/blob/5d6429f5b1362af47954d8cf72f03cbe443ff941/benchmarks/gdpval/prepare.py) does not copy the expert-output `deliverable_files` list, while its [rubric verifier](https://github.com/NVIDIA-NeMo/Gym/blob/5d6429f5b1362af47954d8cf72f03cbe443ff941/resources_servers/gdpval/app.py) reports `missing_rubric` if both rubric fields are empty. Any listed task-input files must be staged, and a candidate output and judge are still needed; the preparation call does not pin a dataset revision. We did not run NeMo Gym or measure its coverage. Its rubric score and gbench's wins-or-ties rate are different measures.

## What the judge can see

Reference availability is only one part of the comparison. The [gbench documentation](https://github.com/google-deepmind/gbench/blob/d3e4283162d6c34353fa933284135d2a7ef3b9fe/docs/evals/gdpval.md) already reports that 32 of the 220 rubrics exceed its default 8,000-character rubric limit. Its formatter keeps whole criteria in their original order until the next one would exceed the limit, then adds a truncation marker; if the first criterion alone exceeds the limit, it shows only part of that criterion. Character counts are not token counts.

We applied that pinned formatter to the task table. Among the 185 rows with a listed expert deliverable, 26 rubrics would be clipped at the default, leaving out 247 whole criteria across those rows. A listed deliverable is not necessarily renderable, and failed or empty generations can prevent a judge call, so these are not counts of judged comparisons. For illustration, changing only the formatter's limit to 12,000 or 16,000 characters would clip two or zero of those 185 rubrics, respectively. We did not run those settings through an evaluator or judge. Counts of omitted criteria do not establish their importance or whether a verdict would change; a larger limit can leave less room for the rest of a fixed context window. Inspect the clipping information recorded by the actual run alongside its reference and scoring counts.

The [rubric screen](rubric-screen.zip) reproduces these counts with DuckDB 1.4.4. It temporarily downloads the complete pinned task table and processes rubric text in memory; its output retains aggregate counts, not task IDs or rubric text. It removes the temporary table after a normal run. It executes only the inspected, hash-verified formatter from gbench in a restricted namespace, which is not a security sandbox. The archive includes the upstream license and notice. After the DuckDB setup below, using Python 3.11 to create the environment, download the archive and run on macOS or Linux:

```sh
unzip rubric-screen.zip -d rubric-screen
.gdpval-venv/bin/python rubric-screen/rubric-screen.py > rubric-observations.json
```

## Record the path from tasks to scores

When setting up or comparing an evaluation, record the dataset revision, the required reference type and each eligibility rule. Report counts by group at the starting table, after reference checks and after scoring. Keep separate reasons for an absent listed reference, an unreadable reference, a failed generation and an unusable judge verdict. A single aggregate can otherwise conceal which tasks stopped contributing.

Also state how the remaining tasks and attempts are weighted. Equal weight per task, equal weight per occupation and equal weight per attempt answer different questions when counts differ. None repairs missing evidence by itself; the appropriate summary depends on what the evaluation is meant to measure.

For a task set you are designing yourself, the [agent evaluation guide](https://sapien1.com/agent-evaluation) shows how to define observable checks and interpret repeated trials.

## Pin task inputs as well as the table

A pinned task table does not necessarily pin every file an evaluator downloads. In an [earlier public GDPval table at revision `4bf6426`](https://huggingface.co/datasets/openai/gdpval/tree/4bf642692ef091918ef20797ecca070924815ec1), 122 of 220 task rows list input files. All 262 listed input URLs point to the repository's moving `main` branch. The provider's metadata reports that 62 of those same paths had different sizes and blob IDs by [revision `11e7900`](https://huggingface.co/datasets/openai/gdpval/tree/11e7900cdcac61bc4daf59e65feb238acda98fbf); those 62 paths appear in 20 of the earlier table's rows. These are task inputs, distinct from the expert reference outputs in `deliverable_files` counted above.

This does not show that a particular evaluation used a changed input, that a file is corrupt or that a score changed. It does show why following a saved table's moving URLs later can select a different version of an input. We did not analyze input-file contents.

At [Inspect Evals commit `5c092d5`](https://github.com/UKGovernmentBEIS/inspect_evals/blob/5c092d54ea8591b4680d4dd593beab637f478ffc/src/inspect_evals/gdpval/util.py), the GDPval loader pins its task table to revision `a3848a2`. That table is byte-identical to the earlier table above, and the provider's input-file metadata matches too. The loader maps paired table URLs into sample file mappings; [Inspect's documentation](https://inspect.aisi.org.uk/datasets.html#sample-files) says those mappings can use URLs. This source path can therefore carry moving URLs alongside a fixed table. We did not test Inspect's file downloads or caching, or establish what any run used.

An evaluator can instead resolve dataset-relative paths at the same immutable revision as its table. The [gbench implementation inspected here](https://github.com/google-deepmind/gbench/blob/d3e4283162d6c34353fa933284135d2a7ef3b9fe/gbench/third_party/eval_suites/gdpval.py) explicitly supplies its pinned revision when fetching attachments. Record the revision and, where practical, verify the files actually used. [Hugging Face's download documentation](https://huggingface.co/docs/huggingface_hub/package_reference/file_download#resolve-a-revision) explains how to resolve a moving revision to a commit for consistent downloads.

The [input-URL analysis script](input-url-revisions.py) reproduces the counts using DuckDB 1.4.4. It downloads the earlier task table temporarily, removes it after a normal run and retains no task text, individual IDs, input paths or input contents in its output. An optional flag checks one path's response headers and the current `main` revision; its result can change as `main` moves. For that check, the script sends `HEAD` requests only, including through redirects, and leaves unavailable results unobserved. The Python standard-library checks compare the [Inspect-pinned table and metadata](inspect-dataset-check.py) and inspect the [pinned loader source](inspect-source-check.py) without running it. The table-hash check reads the complete table into memory but does not parse or retain its task content. The scripts do not request input-file bodies.

After the DuckDB setup below, run the checks on macOS or Linux with:

```sh
.gdpval-venv/bin/python input-url-revisions.py > input-url-observations.json
.gdpval-venv/bin/python inspect-dataset-check.py
.gdpval-venv/bin/python inspect-source-check.py
```

## How we counted

For the reference-availability census, we inspected the `train` Parquet table at GDPval revision `11e7900cdcac61bc4daf59e65feb238acda98fbf`, verified 220 distinct task IDs, and grouped only sector, occupation and the length of each `deliverable_files` list. The [analysis script](analyze.py) checks the source hash and reproduces the counts using DuckDB 1.4.4. It downloads the full table temporarily and removes it after a normal run; its output retains no prompts, rubrics, individual task IDs, deliverable paths or deliverables. No model, benchmark or reference renderer was run.

To reproduce locally with Python 3.9 or later on macOS or Linux, make sure `python3` selects that version:

```sh
python3 -m venv .gdpval-venv
.gdpval-venv/bin/python -m pip install duckdb==1.4.4
.gdpval-venv/bin/python analyze.py > observations.json
```

Sapien1 makes these scripts available to download and run locally; no additional license to modify or redistribute them is stated. The source dataset's terms are separate.

OpenAI describes the 220-task gold subset as [open-sourced](https://openai.com/index/gdpval/). The [pinned dataset card](https://huggingface.co/datasets/openai/gdpval/blob/11e7900cdcac61bc4daf59e65feb238acda98fbf/README.md) contains a benchmark canary and no license field; check the terms applicable to your own use rather than inferring them from a card label or from this count. The sector and occupation labels and benchmark remain attributed to their source.
