# Terminal-Bench resource-settings census

Created: 2026-11-17
Updated: 2026-11-17

## Question and population

Which task-specific declarations would a single sandbox preset change in Terminal-Bench v4.1.0? This study compares published settings so evaluation teams can select and document an environment configuration. It does not estimate actual resource use, minimum hardware, price, completion time, model performance or failure probability.

The population is all 66 members of the official v4.1.0 release manifest, not every task-like file in its repository. The release was published on 3 November 2026. We read configuration metadata on 17 November 2026. The [official dataset card](https://huggingface.co/datasets/harborframework/terminal-bench) identifies the release and its [manifest](https://huggingface.co/datasets/harborframework/terminal-bench/resolve/v4.1.0/registry.json). The source repository is pinned to commit `4070d6cef56a7d20cb5dc98d5e10ada88a6b9440` in `harbor-framework/terminal-bench`.

The manifest SHA-256 is `8809dc185dcb431e3a92ea1eaeb8f82ec63aa98050051815d0d6005ce2c99262`. Each observation includes a pinned source URL and SHA-256 of the complete source `task.toml`. Legacy tasks, CI fixtures and templates outside the release manifest are excluded. No task, container, model or verification script was run.

## Extraction

The unit is one manifest task. The collector parses TOML with Python's standard library and records explicit top-level environment CPU, memory, storage, GPU, GPU-type, TPU and build-timeout values. It preserves omitted fields as omitted. Agent and verifier timeout values remain separate. Numeric `1200` and `1200.0` are equivalent in summary counts; the source observations preserve their parsed types.

Verifier-environment block presence is recorded separately from resource-field presence. The collector also retains schema_version, legacy top-level version and task version separately; it flags legacy resource aliases, steps and additional environment blocks. No legacy resource aliases or steps were found in this release. Six tasks have explicit verifier-environment blocks. The full observations retain these blocks' selected fields; the main CPU/RAM distributions concern the task's top-level environment only.

The 66 source TOML files total 85,966 bytes. The published extraction contains selected configuration facts, field names, task identifiers and source hashes. It excludes task instructions, solutions, tests, author contact information and environment-variable values. Full source files and benchmark images are not redistributed in this package.

## Constructed comparison

For each of CPU, memory, storage and GPU count, we selected the most frequent explicit value in the 66 top-level environment declarations: 2 CPUs, 4,096 MB memory, 10,240 MB storage and 0 GPUs. Combining those values creates our comparison preset. It is not a Harbor default, a vendor recommendation or an observed evaluation setup.

A task enters the “above the comparison preset” set if at least one explicitly present numeric value is greater than the corresponding preset value. We take the union of task identifiers, not the sum of per-field counts. Missing GPU values neither count as zero nor create a positive exception. The eight tasks with omitted GPU count remain identifiable in the source data. The 38-task union measures a difference in declarations; it does not predict whether the task can run or pass under any preset.

We also count distinct declaration patterns across CPU, memory, storage, GPU count, GPU type and TPU, excluding timeouts. The raw pattern count is 27. Setting omitted GPU count to zero reduces that count to 22. This sensitivity is why the article leads with individual resource fields and a transparent comparison, rather than presenting the pattern count as a hardware requirement count.

## Runner interpretation

We reviewed Harbor source at commit `ec6aeab7fe0eb51df52b4e72c524daf5ea9cc5f4`, whose project metadata identifies version 0.26.0. These are code-reading findings, not an executed validation of all tasks or backends.

- The [task parser](https://github.com/harbor-framework/harbor/blob/ec6aeab7fe0eb51df52b4e72c524daf5ea9cc5f4/src/harbor/models/task/config.py#L435) leaves CPU, memory, storage and GPU counts nullable and migrates legacy memory/storage strings. Its schema-version field is a string; the version-migration path does not itself establish that every task can execute.
- The [base environment](https://github.com/harbor-framework/harbor/blob/ec6aeab7fe0eb51df52b4e72c524daf5ea9cc5f4/src/harbor/environments/base.py#L468) applies trial overrides. Omitted GPU count resolves to zero there; CPU/memory ignore policies can suppress the values. We retain omissions in the dataset even when this runner maps one to a value.
- The [verifier resolver](https://github.com/harbor-framework/harbor/blob/ec6aeab7fe0eb51df52b4e72c524daf5ea9cc5f4/src/harbor/models/task/verifier_mode.py#L15) uses a supplied verifier environment as a whole; otherwise it copies the task environment. It does not merge an explicit verifier block field by field with agent settings. [Trial creation](https://github.com/harbor-framework/harbor/blob/ec6aeab7fe0eb51df52b4e72c524daf5ea9cc5f4/src/harbor/trial/trial.py#L1104) still passes environment overrides.
- The [Docker adapter](https://github.com/harbor-framework/harbor/blob/ec6aeab7fe0eb51df52b4e72c524daf5ea9cc5f4/src/harbor/environments/docker/docker.py#L568) maps auto CPU/memory policies to limits and explicitly ignores GPU-type selection. Its [Compose resource writer](https://github.com/harbor-framework/harbor/blob/ec6aeab7fe0eb51df52b4e72c524daf5ea9cc5f4/src/harbor/environments/docker/__init__.py#L36) writes CPU, memory and GPU-count constraints, not a storage cap. Existing Compose configuration and other providers can behave differently.
- [Timeout resolution](https://github.com/harbor-framework/harbor/blob/ec6aeab7fe0eb51df52b4e72c524daf5ea9cc5f4/src/harbor/trial/trial.py#L695) and [budget computation](https://github.com/harbor-framework/harbor/blob/ec6aeab7fe0eb51df52b4e72c524daf5ea9cc5f4/src/harbor/trial/trial.py#L1724) allow overrides, maximums and multipliers. Agent, verifier and build timeout declarations are not observed durations. Their sum is not asserted as a bound on total trial duration.

The census does not inspect every Dockerfile, Compose sidecar, image or external service. Those may add resource requirements and constraints outside the top-level configuration fields. No conclusion about benchmark validity or score changes follows from this study.

## Reproduce

With Python 3.11 or later, run `python3 census.py` to regenerate `census-results.json` from the supplied `census.json`. Run `python3 census.py --collect` to refetch the fixed manifest and all 66 pinned TOML files before analysis. The collector uses at most three simultaneous reads, caps the manifest at 300 KB and each task at 100 KB, and fails if manifest membership or the pinned manifest hash differs. It makes no provider mutation and does not execute downloaded code.

Run `python3 render-assets.py` to regenerate the two original SVG figures. Figure generation uses ReportLab; the published masters were rendered with ReportLab 4.4.9 and Python 3.12.14. The distribution figure reads the analysis results directly. The second figure is an author-created diagram of the configuration record, not empirical data. Both SVGs retain text, accessible titles and descriptions.

The source benchmark uses the [Apache License 2.0](https://github.com/harbor-framework/terminal-bench/blob/4070d6cef56a7d20cb5dc98d5e10ada88a6b9440/LICENSE). Source copyrights remain with their authors. This package is Sapien1's selected factual extraction and original analysis; it does not grant rights to third-party task assets or container contents.
