Research
A LeRobot episode holdout can use wider normalization statistics
A team can separate robot episodes for evaluation and still use preprocessing statistics whose scope reaches beyond the training episodes. We traced one LeRobot training path and checked a public metadata snapshot. In a hypothetical 40-training-episode and 10-evaluation-episode split, the saved state and action ranges each extend beyond the training-episode ranges in three of six dimensions. This is a comparison of saved metadata, not an observed training run or a model-performance result. It illustrates why a holdout review should include the statistics supplied to the processor as well as the episode list.
The split and the statistics are separate decisions
At LeRobot commit 7910c20, a configured eval_split groups selected episodes by their first listed task and assigns the last ceil(group size × eval_split) from each group to evaluation. The default configuration has no evaluation split; the example below deliberately sets one. The code constructs separate train and evaluation dataset objects using episode lists, the same dataset root and the same revision.
The dataset metadata loader separately loads meta/stats.json. In the inspected training path, a fresh, non-resumed Diffusion policy with no pretrained processor path receives the dataset metadata statistics when its processors are built. The default Diffusion configuration uses min-max normalization for state and action. The source trace is specific to that configuration and version; other policies, overrides, pretrained or resumed processors, formats and versions can differ.
We did not execute the dataset loader or a policy. The diagram describes the inspected source path under the stated conditions, not a record of an actual run.
One metadata example
The public lerobot/svla_so101_pickplace snapshot at revision f641879 describes 50 episodes and 11,939 frames. All 50 episode records share the same first-task grouping key. If no episodes are preselected or excluded and eval_split=0.2, the inspected rule would assign 40 episodes to training and 10 to evaluation. Their episode summaries declare 9,180 and 2,759 frames respectively.
The saved observation.state and action statistics each report a count of 11,939. Their min/max arrays match the aggregate of all 50 episode summaries within 1e-6 absolute and relative tolerance. Compared directly with the extrema aggregated from the hypothetical 40 training episodes, the saved ranges extend further in three of six dimensions for each feature. The count is unchanged when requiring a difference beyond math.isclose with 1e-6 relative and absolute tolerances.
| Feature | Dimensions where the saved range extends beyond the training-episode range | Narrowest training-episode range as a share of the saved range |
|---|---|---|
observation.state | 3 of 6 | 84.3% |
action | 3 of 6 | 84.2% |
Each percentage is the smallest range-width ratio among that feature's six dimensions; the two rows need not describe the same dimension. The bars compare widths, not where the ranges begin or end. The 40-episode calculation includes all frames declared for those episodes. An actual sampler may drop tail frames, sample windows or use other selection rules, so it is not a measurement of the positions used by a real optimizer.
The held-out episode summaries also extend the all-episode aggregate beyond the 40-training-episode aggregate in three dimensions for each feature. The saved statistics match the all-episode aggregate within the stated tolerance, but we did not independently establish how the saved file was produced or verify the underlying frame files. If those saved ranges are supplied to the fresh min-max processor described above, the wider bounds can change its scale. The example does not show what any historical job loaded, whether a dataset is suitable for a policy or whether a score changed.
Check the scope of a recomputation
The familiar guidance is to fit data-dependent preprocessing on the training set and apply the resulting transformation consistently to evaluation data. Scikit-learn's discussion of data leakage explains the general principle. A previous LeRobot issue also raised concerns about global statistics and validation; our check does not verify the issue's reported robot behavior or diagnose its cause. A different SO-101 dataset card already distinguishes training-only statistics from full-dataset metadata for a controlled evaluation. We did not check that dataset's own counts or files.
Inspect the recomputation path as carefully as the split. In the pinned version, the recompute_stats helper defaults to a numeric-data path that scans Parquet files under the dataset root and iterates their episode indices without a direct dataset.episodes filter. If those files still contain both partitions, passing an episode-selected object alone does not establish training-only numeric statistics. The helper writes statistics to that root and retains existing statistics for features it does not recompute; its default skips image/video features. We did not run the helper or alter a dataset. Its docstring already says it iterates all episodes, and the dataset-tools guide describes the recomputation operation.
Before treating a held-out result as independent, record the selected episode IDs, the dataset revision, the origin and feature scope of each statistic, and the processor configuration actually used. Verify that a proposed training-only computation reads the intended rows, and reuse the appropriate training-derived transformation for evaluation. Check separately for image, video and relative-action features, which may follow different paths. A pretrained or resumed processor may have its own saved statistics, so inspect the actual checkpoint and overrides rather than assuming this example's behavior.
How we checked
We read the pinned source without importing or running LeRobot. Our analysis script verifies source hashes, temporarily downloads the small metadata table, and calculates the hypothetical split and min/max comparisons with DuckDB 1.4.4. It removes the temporary table after a normal run. The aggregate results retain no task labels, individual episode IDs or per-episode statistics. The source-checker bundle contains a script that verifies selected pinned source structures without running upstream code, along with the upstream license and attribution. The figure generator reproduces the two diagrams from the aggregate. No recordings, raw frames, model, training or benchmark job were used.
To reproduce on macOS or Linux with Python 3.11 and DuckDB 1.4.4:
python3.11 -m venv .lerobot-stats-venv
.lerobot-stats-venv/bin/python -m pip install duckdb==1.4.4
.lerobot-stats-venv/bin/python analyze.py > observations-reproduced.json
unzip source-check.zip -d lerobot-source-check
.lerobot-stats-venv/bin/python lerobot-source-check/source-check.py
The dataset's pinned card labels it Apache-2.0; that label does not settle rights for every underlying asset. The source checker references short expressions from LeRobot; its pinned license and copyright attribution are included in the checker bundle. Sapien1 offers its original script contributions for download and local execution; no additional license to modify or redistribute those contributions is stated. Source dataset and upstream code terms are separate.