Skip to main content

Menu

Evaluate an AI dataset for the job it needs to do

Dataset evaluation checks whether the data you already have can support a particular training, retrieval or evaluation decision. Start with the intended task, then examine the examples, their context and labels, their coverage and how they are divided.

For example, a test set may look complete while containing cases the model has already seen during training. The issue matters when the test is meant to estimate performance on new cases. A useful review identifies what was checked, what needs repair and what the remaining evidence can support.

Start with the decision

Write down what you want to do with the dataset. The same records can be useful for one purpose and insufficient for another.

Intended useA question to answer
Training or fine-tuningDo the examples and labels describe the behavior the model should learn?
Evaluating a model or agentAre the expected outcomes checkable, and are the test cases independent of material used to train or tune the system?
RetrievalIs the content current, interpretable and available to the system when it needs it?

Record the dataset version, source, date range, fields and known restrictions. A dataset card can document contents, context and metadata; compare it with the actual files and any relevant usage terms.

For a concrete documentation check, consider the lerobot/svla_so101_pickplace repository at revision f641879, inspected on 17 November 2026. Its dataset card includes a metadata example with codebase_version v2.1 and episode-based file paths. The actual metadata at the same revision lists codebase_version v3.0 and chunked-file paths; the data directory contains a file named in that pattern. Check the metadata and files for the repository revision you intend to use before relying on a path copied from a card. This comparison does not establish the quality of the examples or their fitness for training.

Before combining robot recordings, check what the action and state channels mean and how much time a fixed number of steps represents. FineART and a simulated ALOHA dataset both declare aloha and 14-element float32 action and state vectors. FineART declares 30 frames per second and the simulated dataset declares 50. If each step advances one frame interval at the declared rate, a 15-step horizon covers 0.5 seconds in one dataset and 0.3 seconds in the other. Check recorded timing and the way your pipeline selects steps before treating windows from both sources as equivalent. FineART metadata · Simulated ALOHA metadata

Check splits and context: a worked example

Suppose a team wants to use an existing labeled support dataset as a release check for an agent handling new conversations. The dataset has 120 examples: 90 in training and 30 in the proposed test set. Each example should include a case ID, conversation ID, the applicable policy version and an expected next action.

This example is fictional. The team checks the 30 test cases and compares them with the training set. It finds:

CheckFindingWhat it means for this decision
Repeated casesFour test cases are confirmed copies of cases in training.Those four are not unseen cases.
Shared conversationsThree other test cases have different case IDs but belong to conversations present in training.These are additional cross-split matches for a test intended to cover new conversations.
Policy contextTwo of the remaining 23 test cases lack a link to the policy version.A reviewer cannot yet check their expected actions against the relevant policy using the recorded context.

The first two checks flag seven distinct test cases, leaving 23 without those matches. Two of those 23 lack policy context, leaving 21 test cases not flagged by these checks. That count does not establish that their labels are correct, that coverage is sufficient or that 21 cases are enough for a release decision.

The team should hold use of the current set as an independent new-conversation release check. Quarantine the flagged cases for review and recover or resolve the missing context. Check what the model and development team have already seen: rearranging files alone does not make previously used cases unseen. For a new split, keep each conversation together and check test cases against recorded training and model-selection data. If earlier training provenance cannot be checked, record that limit. If the existing results influenced tuning, reserve fresh cases that were not used in those decisions; retrain with separated data if needed.

Google's guidance on dataset splits explains how duplicate training and test examples can make results look better than performance on new examples. scikit-learn's grouped cross-validation guidance describes keeping related observations from the same group together when the goal is to assess performance on unseen groups. The group that matters depends on the question: it might be a conversation, customer or document. If an image-caption test is meant to use images absent from the training split, keep all captions for each image together. Splitting caption rows separately can otherwise put the same image in both training and test. For a future-facing test, use a time-aware split so training does not include later observations than the test period.

Suppose the team also wants to test new conversations after a cutoff. In a fresh split, conversation A has records on both sides of the cutoff; conversation B's first recorded case appears afterward. One option is to exclude A from both sets and reserve B for the test, provided B's records were not previously used for training or model selection. Keep training data before the cutoff and check the full test set for earlier use. Record exclusions and check whether the remaining cases cover the intended use. A separate, time-respecting test can assess continuing conversations. If the available history is incomplete, it may not be possible to confirm that B is new.

Check what a split tool actually assigns before treating an unspecified fraction as a holdout. For example, the LeRobot fraction helper at revision 7910c20 assigns all 50 episodes in a fictional set when given train: 0.6 followed by test: 0.2: 30 to train and 20 to test, with none left unassigned. Other tools or versions may behave differently, and small sets can require rounding or reassignment. Inspect the actual membership; when groups must stay together, choose and record episode lists that respect those groups.

A LeRobot research note traces how a fresh Diffusion processor can receive saved normalization statistics and compares those ranges with summaries of the training episodes in a hypothetical holdout. Check the origin and scope of the statistics your own processor loads.

Review labels and coverage

For labeled data, compare a sample of expected answers with the instructions and evidence used to create them. A domain reviewer can investigate missing context, inconsistent labels and cases where the correct answer is genuinely ambiguous. Record disagreements and how they were resolved; agreement between reviewers is useful information, but it is not the same as proof that a label is correct. Research on annotation quality management discusses review, agreement and adjudication as distinct parts of the process.

Numeric class labels need a mapping from IDs to class names and definitions. Compare those definitions and IDs across dataset versions and splits, and against the mapping the model expects. Document any translation and how you handle missing or merged classes. Ultralytics Platform's dataset guide notes that merging or deleting classes can shift later IDs, so older exports can stop matching the updated class list. Verify the mapping in your own exports and pipeline.

Check whether the dataset covers the situations relevant to the intended use: common requests, important exceptions, languages, time periods or other meaningful groups. A group with few examples may need more data or a separate analysis. It does not automatically need the same number of examples as every other group.

Document how review examples were selected. A hand-picked set can help expose a problem, but its error rate should not be treated as the error rate of the whole dataset. Google's sampling guidance describes how selection can distort conclusions about a larger population.

Even a random sample can miss errors. Suppose a separate fictional batch has 120 records, six of which have incorrect labels. If you select 20 distinct records uniformly at random, the sample will contain none of those six about 33% of the time. Selecting 47 reduces the chance of missing all six below 5% if reviewers identify every incorrect label in the selected records. This is a planning example that assumes six errors; a clean sample does not tell you there are exactly six, or prove that the rest are error-free. Choose a sample size for the error frequency and chance of missing it that matter to your decision, and record how examples were selected and reviewed. NIST's sampling primer discusses selection and sample size; research on annotation quality explains statistical quality estimation and acceptance sampling.

Decide what to change

Turn the findings into actions the team can take:

  • Repair: Recover missing context, correct verified errors or update the labeling instructions, then recheck the affected records.
  • Restrict: Use the dataset only for the tasks or slices it can support, and describe known gaps.
  • Hold: Keep the dataset out of the intended training or release decision until a material issue is resolved.
  • Add data: Collect or source cases for missing situations, or create a fresh test set that has not been used for training or tuning.

Keep the original and revised versions identifiable. Record the checks, selection method, findings, reviewer decisions and unresolved questions so another team can understand what changed and why.

When a COCO object detection dataset is converted, compare the original and exported image maps. Check that each source image has one distinct numeric ID in the export and that its annotations and predictions still refer to the intended image. With a stored ID map configured, the tested FiftyOne exporter converts 01 and 1 to the same integer. The tested COCO index keeps the last image row for that ID and groups annotations under it. Resolve duplicate IDs before scoring. FiftyOne exporter · COCO index

Keep the original IDs if possible. If IDs must change, preserve their relative numeric order and replay average precision (AP) on the original and converted data. Use the same evaluator revision, settings and unrounded prediction scores, with the predictions mapped to the corresponding images in each run. Include a small check with two equal-confidence detections in the same category on different images, one with a ground-truth match and one false positive. The tested COCO evaluator sorts IDs and keeps that order for cross-image score ties, so renumbering can change AP while detection matches stay the same. Resolve any score difference before comparing the two versions. COCO image order · COCO score accumulation

Sapien1 helps teams assess datasets and plan evaluations around the work an AI system needs to do. For the next step after reviewing the data, see our agent evaluation guide. If the review exposes a data gap, our data sourcing guide can help describe what to find.