Research
How to check RL episode logs before comparing returns
Before comparing mean episode return, check which actions and episodes made it into the calculation. A collector can count a reset as a transition, store the next episode’s initial observation on the previous episode’s last transition, or miss an episode that ended at a time limit. Three small Gymnasium 1.2.1 replays show what to check.
Count applied actions and check the final observation. In a synchronous vector replay with two environments and five calls to step(), both reset modes returned 10 slot results. Under NextStep, two results came from resets and eight actions were applied. Under SameStep, all 10 actions were applied, and three results carried the reset observation where the transition needed the final observation. For those slots, read infos["final_obs"] using its _final_obs mask. Reconcile applied actions and reset-only results per environment slot, and inspect the stored next observation on ending transitions. Gymnasium 1.2.1 vector implementation
Keep both end flags. In a scalar replay, two of three episodes originally ended with truncated=True. One of the two also had terminated=True. After conversion to the legacy done API and back, only the cutoff-only episode still had truncated=True. Preserve each flag before conversion and check that both survive export and import, including when they are simultaneously true. Gymnasium 1.2.1 compatibility implementation
Reconcile recorded episodes with the outer step results. In a separate scalar replay, both wrapper orders returned the same observations, rewards and end flags. The two episodes had returns of 1 and 2. With RecordEpisodeStatistics outside TimeLimit, both episodes appeared in the statistics and their recorded mean was 1.5. With the statistics wrapper inside, only the naturally terminated episode appeared and the recorded mean was 1.0. The outer time limit set the cutoff flag after the inner statistics wrapper processed the step. Match the summaries to the episode IDs and end flags in the outer trace, then recompute the mean from the returns for the episodes your metric includes. Gymnasium 1.2.1 wrapper implementation
For each comparison, save the library version, wrapper order, reset mode where applicable, and the episode count used as the mean’s denominator. Verify the counts and per-episode returns on a short trace with a natural termination and a forced cutoff before using the aggregate.