Skip to main content

Menu

How to specify an RL environment for an AI agent

A reinforcement learning (RL) environment gives an agent a place to attempt a task, take actions and receive feedback. For a business workflow, the environment can simulate records and tools so each attempt starts from a known state. A useful specification tells the builder what the agent can see and do, how the state changes, when an attempt ends and how the result is scored.

One attempt is an episode. An observation is information the environment gives the agent, and a reward is a numeric signal about its behavior. OpenEnv's core concepts describe an environment in terms of reset, actions, observations and rewards.

The environment supplies the task and feedback. An RL trainer can use rewards to update an agent's decision-making policy; an evaluation can instead use an environment to measure an agent without updating it. See OpenEnv's training overview and our guide to agent evaluation.

Example: an agent reviews a meal expense

Suppose an agent is asked to review a meal expense and record the next step. The following is a fictional environment specification; it has not been built or tested. Its records and actions are simulated and cannot make a payment.

The task given to the agent is: “Review expense EX-104 under the policy and record the next step.”

Starting state and reset

At the start of every episode, create independent copies of these records:

FieldEX-104EX-105
TypeMealMeal
Amount$34$20
ReceiptPresentPresent
Statuspendingpending
Assigned managerM1M2
Manager approvalNoneNone

EX-105 is an unrelated record; it helps the verifier check that the agent leaves other work alone. Reset also clears all approval requests and the action log, then assigns a fresh episode ID. No manager responds during the episode. Episodes must keep their mutable records, requests and logs separate, including when they run at the same time. OpenEnv's core API reference describes isolation of concurrent sessions as a requirement for its concurrent environments.

The policy available to the agent says that a meal with a receipt is eligible up to and including $50. A meal above $30 requires the assigned manager's approval before it can be approved. When that approval is required but missing, the agent should request it and set the expense to awaiting_approval.

For EX-104, the receipt is present and $34 is within the meal limit, but the amount exceeds the approval threshold. Since M1 has not approved it, the expected next step is to request M1's approval and leave EX-104 awaiting approval.

Observations and allowed actions

At reset, the agent receives the task, the available action schemas and a count of ten actions remaining. Each nonterminal response, including an invalid-call error before the limit, shows the number of actions still available. A terminal response marks the episode ended. The simulated action names below are illustrative and belong to this example.

ActionResult
read_expense(id)Returns the named expense's type, amount, receipt, status and assigned manager.
read_policy()Returns the policy above.
read_approval(id)Returns the named expense's approval and request state.
request_approval(id, manager)Creates a pending request for the expense and manager. A repeated request for the same pair leaves one pending request. It does not change the expense status.
set_status(id, status)Sets the expense to pending, awaiting_approval, approved or rejected.
finish(reason)Ends the episode and records the agent's reason.

The agent receives each action's response and reward. Record every call and its result, including errors, in the episode's action log. The verifier, its baseline and the log are not visible or editable through these actions. The only expense IDs are EX-104 and EX-105; the only managers are M1 and M2. The mock accepts either known manager for a request and permits status changes among the listed values, even when a change violates the policy. That lets the verifier observe and score an unsafe decision. An invalid call returns an error, changes neither the expense records nor the approval requests, is recorded in the action log, and counts as an action.

Reward and termination

The agent has at most 10 actions, including finish. Every nonterminal action receives a reward of 0. When the agent finishes, the verifier gives +1 only if all of these checks pass:

  1. EX-104 ends in awaiting_approval with its $34 amount unchanged.
  2. Exactly one pending request exists, for EX-104 and M1, and the finish reason is awaiting_approval.
  3. EX-105 is unchanged, and the action log contains no successful set_status call for it.
  4. The action log contains no transition of EX-104 to approved or rejected, and no request_approval call for another expense or manager.

Any other finished episode receives 0. If the tenth action does not successfully execute finish, the episode ends with 0. A sandbox crash or unavailable backend makes the trial invalid and unscored; record those failures separately from agent outcomes.

The ten-action limit is part of this task. If a rollout collector ends an unfinished attempt earlier for its own limit, record that cutoff and its last valid observation separately from a scored task ending. Gymnasium calls this external ending truncation and distinguishes it from task termination. That distinction matters to learning methods that bootstrap from unfinished states. The OpenEnv 0.10.0 base observation lists a done flag. AgileRL's text observation documents carrying an additional truncated field to reconstruct a five-part response. Check how your own environment, adapter and learner preserve the ending reason. A sandbox crash or unavailable backend remains invalid and unscored as described above.

This is a sparse reward: the positive signal arrives at the end of a successful episode. OpenEnv's reward guidance recommends starting with a simple outcome signal and explains the tradeoff: sparse signals can be slow to learn from, while intermediate rewards can encourage an agent to optimize a proxy rather than the intended result.

A passing trace

This six-action sequence satisfies the checks. The table summarizes the observations and resulting state. Nonterminal responses also include the remaining-action count.

ActionObservation or resulting state
1. read_expense("EX-104")A $34 meal with a receipt, assigned to M1 and still pending.
2. read_policy()Meals above $30 require the assigned manager's approval.
3. read_approval("EX-104")No approval and no request.
4. request_approval("EX-104", "M1")A pending request for M1 is created.
5. set_status("EX-104", "awaiting_approval")EX-104 is awaiting approval.
6. finish("awaiting_approval")The checks pass; terminal reward is +1.

A final state that should fail

Consider this eight-action sequence from the same reset state:

1. read_expense("EX-104")
2. read_policy()
3. read_approval("EX-104")
4. set_status("EX-104", "approved")
5. set_status("EX-104", "pending")
6. request_approval("EX-104", "M1")
7. set_status("EX-104", "awaiting_approval")
8. finish("awaiting_approval")

The expense, request and finish reason now match the passing sequence. But action 4 approved the expense without the manager's approval. The log records that transition, so the verifier returns 0. A check of the final state alone would miss it. Anthropic's guide to agent evaluations distinguishes the outcome from the trace of actions that produced it.

A specification to adapt

Give your environment builder enough detail to answer these questions for one representative task:

  • Task: What is the agent asked to do, and why does the workflow matter?
  • Reset: Which records, files, policies and external responses exist at the start? What must be cleared, and how are simultaneous episodes isolated?
  • Observations: What can the agent read, and what stays with the verifier? Make the rules needed to perform the task available to the agent.
  • Actions and transitions: Which calls are allowed, what does each change or return, and what happens on invalid or repeated calls?
  • Checks and reward: What outcome earns each reward? Which unrelated state must stay unchanged, and which actions in the log must be allowed or forbidden?
  • Termination: What finishes an episode, what is the action or time limit, and how are infrastructure failures recorded?
  • Coverage: Which examples should exercise policy boundaries and failure cases? Which cases will be held out for evaluation?

Check the environment before using it for training

Ask the builder to exercise passing and failing traces, including an action that is later reversed. Check that a reset removes the previous episode's changes, parallel episodes do not share mutable state, and the agent cannot change the verifier or its log. Keep the policy, fixtures, verifier, agent harness and backend versioned so a result can be reproduced. For an evaluation example of why task declarations, runner overrides and backend policies should be recorded separately, see our Terminal-Bench configuration study.

One fixed expense can be memorized. Add varied cases, such as amounts at the approval threshold and missing receipts, with an expected outcome for each; keep separate cases for evaluation. Run the passing trace to verify that the reward works, then measure how often initial agent rollouts reach it before relying on it for training.

Sapien1 helps teams scope task environments and evaluate agents. Start with a workflow, a representative task and the outcome you need to observe.