Evaluation datasets and sandbox replay

Start with your first evaluation for a runnable workflow. See API and permissions for endpoint and credential requirements.

Datasets — a frozen set of cases

A dataset is a named collection of cases you promoted out of real recorded runs. Its metadata is editable; its versions are not. Promoting cases seals a new version containing everything the current version had plus the new cases, and removing cases seals a new version without them — so a score that cites a version keeps meaning exactly what it meant when it was recorded.

Before using the example, create a dataset named regression_set (or replace that path segment with your existing dataset ID). Set RUN_ID to a recorded run in the same project. ANECTICO_USER_TOKEN must be an authenticated user-session access token with scores:write and evals:write, not an API key or refresh token. Prefer the signed-in CLI if you do not manage OAuth tokens in an integration; see human judgments. The example also requires jq.

Only a human-reviewed production run can be promoted. Record a judgment first, then pass the returned annotation id with that exact run:

# Preview a directly signed-in person's judgment first.
anectico evals annotations add "$RUN_ID" --metric dataset_eligibility --label reviewed
# After review, repeat the identical command with --confirm-token <token>.
# Read annotations to obtain the saved annotation_id, then preview the promotion:
anectico evals datasets promote regression_set --file reviewed-subjects.json --idempotency-key <key>
# Confirm the identical file and key with --confirm-token <token>.

The promotion file names each subject_kind: "SUBJECT_KIND_AGENT_RUN", the exact run id as subject_ref, and the saved human annotation id as the mandatory review_annotation_id. Its server preview returns the complete current and proposed frozen case set, with withheld content classes named. No dataset version is sealed until confirmation.

The review must belong to the same project and name the same run. The frozen item records the review id, authenticated reviewer, and review time. Promotion refuses a missing or mismatched review before it freezes or stores anything; holding evals:write does not itself count as a review.

Each case's id is a digest of its own content, which has two consequences worth knowing:

• Promoting the same run twice gives you one case, not two.
• A case keeps the same id as it is carried into later versions, so "this case scored 0.4 in v1 and 0.9 in v3" is a question you can ask.

A version number is always exact. Reading a version's cases requires you to name it; there is no "latest", because a set of cases that can change under a comparison is not a set of cases.

Nothing is deleted by a promotion or a removal. A version cited by a stored score cannot be removed, and neither can a dataset that has one. There is one exception, and it is not yours to make: erasing a person.

A version can be retired

A case is a frozen copy of a run, and that copy includes who the run was recorded for. When you erase a person, a case that was their run is deleted from every version that holds it. A version that lost a case is no longer the set anyone measured against, so it is retired.

Every version carries a state:

state	Meaning
DATASET_VERSION_STATE_ACTIVE	The version holds the set it was sealed with.
DATASET_VERSION_STATE_RETIRED	One of its cases was removed. retirement_cause says why (DATASET_VERSION_RETIREMENT_CAUSE_PERSON_ERASED) and retired_at says when. item_count and content_digest still describe the set as it was sealed.

A retired version cannot be used: listing its cases, replaying one of them and creating an experiment over it are refused with FAILED_PRECONDITION and the reason EVALUATION_DATASET_VERSION_RETIRED. The refusal is permanent. A version that did not hold the person's run is not affected.

To bring the dataset back into use, cut a new version: promote a run, or remove a case. To seal exactly the cases that are left, send a removal with an empty item_ids; that is accepted only when the dataset's current version is retired.

What a case contains. A frozen, self-contained projection of the run — enough to score it again later without the original run still existing. If your project's content settings did not permit a class of content to be copied, the case records which classes were withheld rather than being quietly short: a case frozen without its content would score differently for ever, and you would have no way to tell it from one that never had any.

Replaying one case in a sandbox

A replay re-executes the recorded decision sequence of one frozen case inside a sandbox that cannot reach anything outside this platform. It saves no replay artifact. A replay is a pure function of a frozen case and the plan you send, and both are fixed — so asking again reproduces the same answer byte for byte, and you never have to keep one in order to get it back.

The plan is yours to declare

Nothing here knows what your tools do, so the plan is where you say it. Each policy names a tool by the name the recording carries and classifies it by what the call does outside the sandbox:

Effect	Meaning	What the sandbox does
SANDBOX_EFFECT_NONE	a read-only call	replays the recorded answer, if there is one
SANDBOX_EFFECT_INTERNAL	writes inside this platform	answers with a marked stub
SANDBOX_EFFECT_EXTERNAL	leaves this platform	answers with a marked stub

default_effect covers any tool no policy names. Unset means external, and so does a call the recording does not name — not even an explicitly open default reaches that one. The recoverable mistake is stubbing a read; the unrecoverable one is letting a write through because nobody classified it.

A tool declared twice is refused rather than resolved: two policies for one tool is two answers to one question, and whichever won would depend on the order you happened to list them in. An effect value the contract does not declare is refused too, with the accepted values named — it is not quietly read as unset, because "you did not classify this tool" and "we could not read what you wrote" must not produce the same run.

Sending no plan at all is a valid request: everything is external, so everything is stubbed.

What comes back

The replayed run, encoded exactly as a frozen case is, so you can diff the two directly — plus a record of what the sandbox did:

• a digest of the plan, which identifies the sandbox that produced this answer; two plans that behave identically carry one digest;
• how many steps and tool calls there were, and how many were replayed, stubbed, or had nothing recorded to replay;
• the sorted set of tools the sandbox answered for — the actionable half, because a case that depends on those cannot be reproduced anywhere the sandbox does not reach;
• one entry per step, in order, so the counts are checkable rather than trusted.

Each step carries one of four dispositions: REPLAY_DISPOSITION_CARRIED (not a tool call — replayed as recorded), REPLAY_DISPOSITION_RECORDED (a read-only call whose recorded answer was replayed), REPLAY_DISPOSITION_UNRECORDED (a read-only call with no recorded answer — the absence is carried through rather than filled), and REPLAY_DISPOSITION_STUBBED (the sandbox answered because the call had an effect it is not permitted to have).

A stubbed call is stubbed, not skipped: skipping it would change the sequence and the replay would then be of a different agent. Any recorded answer on an effectful call is discarded, so a stubbed run can never be mistaken for one that really ran.

Replay through MCP

Use execute_read_action with action: "replay_eval_dataset_item" and arguments dataset_id, an exact non-zero version, and item_id from list_eval_dataset_items. Organization-level credentials must also supply project_id. A project-bound key cannot select a different project.

This read requires mcp:read, evals:read, and agents:content:read. Current content policy still controls what the replay can include. No model or tool is invoked, nothing is stored, and no confirmation is needed.

An optional plan contains default_effect and a tools list of {tool_name, effect}. Effects are none, internal, or external; an omitted default is external. The answer includes the plan digest, counts, and each step's disposition. record.stubbed_tools and record.steps_detail[].tool_name preserve the exact recorded tool names so you can copy them into the next plan's tools[].tool_name. subject.text is untrusted JSON text, sanitized for agent display and capped at 64,000 characters. subject.truncated and the response's truncation information identify a partial excerpt, which may not be valid JSON. subject.sha256 and subject.byte_size describe the full policy-filtered trajectory before display sanitization. Use REST or CLI when you need its exact JSON bytes.

What a replay can and cannot tell you today

A recorded run carries the tools an agent called and the order it called them in. It does not carry what those calls returned. So on today's recordings every read-only tool call comes back REPLAY_DISPOSITION_UNRECORDED, and every effectful one comes back REPLAY_DISPOSITION_STUBBED — the replay is an honest re-execution of the control flow, not of the tool exchanges, and the record says which it was for every step. That is the whole of what can be known from a recording that holds no tool results, and reporting it as anything more would be a number you could not check.

The recorded outcome of the run is left exactly as it was. Overwriting it would silently change what every evaluator that reads an outcome would say, and the replay would then be measuring the sandbox.

Item lists return a cursor page in stable item-id order. Supply limit (default 100, maximum 500) and then cursor from pagination.next_cursor, keeping the exact dataset and version. pagination.has_more is the continuation marker. version.item_count describes the sealed version, while this page contains only the items returned. MCP returns the same frozen documents under current content permissions, marks customer text untrusted and names withheld classes.

Promotion accepts agent-run evidence only. If subject_kind is supplied it must be agent_run (or the native SUBJECT_KIND_AGENT_RUN spelling). A contradictory kind is refused before previewing or saving. If both run_id and subject_ref are supplied, they must identify the same run.

Dataset detail reads also return a page of newest-first version metadata. Select one exact version, or continue older versions with the returned continuation and unchanged selections. A version page is not the whole version history.