Evaluation datasets and sandbox replay

Start with your first evaluation for a runnable workflow. See API and permissions for endpoint and credential requirements.

Datasets — a frozen set of cases

A dataset is a named collection of cases you promoted out of real recorded runs. Its metadata is editable; its versions are not. Promoting cases seals a new version containing everything the current version had plus the new cases, and removing cases seals a new version without them — so a score that cites a version keeps meaning exactly what it meant when it was recorded.

Before using the example, create a dataset named regression_set (or replace that path segment with your existing dataset ID). Set RUN_ID to a recorded run in the same project. ANECTICO_USER_TOKEN must be an authenticated user-session access token with scores:write and evals:write, not an API key or refresh token. Prefer the signed-in CLI if you do not manage OAuth tokens in an integration; see human judgments. The example also requires jq.

Only a human-reviewed production run can be promoted. Record a judgment first, then pass the returned annotation id with that exact run:

REVIEW_ID=$(curl -sS -X POST \
  "https://app.anectico.com/api/v1/agent-runs/$RUN_ID/annotations" \
  -H "Authorization: Bearer $ANECTICO_USER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"metric_name":"dataset_eligibility","label":"reviewed"}' |
  jq -r '.annotation.annotation_id')

curl -X POST \
  "https://app.anectico.com/api/v1/evaluation/datasets/regression_set/promotions" \
  -H "Authorization: Bearer $ANECTICO_USER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"subjects":[{"subject_kind":"SUBJECT_KIND_AGENT_RUN","subject_ref":"'"$RUN_ID"'","review_annotation_id":"'"$REVIEW_ID"'"}]}'

The review must belong to the same project and name the same run. The frozen item records the review id, authenticated reviewer, and review time. Promotion refuses a missing or mismatched review before it freezes or stores anything; holding evals:write does not itself count as a review.

Each case's id is a digest of its own content, which has two consequences worth knowing:

• Promoting the same run twice gives you one case, not two.
• A case keeps the same id as it is carried into later versions, so "this case scored 0.4 in v1 and 0.9 in v3" is a question you can ask.

A version number is always exact. Reading a version's cases requires you to name it; there is no "latest", because a set of cases that can change under a comparison is not a set of cases.

Nothing is ever deleted. A version cited by a stored score cannot be removed, and neither can a dataset that has one.

What a case contains. A frozen, self-contained projection of the run — enough to score it again later without the original run still existing. If your project's content settings did not permit a class of content to be copied, the case records which classes were withheld rather than being quietly short: a case frozen without its content would score differently for ever, and you would have no way to tell it from one that never had any.

Replaying one case in a sandbox

A replay re-executes the recorded decision sequence of one frozen case inside a sandbox that cannot reach anything outside this platform. It stores nothing. A replay is a pure function of a frozen case and the plan you send, and both are fixed — so asking again reproduces the same answer byte for byte, and you never have to keep one in order to get it back.

The plan is yours to declare

Nothing here knows what your tools do, so the plan is where you say it. Each policy names a tool by the name the recording carries and classifies it by what the call does outside the sandbox:

Effect	Meaning	What the sandbox does
SANDBOX_EFFECT_NONE	a read-only call	replays the recorded answer, if there is one
SANDBOX_EFFECT_INTERNAL	writes inside this platform	answers with a marked stub
SANDBOX_EFFECT_EXTERNAL	leaves this platform	answers with a marked stub

default_effect covers any tool no policy names. Unset means external, and so does a call the recording does not name — not even an explicitly open default reaches that one. The recoverable mistake is stubbing a read; the unrecoverable one is letting a write through because nobody classified it.

A tool declared twice is refused rather than resolved: two policies for one tool is two answers to one question, and whichever won would depend on the order you happened to list them in. An effect value the contract does not declare is refused too, with the accepted values named — it is not quietly read as unset, because "you did not classify this tool" and "we could not read what you wrote" must not produce the same run.

Sending no plan at all is a valid request: everything is external, so everything is stubbed.

What comes back

The replayed run, encoded exactly as a frozen case is, so you can diff the two directly — plus a record of what the sandbox did:

• a digest of the plan, which identifies the sandbox that produced this answer; two plans that behave identically carry one digest;
• how many steps and tool calls there were, and how many were replayed, stubbed, or had nothing recorded to replay;
• the sorted set of tools the sandbox answered for — the actionable half, because a case that depends on those cannot be reproduced anywhere the sandbox does not reach;
• one entry per step, in order, so the counts are checkable rather than trusted.

Each step carries one of four dispositions: carried (not a tool call — replayed as recorded), recorded (a read-only call whose recorded answer was replayed), unrecorded (a read-only call with no recorded answer — the absence is carried through rather than filled), and stubbed (the sandbox answered because the call had an effect it is not permitted to have).

A stubbed call is stubbed, not skipped: skipping it would change the sequence and the replay would then be of a different agent. Any recorded answer on an effectful call is discarded, so a stubbed run can never be mistaken for one that really ran.

What a replay can and cannot tell you today

A recorded run carries the tools an agent called and the order it called them in. It does not carry what those calls returned. So on today's recordings every read-only tool call comes back unrecorded, and every effectful one comes back stubbed — the replay is an honest re-execution of the control flow, not of the tool exchanges, and the record says which it was for every step. That is the whole of what can be known from a recording that holds no tool results, and reporting it as anything more would be a number you could not check.

The recorded outcome of the run is left exactly as it was. Overwriting it would silently change what every evaluator that reads an outcome would say, and the replay would then be measuring the sandbox.