Evaluate an agent run

Start with one recorded run and a deterministic rule. This verifies your setup without calling a model or spending the paid-evaluation budget. Your agent can create rules and measurements; a directly signed-in person records human judgments. The examples below show the exact requests, so you can check what it sends.

Ask your agent

"Create a deterministic evaluator called task_completion that scores 1 when a run completed with no error steps. Score run <run-id> with it and show me the stored result."

Job	MCP tool or action	CLI command
Create an evaluator	create_evaluator (write action)	anectico evals evaluators create
Score one run now	evaluate_subject (write action)	anectico evals evaluate
Inspect job control and plan refusals	list_evaluation_jobs (read action)	anectico evals jobs
Read stored results	list_evaluation_results (read action)	anectico evals results
Select production runs for automatic checks	upsert_eval_sampling_rule (write action)	anectico evals rules set
Read the sampling rules	list_eval_sampling_rules (read action)	anectico evals rules list
Record your own judgment of a run	submit_annotation (write action)	anectico evals annotations add

The key needs evals:read for the reads and evals:write for the writes, plus mcp:read and mcp:write. A human judgment needs scores:read and scores:write in a directly signed-in person's session. Write actions show a preview first and apply with the returned confirm_token.

What your agent gets back

A stored result has a numeric value and its provenance: the subject, the exact evaluator version, when the run happened and when it was scored. The checks in 3. Verify the result say what to confirm.

Open the proof

A score belongs to a recorded agent run. The result of submit_annotation carries a link to that run's proof page. The page is read-only and shows the run in its project. The stored scores have no proof page of their own; read them with list_evaluation_results.

Before you start

• Record an agent run and copy its run (trace) ID.
• Use a project-scoped API key with evals:read and evals:write for that project.
• Set ANECTICO_API_KEY in your shell from your secret store. Do not paste it into an agent prompt.
• The examples use the managed API at https://app.anectico.com.

An evaluator is a versioned rule. A result records its score, exact version, and subject. A sampling rule optionally selects production runs for automatic evaluation.

1. Create an evaluator

This rule scores 1 when the run completed with no error steps, and 0 otherwise.

curl -X POST https://app.anectico.com/api/v1/evaluation/evaluators \
  -H "X-Anectico-API-Key: $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator_id": "task_completion",
    "display_name": "Task completion",
    "metric_name": "task_completion",
    "kind": "EVALUATOR_KIND_DETERMINISTIC",
    "config_json": "{\"aggregation\":\"all\",\"assertions\":[{\"field\":\"run.status\",\"op\":\"equals\",\"value\":\"AGENT_RUN_STATUS_COMPLETED\"},{\"field\":\"run.error_steps\",\"op\":\"lte\",\"number\":0}]}"
  }'

This first response is a preview. Inspect structuredContent.preview.details, then repeat the identical request with its structuredContent.preview.confirm_token as the confirm_token query argument. The confirmed receipt identifies evaluator version 1; read the evaluator back and use that exact version below. To change the rule later, create a new version; versions cannot be edited and references cannot use “latest”.

2. Score one run

Replace the example subject_ref with your recorded run ID.

curl -X POST https://app.anectico.com/api/v1/evaluation/evaluate \
  -H "X-Anectico-API-Key: $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator": {"evaluator_id": "task_completion", "version": 1},
    "subject_kind": "SUBJECT_KIND_AGENT_RUN",
    "subject_ref": "0af7651916cd43dd8448eb211c80319c"
  }'

This request returns a preview. Confirm its exact arguments with the server token to start the synchronous measurement. Repeating a completed subject and evaluator pair reuses its stored result and produces no second score. Read the result afterward with list_evaluation_results or the results endpoint.

A refusal or failed attempt leaves the subject available for an immediate explicit retry. Raise the exhausted budget or correct the refusal first. Content permission is checked before any charge or model call, including retries. Background attempts use a bounded retry delay; an explicit request can retry sooner, but cannot take a live evaluation from another caller.

A decision-score evaluator calls a model, which makes this call wait longer. Give this endpoint a six-minute client timeout. The CLI already applies a longer timeout to evaluation commands. See subjects and timeouts.

3. Verify the result

Read results for your run using GET /api/v1/evaluation/results, filtering by subject_ref and evaluator_id=task_completion. Confirm:

1. provenance.subject_ref is the run you intended to score.
2. provenance.evaluator names task_completion, version 1.
3. numeric_value matches that run's completed status and error-step count.
4. occurred_at identifies the run time; evaluated_at identifies the scoring time.

A missing result is not a zero. If the run has no measurable fields, fix its instrumentation before using evaluation scores to make release decisions. For result history and truncated batch reads, see Evaluation results.

4. Enable automatic checks, if needed

After the manual check succeeds, this rule samples 10% of eligible runs, starting at most 50 evaluations per cycle:

curl -X PUT https://app.anectico.com/api/v1/evaluation/sampling-rules \
  -H "X-Anectico-API-Key: $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator": {"evaluator_id": "task_completion", "version": 1},
    "subject_kind": "SUBJECT_KIND_AGENT_RUN",
    "rule_id": "00000000-0000-4000-8000-000000000010",
    "sampling_rate": 0.1,
    "per_tick_limit": 50,
    "enabled": true
  }'

Confirm this preview before it applies. Selection is stable for each run; repeatedly processing a run does not give it another chance to enter the sample. Nothing is evaluated automatically until you create an enabled rule. Setting enabled: false, sampling_rate: 0, or per_tick_limit: 0 disables it.

Choose the next evaluation task

You need to…	Read
Choose deterministic rules, trajectories, or decision scores	Evaluator configuration
Inspect score history and human judgments	Results and human judgments
Freeze reviewed cases or replay their recorded control flow	Datasets and sandbox replay
Compare recorded configurations on the same cases	Experiments
Make a release decision from existing scores	Release gates
Find an endpoint, permission, CLI command, or MCP action	API and permissions

A decision-score evaluator calls a model. It requires a configured paid-evaluation budget and a content policy that permits the selected evidence at judge_transfer. Its result is a score and the name of a level you declared; it carries no written explanation, and when the model cannot answer, nothing is scored. Human judgments require an attributed user session, not a machine API key. Review these requirements before switching from the deterministic example.

A run's recorded context assembly and tool-call evidence — which sources fed the model step, and which tool call each one traces back to — is part of the evidence a decision-score evaluator can draw on; see Record context assembly.

Scheduled work budgets

list_eval_sampling_rules and anectico evals rules list show budget_origin and budget. A sampling rule records its verified key, connected agent or person origin. Changing the evaluator, sample rate or per-tick work moves accounting to the updater; disabling it keeps the origin. Older definitions and missing credentials use scheduled work in the workspace.

An accepted sampled subject and evaluator job counts one write operation, even for a free evaluator. Reclaims and evaluation retries retain that charge. Exhausted limits show budget_exceeded; unavailable limits show limits_unavailable. The job waits for the next sampler slot and is not retried in a loop. These are operation limits; the separate evaluation spend limit still applies.

Experiments run when called. Release gates read existing results and do not run an evaluator. Their definitions show origin provenance; their calls follow interactive limits. See Scheduled work.

Human judgments require a directly signed-in person. A machine key is refused even when it names its owner, and an agent delegated through sign-in is also refused. This keeps the human labels used to calibrate an evaluator independent of that evaluator. Use anectico login and unset ANECTICO_API_KEY to record your own judgment.

Continue and confirm

All setup and scoring writes use the complete server preview, including the CLI counterparts. Review before, proposed, required scopes and effects, then repeat identical arguments with --confirm-token (MCP confirm_token). Keep one stable --idempotency-key on creates and appends. Read back the resulting definitions. A changed current state refuses the old confirmation. Lists, dataset items, version histories, experiment bindings and gate decisions are pages; follow pagination.next_cursor with cursor and unchanged filters. Result and release-series windows select subject occurrence using inclusive start and exclusive end, rather than scoring time.

Spending refusals and held count capacity

Paid quality measurements check permitted evidence before preparing spending and reserving daily count capacity. A spending refusal is never a score of zero and creates no evaluation result. budget_exhausted, spend_accounting_unavailable, spend_contract_unsupported and budget_contract_breach remain distinct errors. Changing a count limit cannot repair unavailable spending accounting or an unsupported provider charge bound.

The daily consumption read includes charged measurements and unresolved count holds. A lost response can leave capacity held while its outcome is checked. Capacity is released only when durable proof establishes that the entire attempt generation could never send; a timeout alone is insufficient. Release applies to the original admission day. A confirmed or uncertain sent attempt retains its count and possible spending. Repeating a stored verdict or recovering its delivery makes no new model call and consumes no extra count.

Sampled measurements use their resolved initiating credential when active, or workspace accounting when that origin is unavailable before admission. Admitted work keeps its original accounting subject through retries. Revoking a credential does not stop its schedule; stop the schedule separately. Historical attribution never grants permission to run. Deterministic evaluations take no paid count or provider-spending reservation. Positive guaranteed monetary limits remain unsupported until a maximum provider charge is qualified; an explicit zero can prevent sending.

Monthly sampled-work allowance

Each sampled subject and exact evaluator-version job uses one monthly automated-work unit, including deterministic evaluators. Interactive evaluations, experiments run on request and release-gate reads do not use this allowance. Retries of admitted work retain their first admission and month. Delivery recovery reuses the stored score without another unit or model call.

Read job control separately from scores:

anectico evals jobs --project <project-id> --rule <rule-id> \
  --from 2026-10-01T00:00:00Z --to 2026-10-17T00:00:00Z --limit 100 --json

Use MCP read action list_evaluation_jobs with the same project_id, optional rule_id, and required from and to. The interval is on job creation, at most 35 days. Continue with next_cursor and identical selectors, bounds and limit. Optional filters are opaque job or work ID, evaluator and version, status, and retry reason. The read needs evals:read, plus mcp:read for MCP. It contains no subject content and returns no proof link.

A refused job reports plan_limit_reached or plan_limits_unavailable, its plan_admission observations and exact next_attempt_at at the next sampler slot. It produces no score; an absent result is not a score of zero. Operation refusals remain budget_exceeded or limits_unavailable. Do not poll a refused slot in a loop or raise operation limits to evade a plan allowance.

An accepted admission's period, admission_used, admission_limit and plan revision describe its original admission. Optional current_period, current_used, current_limit and reset time are a separately checked observation; missing values mean unknown. An admission alone does not prove a completed evaluation. Use the job state and then read the stored verdict with evals results.

For fresh scheduled work, an inactive originating credential resolves to workspace accounting. Admitted spending attribution remains the original subject on retries; revocation does not stop a schedule. Disable or delete its sampling rule separately to stop future evaluation work. Previously stored verdicts can still finish delivery. Spending attribution never grants source access.

The jobs reader preserves financial refusal codes: budget_exhausted, spend_accounting_unavailable, spend_contract_unsupported, budget_contract_breach, attempt_consumed and paid_count_exhausted. Filter by the exact code to distinguish monthly plan admission, the daily evaluation count and the money budget.