Evaluation API and permissions

Start with your first evaluation for a runnable workflow.

Permissions

Scope	Grants
evals:read	List and read evaluators, sampling rules, the budget, results, the ordered observed-release series, datasets, experiments — including an experiment's comparison — and release gates with their full decision history. Also replay one frozen case in a sandbox: it saves no replay artifact, and it returns strictly less than reading the case does.
evals:write	Create evaluators and versions, manage sampling rules, set the budget, evaluate on demand, create datasets and promote reviewed cases or remove cases, create experiments and their trials, run an experiment, and create a release gate or evaluate one. Evaluating a gate is a write because it permanently records whether a release is admitted or held — a key minted only to look at gates must not be able to create it.
agents:content:read	Additionally see the explanation field on results, when the project's content policy also allows it.
scores:write	Record or replace your own human judgment of a run. It is a member capability rather than an administrative one, deliberately: rating a run is something anyone reviewing the product does.
scores:read	Read the human judgments recorded for runs.

A project-scoped key can only reach its own project; a request naming another project returns 404 Not Found, the same answer an evaluator that does not exist returns.

Endpoints

Method	Path
GET	/api/v1/evaluation/evaluators
POST	/api/v1/evaluation/evaluators
GET	/api/v1/evaluation/evaluators/{evaluatorId}
POST	/api/v1/evaluation/evaluators/{evaluatorId}/versions
GET	/api/v1/evaluation/evaluators/{evaluatorId}/versions/{version}
POST	/api/v1/evaluation/evaluators/{evaluatorId}/versions/{version}/calibrations
GET	/api/v1/evaluation/evaluators/{evaluatorId}/calibrations
GET	/api/v1/evaluation/sampling-rules
PUT	/api/v1/evaluation/sampling-rules
DELETE	/api/v1/evaluation/sampling-rules/{ruleId}
GET	/api/v1/evaluation/budget
PUT	/api/v1/evaluation/budget
POST	/api/v1/evaluation/evaluate
GET	/api/v1/evaluation/results
GET	/api/v1/evaluation/release-series
POST	/api/v1/agent-runs/{runId}/annotations
GET	/api/v1/evaluation/annotations
GET	/api/v1/evaluation/datasets
POST	/api/v1/evaluation/datasets
GET	/api/v1/evaluation/datasets/{datasetId}
GET	/api/v1/evaluation/datasets/{datasetId}/versions/{version}/items
POST	/api/v1/evaluation/datasets/{datasetId}/promotions
POST	/api/v1/evaluation/datasets/{datasetId}/removals
POST	/api/v1/evaluation/datasets/{datasetId}/versions/{version}/items/{itemId}/replays
GET	/api/v1/evaluation/experiments
POST	/api/v1/evaluation/experiments
GET	/api/v1/evaluation/experiments/{experimentId}
POST	/api/v1/evaluation/experiments/{experimentId}/trials
POST	/api/v1/evaluation/experiments/{experimentId}/run
GET	/api/v1/evaluation/experiments/{experimentId}/comparison
GET	/api/v1/evaluation/gates
POST	/api/v1/evaluation/gates
GET	/api/v1/evaluation/gates/{gateId}
POST	/api/v1/evaluation/gates/{gateId}/evaluations

Promoting cases and removing them are both POSTs that seal a new version, rather than a PUT or a DELETE on an existing one — nothing about a sealed version ever changes. Evaluating a gate is a POST onto /evaluations for the same reason: a new explicit idempotency key appends a decision rather than editing one. Its optional JSON body is { "idempotency_key": "your-stable-decision-key" }; an empty body selects the stable default decision intent.

Replaying a case is a POST for a different reason: it carries the sandbox plan, not because anything is written. A replay saves no replay artifact and reproduces its answer every time, which is why it needs only evals:read.

GET /api/v1/evaluation/gates accepts an optional ?release= to narrow to one release label.

All accept an optional ?project_id= query parameter; a project-scoped key is pinned to its own project regardless. /budget is the exception: it is org-wide and reads no project.

From the command line, and from an agent

The API is the complete surface. The CLI mirrors it; MCP deliberately exposes a narrower subset.

The CLI mirrors the endpoints one-for-one under anectico evals:

anectico evals datasets list
anectico evals datasets items refunds --version 3
anectico evals experiments comparison exp-refunds
anectico evals release-series --evaluator task_completion --evaluator-version 1 --metric task_completion
anectico evals gates list --release 2026.08.3
anectico evals gates evaluate gate-1 --idempotency-key release-2026-08-26
anectico evals datasets replay refunds case-17 --version 3 --file plan.json
anectico evals annotations add $RUN_ID --metric helpfulness --label good --comment "answered the question"
anectico evals annotations list $RUN_A $RUN_B $RUN_C

anectico evals annotations add omits the number unless you pass --value, so a --label-only judgment is recorded as "not scored" rather than as a zero. list names every run you give it in one request.

Structured declarations — an experiment, a gate, a promotion — are supplied as JSON with --body or --file (--file - reads standard input), and are sent to the API exactly as you wrote them. Reads print a table on a terminal and JSON when piped; --output json forces it. anectico evals also covers evaluators, rules, budget, evaluate and results. Run anectico evals --help, or see the CLI reference.

An MCP agent can run the whole evaluation loop without the REST API or the CLI. Every action is reached through list_read_actions / execute_read_action or list_write_actions / execute_internal_action, so none adds a host tool. A key bound to one project always acts in that project; an organization-level key names the project with project_id on every call.

Reads, under evals:read unless noted:

Action	Returns
list_evaluators, get_evaluator	Evaluators, their kind, metric, cost class and immutable versions
list_eval_datasets, get_eval_dataset	One continued page of datasets or newest-first sealed versions; pass version for one exact version, or before_version for older history. Continue with next_cursor as cursor and unchanged selections
list_eval_dataset_items	One continued page of an exact version's cases, source runs, human-review provenance and frozen documents under current content permissions and policy. Document inspection needs agents:content:read; withheld content is named
list_eval_experiments, get_eval_experiment, compare_eval_experiment	Declared experiments and the paired comparison against the baseline
list_evaluation_results	Stored results — value, label and provenance; the explanation is not returned
list_evaluation_release_series	One exact metric by observed release
list_eval_annotations (scores:read)	Human judgments of named runs
list_evaluator_calibrations	A judge's agreement with human labels over time
list_eval_sampling_rules, get_evaluation_budget	Production sampling rules and the daily paid-evaluation cap
list_release_gates, get_release_gate	Gates and every decision they reached

Writes, under evals:write:

Action	What a retry does
create_evaluator, create_eval_dataset, create_eval_experiment	Refused: you choose the id, and it is taken
create_evaluator_version	Appends another version — read get_evaluator before retrying
promote_to_dataset, exclude_dataset_items	Refused: the same change would seal a version identical to its parent
set_experiment_trials	Same end state; previews first because re-binding replaces a run
run_eval_experiment, evaluate_subject	Finished work is returned, never paid for twice; both preview first because they spend
calibrate_evaluator	Returns the stored record while the labels are unchanged. The same idempotency_key always returns its original measurement; a new key, or none, measures the labels as they are now
upsert_eval_sampling_rule, delete_eval_sampling_rule, set_evaluation_budget	Same end state; each previews first
create_release_gate	Same key and arguments return the original creation result with a pending verdict, even after later decisions; changed arguments are refused
evaluate_release_gate	Returns the original decision for the same idempotency_key

A promotion must cite, for each run, the human judgment that reviewed it (review_annotation_id from list_eval_annotations); unreviewed runs are refused. run_eval_experiment does a bounded amount of work per call and reports what remains — repeat it until it reports complete=true before quoting compare_eval_experiment.

Gates and decisions are permanent and have no delete, void, or correction path. create_release_gate requires a caller-owned gate_id. evaluate_release_gate requires an idempotency_key over MCP; the API and CLI also accept omission and use the server-derived default intent.

submit_annotation, under scores:write, is also reached through list_write_actions and execute_internal_action. It previews first and applies on a second call with the returned confirm_token, because re-judging replaces the caller's previous judgment of that run. Its receipt reports the metric, the number and the label; it does not repeat the rationale back.

One thing an agent cannot do over MCP, deliberately: read a case's frozen documents — the recorded subject, expectation and metadata. They can carry recorded content, so they are available over the API and the CLI only; MCP returns each case's id and origin, never its contents. The sandboxed replay of a case is API and CLI only for the same reason.

Evaluation jobs

GET /api/v1/evaluation/jobs is a metadata-only job-control read under evals:read. MCP uses list_evaluation_jobs through the read gateway; CLI uses anectico evals jobs. Supply an exact project and explicit from and to UTC bounds, at most 35 days apart. Use rule_id, job_id, work_id, evaluator_id, version, status and retry_reason to narrow it. limit is 1 through 200, default 100; cursor is at most 512 bytes and binds all selectors, the limit and the fixed interval. Read next_cursor and returned_count from the response. See Job control and refusals for states, retry times and admission snapshots. This read returns no proof link and no verdict.