# Quality evaluations

> Score your agent runs automatically, with a record of exactly how each number was produced.

Canonical page: https://anectico.com/docs/agents/quality-evaluations/


An agent run either did its job or it did not, and error rates do not tell you which. Evaluations
attach a **score** to a run — task completion, whether the expected steps happened in the expected
order, anything you can state as a rule — and record enough about how that score was produced that
you can reproduce it months later.

Evaluations run automatically against a sample of your production traffic, or on demand for one run
you name.

## The three pieces

| Piece | What it is |
|---|---|
| **Evaluator** | A named, versioned rule that turns a run into a number and a label. |
| **Sampling rule** | Which runs an evaluator sees, at what rate, with what per-tick ceiling. |
| **Result** | One score, with the full record of what produced it. |

## Evaluator versions are exact, and never float

An evaluator has an id and a list of **versions**. A version is immutable: changing a rule creates a
new version and leaves the old one exactly as it was.

Anywhere you reference an evaluator, you name the version — and the version must be a real number,
never "the latest". This is deliberate and it is not a formality. If a rule could change under a
reference, the same release would pass one day and fail the next with nothing in the record saying
why, and any comparison between two releases would silently be a comparison between two different
questions.

The reference is checked when you **write** the thing that cites it, so a typo or a missing version
is an error on the request you just made rather than a background job quietly doing nothing.

## Creating an evaluator

```bash
curl -X POST https://app.anectico.com/api/v1/evaluation/evaluators \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator_id": "task_completion",
    "display_name": "Task completion",
    "metric_name": "task_completion",
    "kind": "EVALUATOR_KIND_DETERMINISTIC",
    "config_json": "{\"aggregation\":\"all\",\"assertions\":[{\"field\":\"run.status\",\"op\":\"equals\",\"value\":\"AGENT_RUN_STATUS_COMPLETED\"},{\"field\":\"run.error_steps\",\"op\":\"lte\",\"number\":0}]}"
  }'
```

The response carries the evaluator and its version 1. Append a new version with
`POST /api/v1/evaluation/evaluators/{evaluatorId}/versions`; there is no update or patch, by design.

### Evaluator kinds

**`EVALUATOR_KIND_DETERMINISTIC`** — declarative assertions over a run's observable fields.

```json
{
  "aggregation": "all",
  "assertions": [
    {"field": "run.status",       "op": "equals",   "value": "AGENT_RUN_STATUS_COMPLETED"},
    {"field": "run.error_steps",  "op": "lte",      "number": 0},
    {"field": "steps.tool_names", "op": "contains", "value": "search_docs"}
  ]
}
```

`aggregation` is `all` (a gate: every assertion must hold, the score is 1 or 0) or `fraction` (the
share of assertions that held).

| Field | Type | Operators |
|---|---|---|
| `run.status` | one value | `equals`, `not_equals` |
| `run.error_steps`, `run.step_count`, `run.tool_calls`, `run.model_calls`, `run.duration_ms` | number | `lte`, `gte` |
| `steps.outcomes`, `steps.tool_names`, `steps.canonical_operations` | set | `contains`, `not_contains` |
| `session.turn_count`, `session.run_count`, `session.duration_ms`, `session.cost_nanos` | number | `lte`, `gte` |

The `session.*` fields describe a whole conversation and are answerable **only** for
`SUBJECT_KIND_AGENT_SESSION` — see [What you can name as the subject](#what-you-can-name-as-the-subject).

A deterministic evaluator can instead return one native measurement. A measurement config contains
`measurement` and no `assertions` or `aggregation`:

```json
{"measurement":"run.cost_nanos"}
```

| Measurement | Unit | When it is measurable |
|---|---|---|
| `run.duration_ms` | milliseconds | The recorded run reports a duration. |
| `run.cost_nanos` | billionths of a US dollar | The run has client-reported or calculated model cost. A missing or unpriced cost is not zero and produces no score. |
| `safety.error_steps` | count | The run contains explicitly identified safety-check steps with reported outcomes. No safety check, or an unknown outcome, is not a safe zero and produces no score. |
| `session.turn_count` | count | The session's turns carry turn ids. A session with none is not a session of nought turns; it produces no score. |
| `session.run_count` | count | Always measurable for a session that exists. |
| `session.duration_ms` | milliseconds | The time between the session's first and last recorded activity. A session with one moment of activity measures `0`, which is a real answer. |
| `session.cost_nanos` | billionths of a US dollar | The session's runs reported model cost. Nothing on a session says whether an unpriced conversation was free, so a zero is treated as unmeasured and produces no score. |

Use the assertion form for **reliability** (for example, completed status and zero error steps), a
reference-trajectory or judge evaluator for **quality**, and the native measurements for **safety,
latency, and cost**. Experiments and release gates treat each as the evaluator's real numeric result;
they do not fill missing dimensions with defaults.

**`EVALUATOR_KIND_REFERENCE_TRAJECTORY`** — did the agent do the right things **in the right order**.

```json
{
  "step_key": "canonical_operation",
  "match": "ordered_subsequence",
  "reference": [
    "CANONICAL_OPERATION_INVOKE_AGENT",
    "CANONICAL_OPERATION_RETRIEVAL",
    "CANONICAL_OPERATION_CHAT"
  ]
}
```

A run that retrieved *after* answering used the same steps as one that retrieved first, and a
set-membership rule scores both identically. This one does not. `ordered_subsequence` (the default)
scores the longest in-order match against the reference length, so an extra step between two expected
ones is not a deviation; `exact` requires the sequence to match exactly. `step_key` may be
`canonical_operation` or `tool_name`.

**`EVALUATOR_KIND_LLM_JUDGE`** — a model scores the run against a rubric you write.

The two kinds above answer questions you can state as a rule. Some questions cannot be: whether an
answer was *grounded* in what the agent retrieved, whether it actually addressed what was asked. A
judge answers those by reading evidence from the run and scoring it against criteria you declare.

```json
{
  "instructions": "Be strict. Prefer a low score when you are unsure.",
  "criteria": [
    {"name": "grounded",  "description": "every claim in the answer traces to the retrieved context", "weight": 2},
    {"name": "addressed", "description": "the answer resolves the question that was asked"}
  ],
  "evidence": ["run_summary", "model_transcript"],
  "max_evidence_chars": 12000
}
```

| Field | | |
|---|---|---|
| `criteria` | required, 1–8 | Each needs a `name` and a `description`. `weight` defaults to 1. |
| `evidence` | required | What the judge is shown. See below. |
| `instructions` | optional | A preamble, rendered ahead of the criteria. |
| `model` | optional | A specific model id. Omit to use the default. |
| `max_evidence_chars` | optional, 500–40000 | Evidence budget. Defaults to 12000. |

A rubric is a **structure, not a prompt**. That is what makes two versions of "faithfulness"
comparable: the difference between version 4 and version 5 is a criterion added, removed or reworded,
rather than a change somewhere inside a paragraph. An unrecognised field is rejected rather than
ignored, so a misspelled key is an error on the request instead of a rule that silently does nothing.

The result shape is fixed — one score, one label, one sentence of reasoning — and is not something a
rubric sets.

#### What a judge is shown

`evidence` is a declaration, and it is the one that decides what leaves your project:

| Selector | What the judge sees | Content |
|---|---|---|
| `run_summary` | Status, step counts, duration. | — |
| `step_sequence` | The ordered operations, outcomes and tool **names**. | — |
| `model_transcript` | Recorded prompt and response bodies. | `model_transcript` |
| `tool_calls` | Recorded tool and MCP call arguments and results. | `tool_arguments` |

The first two carry no message bodies at all — a tool *name* is a symbol, not the arguments it was
called with — so a rubric built only from them shows a model nothing you recorded.

#### Your content policy decides whether the evidence may be sent

Sending recorded content to an evaluation model is a transfer, and it is governed by your project's
[content policy](/docs/manage/content-policy) at the `judge_transfer` boundary. The policy is
consulted **before** anything is sent, never after.

Two properties are worth stating exactly, because both are easy to get wrong and neither is visible
from the outside:

- **The question asked of your policy is derived from what the evidence actually contains**, not from
  what the rubric might want. A rubric that selects `tool_calls`, running against a run that recorded
  no tool arguments, asks nothing about `tool_arguments` — so a project that permits transcripts and
  refuses tool arguments is evaluated normally on those runs.
- **But a kind of content your policy actually withheld is never quietly dropped.** When the content
  your rubric asked for was removed by the policy, the transfer is put to the policy again for
  exactly that kind, and a refusal is a refusal for the whole evaluation — `403 Forbidden` — even
  when the rubric's other selectors still had something to show. A judge scoring "was the answer
  grounded in the transcript" over a run summary alone is scoring something you did not ask about.
  If the removal was not your policy's decision (a narrower credential, for instance), the
  evaluation proceeds and the model is **told** which kinds of content were withheld, by name, so it
  cannot read an absence as evidence that nothing was there.
- **A refusal is terminal and it is recorded.** The run is not sent, is never scored 0, and is not
  retried in a loop. It is recorded as withheld, naming the kind of content, the boundary, and which
  of three things decided: your policy said so, the project has no policy yet, or the policy could
  not be read. "Nobody judged this" and "nobody sampled this" stay different facts.

The **read** and `judge_transfer` cells are independent for automatic quality checks. A run may be
hidden from product reads and still evaluated when `judge_transfer` allows its content; denying
`judge_transfer` still prevents the transfer whatever the read cell says.

## Why some rules are refused

Evaluator configuration is validated when you write it, and some rules are refused even though they
are well-formed. Each refusal names its reason. The reason is always the same one:

> **A field that nothing fills is not a field you can measure.**

A run's operation, outcome and similar values carry a placeholder when your instrumentation did not
report them. A rule written against that placeholder does not measure your agent — it measures how
completely you are instrumented, and reports the answer as a quality score. Worse, a trajectory
written against it scores *higher* the less instrumented the run is, inverting the metric entirely.

So you cannot assert on `OUTCOME_UNKNOWN`, on `CANONICAL_OPERATION_UNKNOWN`, or on fields no
instrumentation populates. If you need a value that is refused, the fix is to emit it — see
[Instrumenting LLM and agent calls](/docs/instrument/llm-calls).

The same principle applies at run time. If a run carries nothing the evaluator can measure, it is
recorded as **unmeasured** and retried later, never scored zero. A zero would say your agent failed;
the truth is that nobody measured it. A judge is held to the same rule twice over: a run carrying
none of the evidence its rubric selected is not sent to a model at all, and a model that comes back
with something unusable is retried rather than recorded as a failing score.

## Sampling production traffic

```bash
curl -X PUT https://app.anectico.com/api/v1/evaluation/sampling-rules \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator": {"evaluator_id": "task_completion", "version": 1},
    "subject_kind": "SUBJECT_KIND_AGENT_RUN",
    "sampling_rate": 0.1,
    "per_tick_limit": 50,
    "enabled": true
  }'
```

- **`sampling_rate`** (0–1) is a **true cost cap**. Selection is derived from a stable hash of the
  run, so the same run gets the same answer every time and the unselected share is never evaluated at
  all. It is not a per-cycle coin flip, which would eventually select everything.
- **`per_tick_limit`** caps how many evaluations start per cycle. Zero disables the rule, as does
  `enabled: false` or a rate of 0.

A rule is evaluated only where you create one. Nothing is evaluated by default.

## Evaluating one run on demand

```bash
curl -X POST https://app.anectico.com/api/v1/evaluation/evaluate \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator": {"evaluator_id": "task_completion", "version": 1},
    "subject_kind": "SUBJECT_KIND_AGENT_RUN",
    "subject_ref": "0af7651916cd43dd8448eb211c80319c"
  }'
```

This is the same engine, with sampling skipped — naming a run *is* selecting it. Asking twice for the
same run and evaluator version returns the stored answer with `already_evaluated: true`; it does not
re-run and does not spend again.

### Set your client's timeout above 30 seconds

The call is **synchronous**: it reads the subject, and for an LLM-judge evaluator it waits for the
model. A judge call routinely takes 10–30 seconds and is allowed up to five minutes. Clients whose
default request timeout is 30 seconds — which is most of them — will hang up on a judge that is still
working, and the evaluation is then abandoned rather than stored.

Give this one endpoint its own timeout of **six minutes** — a little above the five the server
allows, so that what you read is the server's answer rather than your own client giving up — or use
a deterministic evaluator (which answers in under a second) when you need a fast answer.

If the five minutes do run out, you get **`504 Gateway Timeout`** with a message, not a dropped
connection. The `anectico` CLI already applies its own longer timeout to `anectico evals evaluate`
and `anectico evals experiments run`; every other command keeps the short default.

### What you can name as the subject

`subject_kind` says what `subject_ref` is, and it is also the label the stored result carries. Three
kinds can be evaluated today:

| `subject_kind` | `subject_ref` is | Notes |
|---|---|---|
| `SUBJECT_KIND_AGENT_RUN` | the run (trace) id | The whole run. This is what sampling rules select. |
| `SUBJECT_KIND_AGENT_STEP` | `"<run_id>/<span_id>"` | One step of a run — the `span_id` is the one the run graph reports for that step. The score is about that step: the evaluator sees only it, not its siblings. |
| `SUBJECT_KIND_AGENT_SESSION` | the session key | The whole conversation: every run that shares that session key. Only the `session.*` fields can score it, and only a rubric that asks for **no recorded content** can be run over it — see below. |

`SUBJECT_KIND_AGENT_TURN` is part of the result vocabulary — a stored result can carry it — but
cannot be submitted here: a turn cannot be looked up from its id alone. `SUBJECT_KIND_DATASET_ITEM`
is a batch subject and is evaluated by running an experiment over a dataset version, not through this
endpoint. Naming either returns **`501 Not Implemented`**. A result is never labelled with a subject
that was not read: an unsupported kind is refused rather than scored from the run behind it.

Which fields an evaluator can assert on depends on what you named, and each family is answerable for
its own subject only:

- **`run.*`** describe the whole run and are answerable for `SUBJECT_KIND_AGENT_RUN` and for dataset
  items. Asking one about a **step** or a **session** returns `400 Bad Request` ("carries nothing
  this evaluator can measure") rather than reporting the run's number under another subject's name.
- **`steps.*`** and `safety.error_steps` are answerable for a run and for a step, and for a step
  subject they describe that step alone.
- **`session.*`** describe a whole conversation and are answerable **only** for
  `SUBJECT_KIND_AGENT_SESSION`. Asking one about a single **run** of that session returns
  `400 Bad Request` for the same reason in the other direction — a run's own duration or cost is not
  the conversation's.

### What a judge reads for each subject

A judge that asked for `model_transcript` or `tool_calls` has to be shown recorded content, and what
that content **is** depends on the subject:

- **A run** — that run's own recorded content.
- **A step** — the content of the run the step belongs to, with the step itself as the subject the
  score is filed under.
- **A session** — nothing. A conversation is a set of runs and has no transcript of its own, and
  there is no answer to "show the judge the session" that would not mean choosing runs on your
  behalf. Rather than pick one, a content-asking rubric over a session is refused with
  `400 Bad Request` naming the kind. A rubric built only from `run_summary` and `step_sequence` is
  not refused for content — it reads none — but a session rollup carries neither a run summary nor a
  step sequence, so it has nothing to show a judge either and answers "carries nothing this
  evaluator can measure". Score a conversation with the `session.*` fields, which read the rollup
  itself.

| Response | Meaning |
|---|---|
| `404 Not Found` | Nothing here matches what you named — the run, the step, or the evaluator version. Look the identifier up again; retrying it unchanged will not help. A run that was just produced may not be readable for a few seconds, and the same call succeeds once it is. |
| `504 Gateway Timeout` | Either the subject could not be read within its own budget, or the whole evaluation ran past the five minutes the call is allowed. **This is not a `404`**: it says the subject could not be *read*, never that it does not exist, so retry the identifier you already have rather than looking for a new one. No result is stored. A judge that had already started may still have been charged, and the retry is free once a verdict exists — asking again for the same run and evaluator version returns the stored answer. |
| `501 Not Implemented` | This `subject_kind` cannot be evaluated on demand. See the table above; retrying will not help. |
| `400 Bad Request` | The `subject_ref` cannot name a subject of that kind — a step reference must be `"<run_id>/<span_id>"`. |
| `409 Conflict` | Already being evaluated. Retry shortly. |
| `429 Too Many Requests` | The daily evaluation budget is spent. |
| `400 Bad Request` | The run carries nothing this evaluator can measure. |
| `403 Forbidden` | Your content policy withholds content your rubric asked for from a judge. The message names the kind of content and the boundary, never the content itself. |
| `400 Bad Request` | A judge whose rubric asks for recorded content cannot be run over this `subject_kind`. Today that is a **session**: a conversation has no single transcript to show, and the answer names the kind rather than reading one of its runs. Score the session with `session.*` fields, or judge its runs individually. |
| `400 Bad Request` | Judging is not configured or its credentials were permanently refused. An operator must repair the judge setup before retrying. |
| `503 Service Unavailable` | Judging is temporarily unavailable because of a timeout, throttle, transport failure, or upstream outage. Retry is appropriate. |

## The spend budget

Evaluators that call a model cost money, so the daily ceiling is a **hard** limit rather than a
throttle: it is enforced transactionally, and it does not open up when anything is degraded.

```bash
curl -X PUT https://app.anectico.com/api/v1/evaluation/budget \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"daily_paid_evaluation_cap": 500}'
```

`GET /api/v1/evaluation/budget` returns the cap and today's consumption.

Two things to know:

- **An organization with no configured budget runs no paid evaluations.** The safe state is the
  default; you opt in to spending. A judge in an organization with no budget never runs.
- **Evaluators that do not call a model never consume budget.** `EVALUATOR_KIND_DETERMINISTIC` and
  `EVALUATOR_KIND_REFERENCE_TRAJECTORY` are in that category, so they keep working after a paid
  budget is exhausted. `EVALUATOR_KIND_LLM_JUDGE` is not: it is charged one unit per run evaluated,
  once — a retried delivery of a score that was already produced never charges again.

The budget is charged **after** your content policy permits the transfer, so a run your policy
withholds costs nothing.

## Reading results

```bash
curl "https://app.anectico.com/api/v1/evaluation/results?metric_name=task_completion&start=2026-08-01T00:00:00Z&limit=100" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"
```

Filters: `metric_name`, `evaluator_id`, `subject_ref`, `subject_refs`, `mode`, `start`, `end`,
`limit`. Omitting `mode` returns every mode; each result states its own, so comparing across them is
something you do deliberately rather than by accident.

`subject_refs` asks about **many** runs at once — repeat it (`?subject_refs=a&subject_refs=b`) or
pass a comma list (`?subject_refs=a,b`); both work, and it combines with the singular `subject_ref`
as one set. Use it to draw a quality badge for a whole page of runs in one request instead of one
request per row. At most **200** runs may be named, counting the two parameters together and
counting a run named twice once; a longer list is **refused** rather than shortened, because a
silently trimmed list comes back looking complete while whole runs are missing from it and you would
then draw "not evaluated" for a run that was.

`limit` means different things on the two shapes, and the difference exists for that same reason:

- Naming **at most one run**, it is the page size — default 100, maximum 500.
- Naming **several**, it bounds *each* run's share of the page — default and maximum 20 — and the
  page is that share times the runs you named. A single total over many runs would be spent by
  whichever runs were evaluated most recently, and the runs at the far end would come back empty
  behind a `200 OK`.

The response also carries **`truncated`** and **`truncated_subject_refs`**. `truncated` is `true`
when a cap cut the page and there are more results matching your filters; it is exact rather than a
guess from a full-looking page, so a page of exactly the cap still tells you truthfully whether
anything was left behind. `truncated_subject_refs` names *which* runs were cut — one busy run sets
the page-level flag for all 200, so if you are drawing a badge per run, treat exactly the runs listed
there as *unknown* rather than as *not evaluated*, and read those again with a narrower window or a
tighter `metric_name`. A run you asked about that is **not** in that list returned everything stored
for it.

**This read is a history, not a current value.** Every evaluator version a run was measured under
keeps its own result, on purpose: that is what makes "this run scored 0.2 under version 3 and 0.9
under version 4" answerable at all, and it is what a release gate compares. So a badge or an average
built by summing everything this endpoint returns counts a run's *old* verdicts alongside its
current ones — a run that was fixed reads as half-fixed for ever. When you want the current
picture, keep the highest `provenance.evaluator.version` for each
(`provenance.evaluator.evaluator_id`, `metric_name`) pair and summarize those.

Every result carries:

| Field | |
|---|---|
| `numeric_value`, `label` | The score. |
| `provenance.evaluator` | The evaluator id and the **exact** version. |
| `provenance.mode` | How it was produced. |
| `provenance.subject_kind`, `provenance.subject_ref` | What was scored. |
| `subject_release`, `release_declaration` | The release observed on that subject, or the explicit answer that it declared none. |
| `occurred_at` | When the **run** happened. |
| `evaluated_at` | When the score was produced. |
| `explanation` | The evaluator's reasoning. |
| `provenance.execution` | What the evaluation actually did — see below. |

`occurred_at` and `evaluated_at` are separate on purpose. They differ by however long sampling and
queueing took, and a trend plotted on the wrong one attributes a regression to the day it was
measured rather than the day it shipped.

### What the evaluation actually did

`provenance.execution` records how each score came to exist: why the subject was picked, what the
evaluation cost, which model answered, and whether the job finished. It is stored on the result
rather than looked up later, so it stays true even after the rule that selected the subject is
edited or the evaluator is retired.

| Field | |
|---|---|
| `sampling` | `SAMPLED` (an online rule picked this subject), `NOT_SAMPLED` (you asked for this subject by name) or `WHOLE_DATASET` (a batch run measured every case). |
| `sampling_rule_id`, `sampling_rate` | The rule that fired and the rate it was applying at that moment. Present **only** for `SAMPLED`. |
| `cost_state` | `CHARGED`, `FREE`, `UNPRICED` or `UNREPORTED`. |
| `charged_cost_nano_usd` | The charge, in billionths of a US dollar. Meaningful **only** when `cost_state` is `CHARGED`. |
| `input_tokens`, `output_tokens` | What the model call consumed. Zero for an evaluator that calls no model. |
| `judge_model_state` | `RECORDED`, `NOT_APPLICABLE` (not a model judge) or `UNREPORTED`. |
| `judge_model` | The model that **actually answered**. Present only for `RECORDED`. |
| `job_status` | `COMPLETED`, `PARTIAL`, `FAILED` or `CANCELLED`. |
| `job_failure_class` | Names what was missing, for every status except `COMPLETED`. |

**Read `cost_state` before the charge.** A charge of zero has two unrelated meanings — an evaluator
that calls no model and costs nothing, and a model whose price is unknown — and the number alone
cannot tell you which. `UNPRICED` is the second: tokens were spent, the price is not known, and the
zero is an absence rather than a bill. It is never reported as a charge of zero.

**`judge_model` is not the model the evaluator was configured with.** The evaluator version records
what was requested; this records what answered, and the two differ whenever a provider resolves a
floating model name to a dated one or falls back to another model. A score attributed to the
requested model would be attributed to a model that never saw the run.

**`PARTIAL` still carries a real number.** It means the job finished and something else about it did
not: your content policy kept the judge's reasoning out of stored results, or the score was
delivered by a retry after an earlier attempt did not finish. The score itself is exactly the one the
evaluation produced, and a release gate accepts it. `FAILED` and `CANCELLED` mean the score's storage
was still owed when the status was written — you can only see them on the immediate reply to a
one-off evaluation, never on a stored result — and a release gate refuses them.

**An objective is an evaluator *at a version*, and another version's results are not its
evidence.** A gate bound to version 2 of an evaluator whose results were all produced by version 1
reads `NO_EVIDENCE` — "nothing has scored what you bound" — never `INADMISSIBLE`: those rows are no
more relevant to it than another evaluator's would be, so there is nothing to refuse.

**A score that cannot answer these questions cannot gate a release.** A result written before this
record existed returns empty values here; it is still readable, and a release gate refuses it for
the same reason it refuses a result with no evaluator version. An unexplained number is not evidence.

### Quality feedback recorded before evaluators existed

Some results predate the evaluator model on this page. They were recorded when quality feedback was
just a number and a label attached to a run — a reviewer's rating, or an automatic judgment with no
versioned configuration behind it. They are returned by the same endpoint, so nothing you recorded
earlier disappeared, and two fields mark them:

| Field | |
|---|---|
| `provenance.legacy_source` | Non-empty, and it is the original kind of the judgment (for example `human`). Every result recorded since evaluators existed leaves this **empty**. |
| `provenance.evaluator.version` | `0`. Every other result carries an exact, non-zero version. |

A result marked this way is **readable and never usable as evidence**. A release gate refuses it, and
an SLO rule cannot select it — both bind an exact evaluator version, and `0` is reserved so that no
exact reference can ever match one. That is deliberate: the whole point of an exact version is that
the same question is being asked each time, and these results cannot say what question was asked.

Their `evaluator_id` reads as `legacy/<author>/<metric>`. It is not an evaluator you can fetch,
version or run — it exists so that two people's separate ratings of the same run stay two separate
results.

One caveat worth stating plainly: `occurred_at` on these results is when the feedback was
**recorded**, not when the run happened, because the original record never captured the run's own
time. It is close enough to find a result by its subject and wrong for a trend — which is another
reason nothing computes one from them.

### Finding the release where a metric changed

Ask for one metric produced by one exact evaluator version. Keeping the version exact is essential:
otherwise changing the judge can look like changing the agent.

```bash
anectico evals release-series \
  --evaluator task_completion --evaluator-version 1 --metric task_completion \
  --start 2026-08-01T00:00:00Z --end 2026-09-01T00:00:00Z -o json
```

The response returns declared releases oldest first. Each point has its sample count, mean, first and
last observed run time, and—after the first point—the arithmetic change from the previous release.
That change is deliberately neutral: a negative completion delta may be a regression, while a
negative latency delta may be an improvement.

If every release name is semantic versioning, semantic precedence orders the series. If even one is
an arbitrary label, the **whole** series uses first-observed run time, with the label only breaking a
tie; `ordering_strategy` tells you which rule was applied. This whole-catalog fallback avoids a
contradictory order made from comparing some pairs by version and other pairs by time.

Runs that explicitly declared no release are returned as a separate `unreleased` aggregate with
`unreleased_present: true`. They are never dropped and are never invented as a release with a place
in the order. The equivalent REST request is:

```bash
curl "https://app.anectico.com/api/v1/evaluation/release-series?evaluator_id=task_completion&evaluator_version=1&metric_name=task_completion&start=2026-08-01T00:00:00Z&end=2026-09-01T00:00:00Z" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"
```

## Human judgments

An evaluator's result answers "what did the judge say". A **human judgment** answers "what did a
person think", and the two are recorded separately and read separately.

```bash
# Record or replace YOUR judgment of one run.
curl -X POST "https://app.anectico.com/api/v1/agent-runs/$RUN_ID/annotations" \
  -H "Authorization: Bearer $ANECTICO_USER_TOKEN" -H 'Content-Type: application/json' \
  -d '{"metric_name":"helpfulness","numeric_value":0.8,"label":"good","comment":"answered the actual question"}'

# Read every reviewer's judgments of one run.
curl "https://app.anectico.com/api/v1/agent-runs/$RUN_ID/annotations" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

# Read judgments for many runs at once, for a badge grid.
curl "https://app.anectico.com/api/v1/evaluation/annotations?subject_refs=$RUN_A&subject_refs=$RUN_B" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"
```

Five things about this surface are deliberate and worth knowing before you use it.

**You are always the author.** There is no author field to send: the judgment is attributed to
whoever the credential belongs to. A credential with no person behind it — an API key — cannot record
one at all, and is told so rather than filing the judgment under nobody. Re-judging the same run on
the same metric replaces **your** judgment and nobody else's.

**An absent number is not a zero.** `numeric_value` is optional, and leaving it out records "not
scored" — which is what a thumbs-down with no number means. Sending `0` records a score of zero,
which is a different statement about the run. At least one of `numeric_value` and `label` is
required: a judgment that states neither says nothing.

**`metric_name` is free text.** It is not a reference to a registered evaluator, because a person may
judge a dimension nothing automated measures yet — and requiring an evaluator first would make the
human loop wait on the thing it exists to calibrate. It is lower-cased and trimmed.

**A judgment is never evidence.** Release gates and SLO objectives read evaluator results and cannot
reach human judgments at all. That is the point of keeping them apart: comparing what a person said
with what a judge said is only a well-posed question while the two are recorded separately.

**Reading one needs no content scope, and neither does writing one.** A reviewer's `comment` is their
own sentence about a run rather than a restatement of what the run contained, so it is returned to
anyone who may read judgments. And recording a judgment needs only the write scope — nothing here
checks whether the run you named exists, so pointing at a run you may not read tells you nothing
about it.

Reading judgments accepts `metric_name`, `author_id` and `limit`. `author_id` narrows the answer and
is **not** a permission boundary: everyone who may read judgments in a project sees every colleague's,
because a rating nobody else can see cannot be disagreed with. The batch read takes `subject_refs` on
the same terms as results — at most 200 runs, refused rather than shortened, `limit` bounding each
run's share — and its response carries the same exact `truncated` and `truncated_subject_refs`.

Historical note: quality feedback recorded before this surface existed is in **results**, not here.
See the section above — nothing was lost, but a report that compares people with judges has to read
both.

### Calibration — how far a judge agrees with your people

A model judge's score is a claim, and until somebody checks it against a person it is a claim nobody
has tested. **Calibration** measures that agreement over subjects you have both judged, and records
it on the exact evaluator version.

```bash
curl -X POST \
  "https://app.anectico.com/api/v1/evaluation/evaluators/$EVALUATOR_ID/versions/4/calibrations" \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"subject_refs":["'"$RUN_A"'","'"$RUN_B"'"],"min_labels":20}'

curl "https://app.anectico.com/api/v1/evaluation/evaluators/$EVALUATOR_ID/calibrations" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"
```

Or from the CLI:

```bash
anectico evals evaluators calibrate quality 4 --min-labels 20 --subject run-a --subject run-b
anectico evals evaluators calibrations quality
```

**Which statistic you get depends on your labels.** Where reviewers recorded a **number**, agreement
is Lin's concordance coefficient between their ratings and the judge's. Where they recorded a
**verdict**, it is Cohen's kappa — chance-corrected agreement, not the raw match rate. That
distinction is the point: a judge answering "good" to everything matches an 80%-good corpus 80% of
the time and has established nothing. The raw rate is still reported as `agreement_rate` because it
is a fact; it is never the calibration.

The same reasoning is why the numeric statistic measures **agreement** rather than correlation. A
judge that rates every response exactly half as highly as your reviewers do tracks them perfectly —
plain correlation scores that a flawless 1.0, and it would clear any threshold you set. But its
numbers are on a scale nobody agreed to, so every absolute threshold you write against them is
wrong. Concordance penalises that offset: the same judge scores about 0.57. If your judge's
agreement drops after you upgrade, this is the likely reason, and the fix is to correct the judge's
scale rather than to lower the gate.

**Every state except one carries no number**, and that is deliberate — an agreement figure nobody
measured must never read as one.

| State | What it means |
|---|---|
| `CALIBRATED` | Agreement was computed. This is the only state carrying `agreement`. |
| `UNCALIBRATED` | A model judge nobody has compared to a person. |
| `INSUFFICIENT_LABELS` | Fewer comparable judgments than `min_labels`. Not a weak number — no number. |
| `UNDEFINED_STATISTIC` | Enough judgments, and a statistic that cannot be computed from them: everyone used one and the same verdict, or both sides gave the identical unvarying rating. Note that only **both** sides being constant is undefined — if your reviewers varied and the judge answered the same number every time, that is measurable disagreement, and it scores zero rather than landing here. |
| `NOT_APPLICABLE` | The evaluator is not a model judge. A deterministic or reference-trajectory evaluator is reproducible from its own definition. |

`min_labels` is required and zero is refused, on the same terms as a gate's `min_samples`: a
calibration you will act on over four judgments is one to ask for out loud.

**A record is immutable per (version, label set).** Recomputing over unchanged judgments returns the
existing record with `created: false`; a reviewer changing their mind, or the judge being re-run,
produces a **new** record beside it, so the history shows whether agreement moved when you changed
the rubric. Each record names the exact judgment ids it was computed from.

Judgments about subjects the judge never scored are **excluded and counted** rather than treated as
disagreements — a sampling gap is not a bad judge — and a judge that could not produce a verdict at
all is excluded on the same terms. `pair_count` is what was compared and `excluded_count` is what was
not.

Calibration reads your judgments and the judge's results and does arithmetic. It calls no model and
**spends no evaluation budget**.

The posture appears in three places, so a number's credibility travels with it: on the evaluator
version, on **every result** that version produced, and as the rule a release gate applies before it
will treat a judge's score as evidence.

### The explanation is treated as content

An explanation describes the run it scored, so it can restate what the run contained. Two independent
controls decide whether you receive it, and **both** have to permit it:

- **Your credential** must hold `agents:content:read`. Without that scope the field is empty.
- **Your project's content policy** must allow model transcripts *and* tool arguments to be read. An
  explanation is a single piece of prose that may quote either, so it is disclosed only when both are
  allowed — no scope overrides that, and the same rule governs whether the reasoning is written down
  in the first place. If your project's policy cannot be read at the moment, the field is empty until
  it can be; nothing is lost, and reading again later returns it.

The **score and the label are not withheld** from anyone with read access, whatever the policy says.
They are the answer, and they are what quality badges are built from — which is also why a label is
held to a short verdict word: a model that answers with a paragraph gets `scored` instead, and its
number is unaffected.

One exception, and it is narrow: quality feedback recorded **before this surface existed** and
migrated into results carries the reviewer's own comment in `explanation`. A comment a colleague
typed is served on the same terms as a judgment's `comment` field below — to anyone who may read
judgments — because it is a person's sentence about a run rather than a machine's restatement of one.
Every migrated *machine* comment stays behind the two controls above.

## Datasets — a frozen set of cases

A **dataset** is a named collection of cases you promoted out of real recorded runs. Its metadata is
editable; its **versions are not**. Promoting cases seals a new version containing everything the
current version had plus the new cases, and removing cases seals a new version without them — so a
score that cites a version keeps meaning exactly what it meant when it was recorded.

Only a **human-reviewed** production run can be promoted. Record a judgment first, then pass the
returned annotation id with that exact run:

```bash
REVIEW_ID=$(curl -sS -X POST \
  "https://app.anectico.com/api/v1/agent-runs/$RUN_ID/annotations" \
  -H "Authorization: Bearer $ANECTICO_USER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"metric_name":"dataset_eligibility","label":"reviewed"}' |
  jq -r '.annotation.annotation_id')

curl -X POST \
  "https://app.anectico.com/api/v1/evaluation/datasets/regression_set/promotions" \
  -H "Authorization: Bearer $ANECTICO_USER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"subjects":[{"subject_kind":"SUBJECT_KIND_AGENT_RUN","subject_ref":"'"$RUN_ID"'","review_annotation_id":"'"$REVIEW_ID"'"}]}'
```

The review must belong to the same project and name the same run. The frozen item records the review
id, authenticated reviewer, and review time. Promotion refuses a missing or mismatched review before
it freezes or stores anything; holding `evals:write` does not itself count as a review.

Each case's id is a digest of its own content, which has two consequences worth knowing:

- Promoting the same run twice gives you **one** case, not two.
- A case keeps the same id as it is carried into later versions, so "this case scored 0.4 in v1 and
  0.9 in v3" is a question you can ask.

A version number is always **exact**. Reading a version's cases requires you to name it; there is no
"latest", because a set of cases that can change under a comparison is not a set of cases.

Nothing is ever deleted. A version cited by a stored score cannot be removed, and neither can a
dataset that has one.

**What a case contains.** A frozen, self-contained projection of the run — enough to score it again
later without the original run still existing. If your project's content settings did not permit a
class of content to be copied, the case records which classes were withheld rather than being
quietly short: a case frozen without its content would score differently for ever, and you would have
no way to tell it from one that never had any.

## Replaying one case in a sandbox

A **replay** re-executes the recorded decision sequence of one frozen case inside a sandbox that
cannot reach anything outside this platform. It **stores nothing**. A replay is a pure function of a
frozen case and the plan you send, and both are fixed — so asking again reproduces the same answer
byte for byte, and you never have to keep one in order to get it back.

### The plan is yours to declare

Nothing here knows what your tools do, so the plan is where you say it. Each policy names a tool by
the name the recording carries and classifies it by what the call does **outside** the sandbox:

| Effect | Meaning | What the sandbox does |
|---|---|---|
| `SANDBOX_EFFECT_NONE` | a read-only call | replays the recorded answer, if there is one |
| `SANDBOX_EFFECT_INTERNAL` | writes inside this platform | answers with a marked stub |
| `SANDBOX_EFFECT_EXTERNAL` | leaves this platform | answers with a marked stub |

`default_effect` covers any tool no policy names. **Unset means external**, and so does a call the
recording does not name — not even an explicitly open default reaches that one. The recoverable
mistake is stubbing a read; the unrecoverable one is letting a write through because nobody
classified it.

A tool declared twice is refused rather than resolved: two policies for one tool is two answers to
one question, and whichever won would depend on the order you happened to list them in. An effect
value the contract does not declare is refused too, with the accepted values named — it is **not**
quietly read as unset, because "you did not classify this tool" and "we could not read what you
wrote" must not produce the same run.

Sending no plan at all is a valid request: everything is external, so everything is stubbed.

### What comes back

The replayed run, encoded exactly as a frozen case is, so you can diff the two directly — plus a
record of what the sandbox did:

- a **digest of the plan**, which identifies the sandbox that produced this answer; two plans that
  behave identically carry one digest;
- how many steps and tool calls there were, and how many were replayed, stubbed, or had nothing
  recorded to replay;
- the **sorted set of tools the sandbox answered for** — the actionable half, because a case that
  depends on those cannot be reproduced anywhere the sandbox does not reach;
- one entry per step, in order, so the counts are checkable rather than trusted.

Each step carries one of four dispositions: `carried` (not a tool call — replayed as recorded),
`recorded` (a read-only call whose recorded answer was replayed), `unrecorded` (a read-only call
with no recorded answer — the absence is carried through rather than filled), and `stubbed` (the
sandbox answered because the call had an effect it is not permitted to have).

A stubbed call is **stubbed, not skipped**: skipping it would change the sequence and the replay
would then be of a different agent. Any recorded answer on an effectful call is **discarded**, so a
stubbed run can never be mistaken for one that really ran.

### What a replay can and cannot tell you today

A recorded run carries the tools an agent called and the order it called them in. It does **not**
carry what those calls returned. So on today's recordings every read-only tool call comes back
`unrecorded`, and every effectful one comes back `stubbed` — the replay is an honest re-execution of
the **control flow**, not of the tool exchanges, and the record says which it was for every step.
That is the whole of what can be known from a recording that holds no tool results, and reporting it
as anything more would be a number you could not check.

The recorded outcome of the run is left exactly as it was. Overwriting it would silently change what
every evaluator that reads an outcome would say, and the replay would then be measuring the sandbox.

## Experiments — comparing two configurations on the same cases

An **experiment** compares two or more configurations of your agent on ONE frozen dataset version,
scored by the same evaluators at the same exact versions. One configuration is the **baseline**;
every other is compared against it.

One experiment can compare the five operational dimensions together: quality, safety, reliability,
latency, and cost. Declare one exact evaluator version for each dimension. The comparison output is
then one paired result per candidate and evaluator; a dimension appears only when both arms produced
real scores for the same cases.

Because each configuration is measured on the **same case**, the comparison is paired — which removes
the case-to-case variation that is usually far larger than the difference you are looking for.
Comparing two configurations' averages over *different* cases asks a question that one hard case can
answer wrongly.

**What you supply.** The platform records what your agent did; it does not run your agent. So for
each case you tell it which recorded run each configuration produced — a **trial**. Leaving the run
out means "score the frozen case itself", which is the fully repeatable arm: frozen cases cannot
change and need no original run to still exist.

**What you get back**, per configuration and per evaluator:

| Field | Meaning |
|---|---|
| `paired_items` | How many cases were scored under **both** configurations. Only these are compared. |
| `baseline_only_items`, `candidate_only_items` | Cases one side has and the other does not. Excluded, and reported — a comparison resting on a third of your dataset looks exactly like one resting on all of it. |
| `mean_difference` | The average per-case difference. |
| `ci_low`, `ci_high`, `confidence_level`, `ci_method` | The interval for that average, and the procedure that produced it. |
| `p_value`, `p_value_method` | Before the multiple-comparison correction. |
| `adjusted_p_value`, `significant` | After it, at the alpha you declared. |
| `small_sample` | Fewer than eight paired cases — the interval is reported, and is not worth much. |
| `degenerate` | Every case differed by the same amount, so the interval has zero width for an arithmetic reason. |
| `unevaluated` (on the response) | Declared trials with no score yet. |

### You must say how multiple comparisons are handled

Comparing several configurations, or several evaluators, makes it more likely that *something* looks
significant by chance. So an experiment cannot be created without saying what to do about that:

| `correction` | What it controls |
|---|---|
| `NONE` | Nothing. The p-values are uncorrected, and you have said so deliberately. |
| `BONFERRONI` | The chance of **any** false positive across the whole comparison. |
| `HOLM` | The same thing, less conservatively. The better of the two if you want that guarantee. |
| `BENJAMINI_HOCHBERG` | The expected **proportion** of false positives among the findings. A weaker, different guarantee. |

There is no default. Every comparison you read back states the method, the alpha, how many
comparisons it covered, and — in words — what it actually controls, so a number is never quotable
without what it claims.

The correction is fixed when the experiment is created and cannot be changed afterwards, along with
the dataset version, the evaluator versions and the candidate set. A choice made after the numbers
are in is not a correction.

### Running one

Running an experiment scores the outstanding cases and is deliberately **resumable**: one call does a
bounded amount of work and tells you how many cases remain. Call it again while `remaining` is above
zero. Repeating a call never re-scores a case that is already done, and never charges you twice for
one.

### Repeatability

Re-running an experiment over **frozen cases** with the deterministic evaluators reproduces every
number exactly. Two things are outside that promise, and are worth knowing before you rely on it: a
trial pointing at a live recorded run depends on that run still being there and unchanged, and an
LLM-judge evaluator varies from run to run on its own.

An experiment is limited to 8 configurations, 8 evaluators, and 20 000 total case-scores. The limit is
checked when you create it, so an experiment too large to compare is refused up front rather than
after you have paid for the scoring.

## Release gates — deciding whether a release may ship

A **release gate** binds a candidate release to evaluation evidence and produces a **pass or a
refusal with the reason**, kept for ever. It is the thing that turns a comparison into a decision.

A gate is fixed when you create it — its evidence, its objectives, its confidence level. A changed
objective is a new gate, because a gate whose bar can move under it is not a gate.

### The two kinds of evidence, which are different designs

| `evidence_kind` | What it compares | Paired? |
|---|---|---|
| `EXPERIMENT` | One arm of a declared experiment against **that experiment's own declared baseline**, on the same frozen cases. | Yes — the same case is scored under both configurations. |
| `CANARY` | Two windows of live traffic that you declare: one before the candidate shipped, one after. | No. The runs in the two windows are different runs, so the interval carries the full case-to-case variance. |

You do not name the comparison arm for experiment evidence — the experiment's own declared baseline
is used. Choosing the arm afterwards is how a flattering pair gets picked.

For a canary, the window boundary is **your declaration** of when the candidate shipped. Everything
else that changed at that boundary is confounded with the release, and every stored canary comparison
says so in words alongside its numbers, because a narrow interval and a causal claim look identical
on a dashboard.

### Objectives

Each objective names an evaluator at an **exact version**, the metric it must produce, what to
compare, and the line to clear:

| `basis` | The number compared against the threshold |
|---|---|
| `CANDIDATE_MEAN` | The candidate's own average. An absolute floor or ceiling; needs no baseline. |
| `DIFFERENCE_MEAN` | The point estimate of candidate minus baseline. |
| `DIFFERENCE_INTERVAL` | The conservative bound of the interval around that difference — the low bound for a floor, the high bound for a ceiling. The strong form: the whole interval must clear the line, so a candidate that passes on noise does not. |

`min_samples` is required and zero is refused. A gate that will pass on a single observation is
something you have to ask for out loud.

`min_judge_agreement` is optional and puts a **floor** on how far the objective's evaluator must
agree with human judgments — see [Calibration](#calibration--how-far-a-judge-agrees-with-your-people).
It is **not** a way to opt out: an objective bound to a model judge with no calibration at all is
inadmissible whatever this field says, and a gate whose only evidence comes from that judge answers
`INADMISSIBLE` rather than passing. Leaving it at zero asks only that a calibration exist.

The rule applies to model judges alone. A deterministic or reference-trajectory evaluator is
reproducible from its own definition and is calibrated against nobody, so nothing is asked of it.

Calibration is resolved when the gate is **evaluated**, not when it is created. A gate written before
its judge was calibrated passes once the judge is; a judge whose calibration was superseded by an
edited judgment set stops passing until it is recomputed.

An objective is checked against your evaluator registry when the gate is **created**, including that
the evaluator you named actually produces the metric you named — so a gate that could only ever
measure nothing is refused before it exists rather than after a release waited on it.

### Verdicts: only `PASS` passes

| Verdict | What it means | Where to look |
|---|---|---|
| `PASS` | Every objective was measured on admissible evidence and met. | — |
| `FAIL` | At least one objective was measured and missed. | The objective's numbers. |
| `PENDING` | Created, never evaluated. **Not** a pass. | Evaluate it. |
| `NO_EVIDENCE` | An objective had nothing admissible to measure, or less than it declared it needs. | Your sampling — nothing ran. |
| `INADMISSIBLE` | Results exist for an objective and **none of them may be used** — incomplete provenance, an execution record that cannot say how the score was produced, a job that did not complete, or a model judge that has not been calibrated. | Your evaluator reference — things ran, and cannot be used. |
| `RELEASE_MISMATCH` | Every objective met, but the candidate runs did not all observably declare the release the gate names. **Not** a pass. | The decision's `release_binding`: it distinguishes another release, no declared release, partial instrumentation, and an unmeasured subject shape. |

The last two are kept apart deliberately: "nothing ran" and "what ran cannot be used" are different
problems with different fixes, and one empty answer cannot tell them apart.

Per objective you also get `observed_present`, which is the **only** way to tell "the value was 0"
from "there was no value". A score of `0.0` is a legitimate measurement, so the number can never be
that discriminator — never infer absence from `observed_value`. `inadmissible_samples` counts results
that exist and may not be used; they are reported rather than filtered away, for the same reason.

### Evaluating one

The first call reaches a verdict over the evidence available then and appends one permanent
decision. You may supply an `idempotency_key`; repeating the same gate, caller and key returns that
existing decision with `replayed: true` — even if new results arrived between the calls. If you omit
the key, Anectico derives one stable default decision identity from the organization, project, gate
and authenticated caller. Repeating the omitted-key call therefore replays that first decision too;
it never silently regains at-least-once append behavior. The original `decided_at`, sample counts
and outcomes show which evidence snapshot you received.

Use a **new explicit** key when you deliberately want to evaluate newly arrived evidence. That
creates a new decision beside the first, and the whole history stays readable. Evidence is
intentionally not part of the retry identity: otherwise a response-loss retry could silently become
a contradictory second decision just because a result landed in between.

A gate never runs an evaluator and never spends your evaluation budget. That is deliberate — a gate
that produced its own evidence on demand could never report `NO_EVIDENCE`, which is the one answer it
exists to make reachable. Score first, then gate.

If the evidence store cannot be read, you get an **error and nothing is stored**. "We could not look"
is never recorded as "we looked and found nothing".

### A release label is a claim that the evidence must prove

`release` is the label you declare when creating the gate. Evaluation reconciles it against the
release actually carried by every admissible candidate run, deduplicated across objectives; baseline
runs are expected to belong to the earlier comparison arm and do not participate in this check.

Every decision stores `release_binding` with the observed release names, released, unreleased, and
unmeasured subject counts, a reason, and one of `MATCHED`, `MISMATCH`, `UNDECLARED`, `PARTIAL`,
`UNMEASURED`, or `NO_EVIDENCE`. Only `MATCHED` may accompany `PASS`. If the objective numbers meet
but the label does not, the verdict is `RELEASE_MISMATCH` rather than a green decision carrying a
warning. If an objective already missed, the verdict remains the more direct `FAIL`, but the same
binding discrepancy is still recorded.

“No release declared” is a first-class answer. A gate evaluated over those runs returns binding
`UNDECLARED` and cannot pass; it does not error, silently discard those runs, or pretend they belong
to the gate's label.

## Permissions

| Scope | Grants |
|---|---|
| `evals:read` | List and read evaluators, sampling rules, the budget, results, the ordered observed-release series, datasets, experiments — including an experiment's comparison — and release gates with their full decision history. Also replay one frozen case in a sandbox: it stores nothing, and it returns strictly less than reading the case does. |
| `evals:write` | Create evaluators and versions, manage sampling rules, set the budget, evaluate on demand, create datasets and promote reviewed cases or remove cases, create experiments and their trials, run an experiment, and create a release gate or evaluate one. Evaluating a gate is a write because it permanently records whether a release is admitted or held — a key minted only to look at gates must not be able to create it. |
| `agents:content:read` | Additionally see the `explanation` field on results, when the project's content policy also allows it. |
| `scores:write` | Record or replace your own human judgment of a run. It is a **member** capability rather than an administrative one, deliberately: rating a run is something anyone reviewing the product does. |
| `scores:read` | Read the human judgments recorded for runs. |

A project-scoped key can only reach its own project; a request naming another project returns
`404 Not Found`, the same answer an evaluator that does not exist returns.

## Endpoints

| Method | Path |
|---|---|
| `GET` | `/api/v1/evaluation/evaluators` |
| `POST` | `/api/v1/evaluation/evaluators` |
| `GET` | `/api/v1/evaluation/evaluators/{evaluatorId}` |
| `POST` | `/api/v1/evaluation/evaluators/{evaluatorId}/versions` |
| `GET` | `/api/v1/evaluation/evaluators/{evaluatorId}/versions/{version}` |
| `GET` | `/api/v1/evaluation/sampling-rules` |
| `PUT` | `/api/v1/evaluation/sampling-rules` |
| `DELETE` | `/api/v1/evaluation/sampling-rules/{ruleId}` |
| `GET` | `/api/v1/evaluation/budget` |
| `PUT` | `/api/v1/evaluation/budget` |
| `POST` | `/api/v1/evaluation/evaluate` |
| `GET` | `/api/v1/evaluation/results` |
| `GET` | `/api/v1/evaluation/release-series` |
| `POST` | `/api/v1/agent-runs/{runId}/annotations` |
| `GET` | `/api/v1/agent-runs/{runId}/annotations` |
| `GET` | `/api/v1/evaluation/annotations` |
| `GET` | `/api/v1/evaluation/datasets` |
| `POST` | `/api/v1/evaluation/datasets` |
| `GET` | `/api/v1/evaluation/datasets/{datasetId}` |
| `GET` | `/api/v1/evaluation/datasets/{datasetId}/versions/{version}/items` |
| `POST` | `/api/v1/evaluation/datasets/{datasetId}/promotions` |
| `POST` | `/api/v1/evaluation/datasets/{datasetId}/removals` |
| `POST` | `/api/v1/evaluation/datasets/{datasetId}/versions/{version}/items/{itemId}/replays` |
| `GET` | `/api/v1/evaluation/experiments` |
| `POST` | `/api/v1/evaluation/experiments` |
| `GET` | `/api/v1/evaluation/experiments/{experimentId}` |
| `POST` | `/api/v1/evaluation/experiments/{experimentId}/trials` |
| `POST` | `/api/v1/evaluation/experiments/{experimentId}/run` |
| `GET` | `/api/v1/evaluation/experiments/{experimentId}/comparison` |
| `GET` | `/api/v1/evaluation/gates` |
| `POST` | `/api/v1/evaluation/gates` |
| `GET` | `/api/v1/evaluation/gates/{gateId}` |
| `POST` | `/api/v1/evaluation/gates/{gateId}/evaluations` |

Promoting cases and removing them are both `POST`s that seal a **new** version, rather than a `PUT`
or a `DELETE` on an existing one — nothing about a sealed version ever changes. Evaluating a gate is
a `POST` onto `/evaluations` for the same reason: a new explicit idempotency key appends a decision
rather than editing one. Its optional JSON body is
`{ "idempotency_key": "your-stable-decision-key" }`; an empty body selects the stable default
decision intent.

Replaying a case is a `POST` for a different reason: it carries the sandbox plan, not because
anything is written. A replay stores nothing and reproduces its answer every time, which is why
it needs only `evals:read`.

`GET /api/v1/evaluation/gates` accepts an optional `?release=` to narrow to one release label.

All accept an optional `?project_id=` query parameter; a project-scoped key is pinned to its own
project regardless. `/budget` is the exception: it is org-wide and reads no project.

## From the command line, and from an agent

The API is the complete surface. The CLI mirrors it; MCP deliberately exposes a narrower subset.

**The CLI** mirrors the endpoints one-for-one under `anectico evals`:

```bash
anectico evals datasets list
anectico evals datasets items refunds --version 3
anectico evals experiments comparison exp-refunds
anectico evals release-series --evaluator task_completion --evaluator-version 1 --metric task_completion
anectico evals gates list --release 2026.08.3
anectico evals gates evaluate gate-1 --idempotency-key release-2026-08-26
anectico evals datasets replay refunds case-17 --version 3 --file plan.json
anectico evals annotations add $RUN_ID --metric helpfulness --label good --comment "answered the question"
anectico evals annotations list $RUN_A $RUN_B $RUN_C
```

`anectico evals annotations add` omits the number unless you pass `--value`, so a `--label`-only
judgment is recorded as "not scored" rather than as a zero. `list` names every run you give it in one
request.

Structured declarations — an experiment, a gate, a promotion — are supplied as JSON with `--body` or
`--file` (`--file -` reads standard input), and are sent to the API exactly as you wrote them. Reads
print a table on a terminal and JSON when piped; `--output json` forces it. `anectico evals` also
covers `evaluators`, `rules`, `budget`, `evaluate` and `results`. Run `anectico evals --help`, or see
the [CLI reference](/docs/reference/cli).

**An MCP agent** can read this surface through the read gateway: `list_eval_datasets`,
`get_eval_dataset`, `list_eval_experiments`, `get_eval_experiment`, `compare_eval_experiment`,
`list_release_gates` and `get_release_gate`, all under `evals:read`. They are reached through
`list_read_actions` and `execute_read_action` rather than appearing as tools of their own.

Three writes are available through `list_write_actions` and `execute_internal_action`.
`create_release_gate` and `evaluate_release_gate` require `evals:write`; both preview first because
gates and decisions are permanent and have no delete, void, or correction path. Creation requires a
caller-owned `gate_id`, so its retry is refused by the unique ID instead of minting a duplicate.
Evaluation through MCP requires `idempotency_key`; reusing it safely returns the original decision,
while a new key asks for a fresh evidence snapshot. The API and CLI also accept omission and use the
server-derived default intent described above. Both actions are gatewayed and add no first-class
host tool.

`submit_annotation`, under `scores:write`, is also reached through
`list_write_actions` and `execute_internal_action`. It previews first and applies on a second call
with the returned `confirm_token`, because re-judging replaces the caller's previous judgment of that
run. Its receipt reports the metric, the number and the label; it does not repeat the rationale back.

Three things an agent cannot do over MCP, deliberately: it cannot read a dataset version's frozen
**cases** — those can carry recorded content, and the read that returns them is available over the
API and the CLI only — it cannot read the ordered release series yet, and it cannot change the
evaluation **configuration**. Creating a dataset, promoting cases, and declaring or running an
experiment remain API and CLI operations.