Quality evaluations

An agent run either did its job or it did not, and error rates do not tell you which. Evaluations attach a score to a run — task completion, whether the expected steps happened in the expected order, anything you can state as a rule — and record enough about how that score was produced that you can reproduce it months later.

Evaluations run automatically against a sample of your production traffic, or on demand for one run you name.

The three pieces

Piece	What it is
Evaluator	A named, versioned rule that turns a run into a number and a label.
Sampling rule	Which runs an evaluator sees, at what rate, with what per-tick ceiling.
Result	One score, with the full record of what produced it.

Evaluator versions are exact, and never float

An evaluator has an id and a list of versions. A version is immutable: changing a rule creates a new version and leaves the old one exactly as it was.

Anywhere you reference an evaluator, you name the version — and the version must be a real number, never “the latest”. This is deliberate and it is not a formality. If a rule could change under a reference, the same release would pass one day and fail the next with nothing in the record saying why, and any comparison between two releases would silently be a comparison between two different questions.

The reference is checked when you write the thing that cites it, so a typo or a missing version is an error on the request you just made rather than a background job quietly doing nothing.

Creating an evaluator

curl -X POST https://app.anectico.com/api/v1/evaluation/evaluators \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator_id": "task_completion",
    "display_name": "Task completion",
    "metric_name": "task_completion",
    "kind": "EVALUATOR_KIND_DETERMINISTIC",
    "config_json": "{\"aggregation\":\"all\",\"assertions\":[{\"field\":\"run.status\",\"op\":\"equals\",\"value\":\"AGENT_RUN_STATUS_COMPLETED\"},{\"field\":\"run.error_steps\",\"op\":\"lte\",\"number\":0}]}"
  }'

The response carries the evaluator and its version 1. Append a new version with POST /api/v1/evaluation/evaluators/{evaluatorId}/versions; there is no update or patch, by design.

Evaluator kinds

EVALUATOR_KIND_DETERMINISTIC — declarative assertions over a run’s observable fields.

{
  "aggregation": "all",
  "assertions": [
    {"field": "run.status",       "op": "equals",   "value": "AGENT_RUN_STATUS_COMPLETED"},
    {"field": "run.error_steps",  "op": "lte",      "number": 0},
    {"field": "steps.tool_names", "op": "contains", "value": "search_docs"}
  ]
}

aggregation is all (a gate: every assertion must hold, the score is 1 or 0) or fraction (the share of assertions that held).

Field	Type	Operators
run.status	one value	equals, not_equals
run.error_steps, run.step_count, run.tool_calls, run.model_calls, run.duration_ms	number	lte, gte
steps.outcomes, steps.tool_names, steps.canonical_operations	set	contains, not_contains
session.turn_count, session.run_count, session.duration_ms, session.cost_nanos	number	lte, gte

The session.* fields describe a whole conversation and are answerable only for SUBJECT_KIND_AGENT_SESSION — see What you can name as the subject.

A deterministic evaluator can instead return one native measurement. A measurement config contains measurement and no assertions or aggregation:

{"measurement":"run.cost_nanos"}

Measurement	Unit	When it is measurable
run.duration_ms	milliseconds	The recorded run reports a duration.
run.cost_nanos	billionths of a US dollar	The run has client-reported or calculated model cost. A missing or unpriced cost is not zero and produces no score.
safety.error_steps	count	The run contains explicitly identified safety-check steps with reported outcomes. No safety check, or an unknown outcome, is not a safe zero and produces no score.
session.turn_count	count	The session’s turns carry turn ids. A session with none is not a session of nought turns; it produces no score.
session.run_count	count	Always measurable for a session that exists.
session.duration_ms	milliseconds	The time between the session’s first and last recorded activity. A session with one moment of activity measures 0, which is a real answer.
session.cost_nanos	billionths of a US dollar	The session’s runs reported model cost. Nothing on a session says whether an unpriced conversation was free, so a zero is treated as unmeasured and produces no score.

Use the assertion form for reliability (for example, completed status and zero error steps), a reference-trajectory or judge evaluator for quality, and the native measurements for safety, latency, and cost. Experiments and release gates treat each as the evaluator’s real numeric result; they do not fill missing dimensions with defaults.

EVALUATOR_KIND_REFERENCE_TRAJECTORY — did the agent do the right things in the right order.

{
  "step_key": "canonical_operation",
  "match": "ordered_subsequence",
  "reference": [
    "CANONICAL_OPERATION_INVOKE_AGENT",
    "CANONICAL_OPERATION_RETRIEVAL",
    "CANONICAL_OPERATION_CHAT"
  ]
}

A run that retrieved after answering used the same steps as one that retrieved first, and a set-membership rule scores both identically. This one does not. ordered_subsequence (the default) scores the longest in-order match against the reference length, so an extra step between two expected ones is not a deviation; exact requires the sequence to match exactly. step_key may be canonical_operation or tool_name.

EVALUATOR_KIND_LLM_JUDGE — a model scores the run against a rubric you write.

The two kinds above answer questions you can state as a rule. Some questions cannot be: whether an answer was grounded in what the agent retrieved, whether it actually addressed what was asked. A judge answers those by reading evidence from the run and scoring it against criteria you declare.

{
  "instructions": "Be strict. Prefer a low score when you are unsure.",
  "criteria": [
    {"name": "grounded",  "description": "every claim in the answer traces to the retrieved context", "weight": 2},
    {"name": "addressed", "description": "the answer resolves the question that was asked"}
  ],
  "evidence": ["run_summary", "model_transcript"],
  "max_evidence_chars": 12000
}

Field
criteria	required, 1–8	Each needs a name and a description. weight defaults to 1.
evidence	required	What the judge is shown. See below.
instructions	optional	A preamble, rendered ahead of the criteria.
model	optional	A specific model id. Omit to use the default.
max_evidence_chars	optional, 500–40000	Evidence budget. Defaults to 12000.

A rubric is a structure, not a prompt. That is what makes two versions of “faithfulness” comparable: the difference between version 4 and version 5 is a criterion added, removed or reworded, rather than a change somewhere inside a paragraph. An unrecognised field is rejected rather than ignored, so a misspelled key is an error on the request instead of a rule that silently does nothing.

The result shape is fixed — one score, one label, one sentence of reasoning — and is not something a rubric sets.

What a judge is shown

evidence is a declaration, and it is the one that decides what leaves your project:

Selector	What the judge sees	Content
run_summary	Status, step counts, duration.	—
step_sequence	The ordered operations, outcomes and tool names.	—
model_transcript	Recorded prompt and response bodies.	model_transcript
tool_calls	Recorded tool and MCP call arguments and results.	tool_arguments

The first two carry no message bodies at all — a tool name is a symbol, not the arguments it was called with — so a rubric built only from them shows a model nothing you recorded.

Your content policy decides whether the evidence may be sent

Sending recorded content to an evaluation model is a transfer, and it is governed by your project’s content policy at the judge_transfer boundary. The policy is consulted before anything is sent, never after.

Two properties are worth stating exactly, because both are easy to get wrong and neither is visible from the outside:

• The question asked of your policy is derived from what the evidence actually contains, not from what the rubric might want. A rubric that selects tool_calls, running against a run that recorded no tool arguments, asks nothing about tool_arguments — so a project that permits transcripts and refuses tool arguments is evaluated normally on those runs.
• But a kind of content your policy actually withheld is never quietly dropped. When the content your rubric asked for was removed by the policy, the transfer is put to the policy again for exactly that kind, and a refusal is a refusal for the whole evaluation — 403 Forbidden — even when the rubric’s other selectors still had something to show. A judge scoring “was the answer grounded in the transcript” over a run summary alone is scoring something you did not ask about. If the removal was not your policy’s decision (a narrower credential, for instance), the evaluation proceeds and the model is told which kinds of content were withheld, by name, so it cannot read an absence as evidence that nothing was there.
• A refusal is terminal and it is recorded. The run is not sent, is never scored 0, and is not retried in a loop. It is recorded as withheld, naming the kind of content, the boundary, and which of three things decided: your policy said so, the project has no policy yet, or the policy could not be read. “Nobody judged this” and “nobody sampled this” stay different facts.

The read and judge_transfer cells are independent for automatic quality checks. A run may be hidden from product reads and still evaluated when judge_transfer allows its content; denying judge_transfer still prevents the transfer whatever the read cell says.

Why some rules are refused

Evaluator configuration is validated when you write it, and some rules are refused even though they are well-formed. Each refusal names its reason. The reason is always the same one:

A field that nothing fills is not a field you can measure.

A run’s operation, outcome and similar values carry a placeholder when your instrumentation did not report them. A rule written against that placeholder does not measure your agent — it measures how completely you are instrumented, and reports the answer as a quality score. Worse, a trajectory written against it scores higher the less instrumented the run is, inverting the metric entirely.

So you cannot assert on OUTCOME_UNKNOWN, on CANONICAL_OPERATION_UNKNOWN, or on fields no instrumentation populates. If you need a value that is refused, the fix is to emit it — see Instrumenting LLM and agent calls.

The same principle applies at run time. If a run carries nothing the evaluator can measure, it is recorded as unmeasured and retried later, never scored zero. A zero would say your agent failed; the truth is that nobody measured it. A judge is held to the same rule twice over: a run carrying none of the evidence its rubric selected is not sent to a model at all, and a model that comes back with something unusable is retried rather than recorded as a failing score.

Sampling production traffic

curl -X PUT https://app.anectico.com/api/v1/evaluation/sampling-rules \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator": {"evaluator_id": "task_completion", "version": 1},
    "subject_kind": "SUBJECT_KIND_AGENT_RUN",
    "sampling_rate": 0.1,
    "per_tick_limit": 50,
    "enabled": true
  }'

• sampling_rate (0–1) is a true cost cap. Selection is derived from a stable hash of the run, so the same run gets the same answer every time and the unselected share is never evaluated at all. It is not a per-cycle coin flip, which would eventually select everything.
• per_tick_limit caps how many evaluations start per cycle. Zero disables the rule, as does enabled: false or a rate of 0.

A rule is evaluated only where you create one. Nothing is evaluated by default.

Evaluating one run on demand

curl -X POST https://app.anectico.com/api/v1/evaluation/evaluate \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator": {"evaluator_id": "task_completion", "version": 1},
    "subject_kind": "SUBJECT_KIND_AGENT_RUN",
    "subject_ref": "0af7651916cd43dd8448eb211c80319c"
  }'

This is the same engine, with sampling skipped — naming a run is selecting it. Asking twice for the same run and evaluator version returns the stored answer with already_evaluated: true; it does not re-run and does not spend again.

Set your client’s timeout above 30 seconds

The call is synchronous: it reads the subject, and for an LLM-judge evaluator it waits for the model. A judge call routinely takes 10–30 seconds and is allowed up to five minutes. Clients whose default request timeout is 30 seconds — which is most of them — will hang up on a judge that is still working, and the evaluation is then abandoned rather than stored.

Give this one endpoint its own timeout of six minutes — a little above the five the server allows, so that what you read is the server’s answer rather than your own client giving up — or use a deterministic evaluator (which answers in under a second) when you need a fast answer.

If the five minutes do run out, you get 504 Gateway Timeout with a message, not a dropped connection. The anectico CLI already applies its own longer timeout to anectico evals evaluate and anectico evals experiments run; every other command keeps the short default.

What you can name as the subject

subject_kind says what subject_ref is, and it is also the label the stored result carries. Three kinds can be evaluated today:

subject_kind	subject_ref is	Notes
SUBJECT_KIND_AGENT_RUN	the run (trace) id	The whole run. This is what sampling rules select.
SUBJECT_KIND_AGENT_STEP	"<run_id>/<span_id>"	One step of a run — the span_id is the one the run graph reports for that step. The score is about that step: the evaluator sees only it, not its siblings.
SUBJECT_KIND_AGENT_SESSION	the session key	The whole conversation: every run that shares that session key. Only the session.* fields can score it, and only a rubric that asks for no recorded content can be run over it — see below.

SUBJECT_KIND_AGENT_TURN is part of the result vocabulary — a stored result can carry it — but cannot be submitted here: a turn cannot be looked up from its id alone. SUBJECT_KIND_DATASET_ITEM is a batch subject and is evaluated by running an experiment over a dataset version, not through this endpoint. Naming either returns 501 Not Implemented. A result is never labelled with a subject that was not read: an unsupported kind is refused rather than scored from the run behind it.

Which fields an evaluator can assert on depends on what you named, and each family is answerable for its own subject only:

• run.* describe the whole run and are answerable for SUBJECT_KIND_AGENT_RUN and for dataset items. Asking one about a step or a session returns 400 Bad Request (“carries nothing this evaluator can measure”) rather than reporting the run’s number under another subject’s name.
• steps.* and safety.error_steps are answerable for a run and for a step, and for a step subject they describe that step alone.
• session.* describe a whole conversation and are answerable only for SUBJECT_KIND_AGENT_SESSION. Asking one about a single run of that session returns 400 Bad Request for the same reason in the other direction — a run’s own duration or cost is not the conversation’s.

What a judge reads for each subject

A judge that asked for model_transcript or tool_calls has to be shown recorded content, and what that content is depends on the subject:

• A run — that run’s own recorded content.
• A step — the content of the run the step belongs to, with the step itself as the subject the score is filed under.
• A session — nothing. A conversation is a set of runs and has no transcript of its own, and there is no answer to “show the judge the session” that would not mean choosing runs on your behalf. Rather than pick one, a content-asking rubric over a session is refused with 400 Bad Request naming the kind. A rubric built only from run_summary and step_sequence is not refused for content — it reads none — but a session rollup carries neither a run summary nor a step sequence, so it has nothing to show a judge either and answers “carries nothing this evaluator can measure”. Score a conversation with the session.* fields, which read the rollup itself.

Response	Meaning
404 Not Found	Nothing here matches what you named — the run, the step, or the evaluator version. Look the identifier up again; retrying it unchanged will not help. A run that was just produced may not be readable for a few seconds, and the same call succeeds once it is.
504 Gateway Timeout	Either the subject could not be read within its own budget, or the whole evaluation ran past the five minutes the call is allowed. This is not a 404: it says the subject could not be read, never that it does not exist, so retry the identifier you already have rather than looking for a new one. No result is stored. A judge that had already started may still have been charged, and the retry is free once a verdict exists — asking again for the same run and evaluator version returns the stored answer.
501 Not Implemented	This subject_kind cannot be evaluated on demand. See the table above; retrying will not help.
400 Bad Request	The subject_ref cannot name a subject of that kind — a step reference must be "<run_id>/<span_id>".
409 Conflict	Already being evaluated. Retry shortly.
429 Too Many Requests	The daily evaluation budget is spent.
400 Bad Request	The run carries nothing this evaluator can measure.
403 Forbidden	Your content policy withholds content your rubric asked for from a judge. The message names the kind of content and the boundary, never the content itself.
400 Bad Request	A judge whose rubric asks for recorded content cannot be run over this subject_kind. Today that is a session: a conversation has no single transcript to show, and the answer names the kind rather than reading one of its runs. Score the session with session.* fields, or judge its runs individually.
400 Bad Request	Judging is not configured or its credentials were permanently refused. An operator must repair the judge setup before retrying.
503 Service Unavailable	Judging is temporarily unavailable because of a timeout, throttle, transport failure, or upstream outage. Retry is appropriate.

The spend budget

Evaluators that call a model cost money, so the daily ceiling is a hard limit rather than a throttle: it is enforced transactionally, and it does not open up when anything is degraded.

curl -X PUT https://app.anectico.com/api/v1/evaluation/budget \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"daily_paid_evaluation_cap": 500}'

GET /api/v1/evaluation/budget returns the cap and today’s consumption.

Two things to know:

• An organization with no configured budget runs no paid evaluations. The safe state is the default; you opt in to spending. A judge in an organization with no budget never runs.
• Evaluators that do not call a model never consume budget. EVALUATOR_KIND_DETERMINISTIC and EVALUATOR_KIND_REFERENCE_TRAJECTORY are in that category, so they keep working after a paid budget is exhausted. EVALUATOR_KIND_LLM_JUDGE is not: it is charged one unit per run evaluated, once — a retried delivery of a score that was already produced never charges again.

The budget is charged after your content policy permits the transfer, so a run your policy withholds costs nothing.

Reading results

curl "https://app.anectico.com/api/v1/evaluation/results?metric_name=task_completion&start=2026-08-01T00:00:00Z&limit=100" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

Filters: metric_name, evaluator_id, subject_ref, subject_refs, mode, start, end, limit. Omitting mode returns every mode; each result states its own, so comparing across them is something you do deliberately rather than by accident.

subject_refs asks about many runs at once — repeat it (?subject_refs=a&subject_refs=b) or pass a comma list (?subject_refs=a,b); both work, and it combines with the singular subject_ref as one set. Use it to draw a quality badge for a whole page of runs in one request instead of one request per row. At most 200 runs may be named, counting the two parameters together and counting a run named twice once; a longer list is refused rather than shortened, because a silently trimmed list comes back looking complete while whole runs are missing from it and you would then draw “not evaluated” for a run that was.

limit means different things on the two shapes, and the difference exists for that same reason:

• Naming at most one run, it is the page size — default 100, maximum 500.
• Naming several, it bounds each run’s share of the page — default and maximum 20 — and the page is that share times the runs you named. A single total over many runs would be spent by whichever runs were evaluated most recently, and the runs at the far end would come back empty behind a 200 OK.

The response also carries truncated and truncated_subject_refs. truncated is true when a cap cut the page and there are more results matching your filters; it is exact rather than a guess from a full-looking page, so a page of exactly the cap still tells you truthfully whether anything was left behind. truncated_subject_refs names which runs were cut — one busy run sets the page-level flag for all 200, so if you are drawing a badge per run, treat exactly the runs listed there as unknown rather than as not evaluated, and read those again with a narrower window or a tighter metric_name. A run you asked about that is not in that list returned everything stored for it.

This read is a history, not a current value. Every evaluator version a run was measured under keeps its own result, on purpose: that is what makes “this run scored 0.2 under version 3 and 0.9 under version 4” answerable at all, and it is what a release gate compares. So a badge or an average built by summing everything this endpoint returns counts a run’s old verdicts alongside its current ones — a run that was fixed reads as half-fixed for ever. When you want the current picture, keep the highest provenance.evaluator.version for each (provenance.evaluator.evaluator_id, metric_name) pair and summarize those.

Every result carries:

Field
numeric_value, label	The score.
provenance.evaluator	The evaluator id and the exact version.
provenance.mode	How it was produced.
provenance.subject_kind, provenance.subject_ref	What was scored.
subject_release, release_declaration	The release observed on that subject, or the explicit answer that it declared none.
occurred_at	When the run happened.
evaluated_at	When the score was produced.
explanation	The evaluator’s reasoning.
provenance.execution	What the evaluation actually did — see below.

occurred_at and evaluated_at are separate on purpose. They differ by however long sampling and queueing took, and a trend plotted on the wrong one attributes a regression to the day it was measured rather than the day it shipped.

What the evaluation actually did

provenance.execution records how each score came to exist: why the subject was picked, what the evaluation cost, which model answered, and whether the job finished. It is stored on the result rather than looked up later, so it stays true even after the rule that selected the subject is edited or the evaluator is retired.

Field
sampling	SAMPLED (an online rule picked this subject), NOT_SAMPLED (you asked for this subject by name) or WHOLE_DATASET (a batch run measured every case).
sampling_rule_id, sampling_rate	The rule that fired and the rate it was applying at that moment. Present only for SAMPLED.
cost_state	CHARGED, FREE, UNPRICED or UNREPORTED.
charged_cost_nano_usd	The charge, in billionths of a US dollar. Meaningful only when cost_state is CHARGED.
input_tokens, output_tokens	What the model call consumed. Zero for an evaluator that calls no model.
judge_model_state	RECORDED, NOT_APPLICABLE (not a model judge) or UNREPORTED.
judge_model	The model that actually answered. Present only for RECORDED.
job_status	COMPLETED, PARTIAL, FAILED or CANCELLED.
job_failure_class	Names what was missing, for every status except COMPLETED.

Read cost_state before the charge. A charge of zero has two unrelated meanings — an evaluator that calls no model and costs nothing, and a model whose price is unknown — and the number alone cannot tell you which. UNPRICED is the second: tokens were spent, the price is not known, and the zero is an absence rather than a bill. It is never reported as a charge of zero.

judge_model is not the model the evaluator was configured with. The evaluator version records what was requested; this records what answered, and the two differ whenever a provider resolves a floating model name to a dated one or falls back to another model. A score attributed to the requested model would be attributed to a model that never saw the run.

PARTIAL still carries a real number. It means the job finished and something else about it did not: your content policy kept the judge’s reasoning out of stored results, or the score was delivered by a retry after an earlier attempt did not finish. The score itself is exactly the one the evaluation produced, and a release gate accepts it. FAILED and CANCELLED mean the score’s storage was still owed when the status was written — you can only see them on the immediate reply to a one-off evaluation, never on a stored result — and a release gate refuses them.

An objective is an evaluator at a version, and another version’s results are not its evidence. A gate bound to version 2 of an evaluator whose results were all produced by version 1 reads NO_EVIDENCE — “nothing has scored what you bound” — never INADMISSIBLE: those rows are no more relevant to it than another evaluator’s would be, so there is nothing to refuse.

A score that cannot answer these questions cannot gate a release. A result written before this record existed returns empty values here; it is still readable, and a release gate refuses it for the same reason it refuses a result with no evaluator version. An unexplained number is not evidence.

Quality feedback recorded before evaluators existed

Some results predate the evaluator model on this page. They were recorded when quality feedback was just a number and a label attached to a run — a reviewer’s rating, or an automatic judgment with no versioned configuration behind it. They are returned by the same endpoint, so nothing you recorded earlier disappeared, and two fields mark them:

Field
provenance.legacy_source	Non-empty, and it is the original kind of the judgment (for example human). Every result recorded since evaluators existed leaves this empty.
provenance.evaluator.version	0. Every other result carries an exact, non-zero version.

A result marked this way is readable and never usable as evidence. A release gate refuses it, and an SLO rule cannot select it — both bind an exact evaluator version, and 0 is reserved so that no exact reference can ever match one. That is deliberate: the whole point of an exact version is that the same question is being asked each time, and these results cannot say what question was asked.

Their evaluator_id reads as legacy/<author>/<metric>. It is not an evaluator you can fetch, version or run — it exists so that two people’s separate ratings of the same run stay two separate results.

One caveat worth stating plainly: occurred_at on these results is when the feedback was recorded, not when the run happened, because the original record never captured the run’s own time. It is close enough to find a result by its subject and wrong for a trend — which is another reason nothing computes one from them.

Finding the release where a metric changed

Ask for one metric produced by one exact evaluator version. Keeping the version exact is essential: otherwise changing the judge can look like changing the agent.

anectico evals release-series \
  --evaluator task_completion --evaluator-version 1 --metric task_completion \
  --start 2026-08-01T00:00:00Z --end 2026-09-01T00:00:00Z -o json

The response returns declared releases oldest first. Each point has its sample count, mean, first and last observed run time, and—after the first point—the arithmetic change from the previous release. That change is deliberately neutral: a negative completion delta may be a regression, while a negative latency delta may be an improvement.

If every release name is semantic versioning, semantic precedence orders the series. If even one is an arbitrary label, the whole series uses first-observed run time, with the label only breaking a tie; ordering_strategy tells you which rule was applied. This whole-catalog fallback avoids a contradictory order made from comparing some pairs by version and other pairs by time.

Runs that explicitly declared no release are returned as a separate unreleased aggregate with unreleased_present: true. They are never dropped and are never invented as a release with a place in the order. The equivalent REST request is:

curl "https://app.anectico.com/api/v1/evaluation/release-series?evaluator_id=task_completion&evaluator_version=1&metric_name=task_completion&start=2026-08-01T00:00:00Z&end=2026-09-01T00:00:00Z" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

Human judgments

An evaluator’s result answers “what did the judge say”. A human judgment answers “what did a person think”, and the two are recorded separately and read separately.

# Record or replace YOUR judgment of one run.
curl -X POST "https://app.anectico.com/api/v1/agent-runs/$RUN_ID/annotations" \
  -H "Authorization: Bearer $ANECTICO_USER_TOKEN" -H 'Content-Type: application/json' \
  -d '{"metric_name":"helpfulness","numeric_value":0.8,"label":"good","comment":"answered the actual question"}'

# Read every reviewer's judgments of one run.
curl "https://app.anectico.com/api/v1/agent-runs/$RUN_ID/annotations" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

# Read judgments for many runs at once, for a badge grid.
curl "https://app.anectico.com/api/v1/evaluation/annotations?subject_refs=$RUN_A&subject_refs=$RUN_B" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

Five things about this surface are deliberate and worth knowing before you use it.

You are always the author. There is no author field to send: the judgment is attributed to whoever the credential belongs to. A credential with no person behind it — an API key — cannot record one at all, and is told so rather than filing the judgment under nobody. Re-judging the same run on the same metric replaces your judgment and nobody else’s.

An absent number is not a zero. numeric_value is optional, and leaving it out records “not scored” — which is what a thumbs-down with no number means. Sending 0 records a score of zero, which is a different statement about the run. At least one of numeric_value and label is required: a judgment that states neither says nothing.

metric_name is free text. It is not a reference to a registered evaluator, because a person may judge a dimension nothing automated measures yet — and requiring an evaluator first would make the human loop wait on the thing it exists to calibrate. It is lower-cased and trimmed.

A judgment is never evidence. Release gates and SLO objectives read evaluator results and cannot reach human judgments at all. That is the point of keeping them apart: comparing what a person said with what a judge said is only a well-posed question while the two are recorded separately.

Reading one needs no content scope, and neither does writing one. A reviewer’s comment is their own sentence about a run rather than a restatement of what the run contained, so it is returned to anyone who may read judgments. And recording a judgment needs only the write scope — nothing here checks whether the run you named exists, so pointing at a run you may not read tells you nothing about it.

Reading judgments accepts metric_name, author_id and limit. author_id narrows the answer and is not a permission boundary: everyone who may read judgments in a project sees every colleague’s, because a rating nobody else can see cannot be disagreed with. The batch read takes subject_refs on the same terms as results — at most 200 runs, refused rather than shortened, limit bounding each run’s share — and its response carries the same exact truncated and truncated_subject_refs.

Historical note: quality feedback recorded before this surface existed is in results, not here. See the section above — nothing was lost, but a report that compares people with judges has to read both.

Calibration — how far a judge agrees with your people

A model judge’s score is a claim, and until somebody checks it against a person it is a claim nobody has tested. Calibration measures that agreement over subjects you have both judged, and records it on the exact evaluator version.

curl -X POST \
  "https://app.anectico.com/api/v1/evaluation/evaluators/$EVALUATOR_ID/versions/4/calibrations" \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"subject_refs":["'"$RUN_A"'","'"$RUN_B"'"],"min_labels":20}'

curl "https://app.anectico.com/api/v1/evaluation/evaluators/$EVALUATOR_ID/calibrations" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

Or from the CLI:

anectico evals evaluators calibrate quality 4 --min-labels 20 --subject run-a --subject run-b
anectico evals evaluators calibrations quality

Which statistic you get depends on your labels. Where reviewers recorded a number, agreement is Lin’s concordance coefficient between their ratings and the judge’s. Where they recorded a verdict, it is Cohen’s kappa — chance-corrected agreement, not the raw match rate. That distinction is the point: a judge answering “good” to everything matches an 80%-good corpus 80% of the time and has established nothing. The raw rate is still reported as agreement_rate because it is a fact; it is never the calibration.

The same reasoning is why the numeric statistic measures agreement rather than correlation. A judge that rates every response exactly half as highly as your reviewers do tracks them perfectly — plain correlation scores that a flawless 1.0, and it would clear any threshold you set. But its numbers are on a scale nobody agreed to, so every absolute threshold you write against them is wrong. Concordance penalises that offset: the same judge scores about 0.57. If your judge’s agreement drops after you upgrade, this is the likely reason, and the fix is to correct the judge’s scale rather than to lower the gate.

Every state except one carries no number, and that is deliberate — an agreement figure nobody measured must never read as one.

State	What it means
CALIBRATED	Agreement was computed. This is the only state carrying agreement.
UNCALIBRATED	A model judge nobody has compared to a person.
INSUFFICIENT_LABELS	Fewer comparable judgments than min_labels. Not a weak number — no number.
UNDEFINED_STATISTIC	Enough judgments, and a statistic that cannot be computed from them: everyone used one and the same verdict, or both sides gave the identical unvarying rating. Note that only both sides being constant is undefined — if your reviewers varied and the judge answered the same number every time, that is measurable disagreement, and it scores zero rather than landing here.
NOT_APPLICABLE	The evaluator is not a model judge. A deterministic or reference-trajectory evaluator is reproducible from its own definition.

min_labels is required and zero is refused, on the same terms as a gate’s min_samples: a calibration you will act on over four judgments is one to ask for out loud.

A record is immutable per (version, label set). Recomputing over unchanged judgments returns the existing record with created: false; a reviewer changing their mind, or the judge being re-run, produces a new record beside it, so the history shows whether agreement moved when you changed the rubric. Each record names the exact judgment ids it was computed from.

Judgments about subjects the judge never scored are excluded and counted rather than treated as disagreements — a sampling gap is not a bad judge — and a judge that could not produce a verdict at all is excluded on the same terms. pair_count is what was compared and excluded_count is what was not.

Calibration reads your judgments and the judge’s results and does arithmetic. It calls no model and spends no evaluation budget.

The posture appears in three places, so a number’s credibility travels with it: on the evaluator version, on every result that version produced, and as the rule a release gate applies before it will treat a judge’s score as evidence.

The explanation is treated as content

An explanation describes the run it scored, so it can restate what the run contained. Two independent controls decide whether you receive it, and both have to permit it:

• Your credential must hold agents:content:read. Without that scope the field is empty.
• Your project’s content policy must allow model transcripts and tool arguments to be read. An explanation is a single piece of prose that may quote either, so it is disclosed only when both are allowed — no scope overrides that, and the same rule governs whether the reasoning is written down in the first place. If your project’s policy cannot be read at the moment, the field is empty until it can be; nothing is lost, and reading again later returns it.

The score and the label are not withheld from anyone with read access, whatever the policy says. They are the answer, and they are what quality badges are built from — which is also why a label is held to a short verdict word: a model that answers with a paragraph gets scored instead, and its number is unaffected.

One exception, and it is narrow: quality feedback recorded before this surface existed and migrated into results carries the reviewer’s own comment in explanation. A comment a colleague typed is served on the same terms as a judgment’s comment field below — to anyone who may read judgments — because it is a person’s sentence about a run rather than a machine’s restatement of one. Every migrated machine comment stays behind the two controls above.

Datasets — a frozen set of cases

A dataset is a named collection of cases you promoted out of real recorded runs. Its metadata is editable; its versions are not. Promoting cases seals a new version containing everything the current version had plus the new cases, and removing cases seals a new version without them — so a score that cites a version keeps meaning exactly what it meant when it was recorded.

Only a human-reviewed production run can be promoted. Record a judgment first, then pass the returned annotation id with that exact run:

REVIEW_ID=$(curl -sS -X POST \
  "https://app.anectico.com/api/v1/agent-runs/$RUN_ID/annotations" \
  -H "Authorization: Bearer $ANECTICO_USER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"metric_name":"dataset_eligibility","label":"reviewed"}' |
  jq -r '.annotation.annotation_id')

curl -X POST \
  "https://app.anectico.com/api/v1/evaluation/datasets/regression_set/promotions" \
  -H "Authorization: Bearer $ANECTICO_USER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"subjects":[{"subject_kind":"SUBJECT_KIND_AGENT_RUN","subject_ref":"'"$RUN_ID"'","review_annotation_id":"'"$REVIEW_ID"'"}]}'

The review must belong to the same project and name the same run. The frozen item records the review id, authenticated reviewer, and review time. Promotion refuses a missing or mismatched review before it freezes or stores anything; holding evals:write does not itself count as a review.

Each case’s id is a digest of its own content, which has two consequences worth knowing:

• Promoting the same run twice gives you one case, not two.
• A case keeps the same id as it is carried into later versions, so “this case scored 0.4 in v1 and 0.9 in v3” is a question you can ask.

A version number is always exact. Reading a version’s cases requires you to name it; there is no “latest”, because a set of cases that can change under a comparison is not a set of cases.

Nothing is ever deleted. A version cited by a stored score cannot be removed, and neither can a dataset that has one.

What a case contains. A frozen, self-contained projection of the run — enough to score it again later without the original run still existing. If your project’s content settings did not permit a class of content to be copied, the case records which classes were withheld rather than being quietly short: a case frozen without its content would score differently for ever, and you would have no way to tell it from one that never had any.

Replaying one case in a sandbox

A replay re-executes the recorded decision sequence of one frozen case inside a sandbox that cannot reach anything outside this platform. It stores nothing. A replay is a pure function of a frozen case and the plan you send, and both are fixed — so asking again reproduces the same answer byte for byte, and you never have to keep one in order to get it back.

The plan is yours to declare

Nothing here knows what your tools do, so the plan is where you say it. Each policy names a tool by the name the recording carries and classifies it by what the call does outside the sandbox:

Effect	Meaning	What the sandbox does
SANDBOX_EFFECT_NONE	a read-only call	replays the recorded answer, if there is one
SANDBOX_EFFECT_INTERNAL	writes inside this platform	answers with a marked stub
SANDBOX_EFFECT_EXTERNAL	leaves this platform	answers with a marked stub

default_effect covers any tool no policy names. Unset means external, and so does a call the recording does not name — not even an explicitly open default reaches that one. The recoverable mistake is stubbing a read; the unrecoverable one is letting a write through because nobody classified it.

A tool declared twice is refused rather than resolved: two policies for one tool is two answers to one question, and whichever won would depend on the order you happened to list them in. An effect value the contract does not declare is refused too, with the accepted values named — it is not quietly read as unset, because “you did not classify this tool” and “we could not read what you wrote” must not produce the same run.

Sending no plan at all is a valid request: everything is external, so everything is stubbed.

What comes back

The replayed run, encoded exactly as a frozen case is, so you can diff the two directly — plus a record of what the sandbox did:

• a digest of the plan, which identifies the sandbox that produced this answer; two plans that behave identically carry one digest;
• how many steps and tool calls there were, and how many were replayed, stubbed, or had nothing recorded to replay;
• the sorted set of tools the sandbox answered for — the actionable half, because a case that depends on those cannot be reproduced anywhere the sandbox does not reach;
• one entry per step, in order, so the counts are checkable rather than trusted.

Each step carries one of four dispositions: carried (not a tool call — replayed as recorded), recorded (a read-only call whose recorded answer was replayed), unrecorded (a read-only call with no recorded answer — the absence is carried through rather than filled), and stubbed (the sandbox answered because the call had an effect it is not permitted to have).

A stubbed call is stubbed, not skipped: skipping it would change the sequence and the replay would then be of a different agent. Any recorded answer on an effectful call is discarded, so a stubbed run can never be mistaken for one that really ran.

What a replay can and cannot tell you today

A recorded run carries the tools an agent called and the order it called them in. It does not carry what those calls returned. So on today’s recordings every read-only tool call comes back unrecorded, and every effectful one comes back stubbed — the replay is an honest re-execution of the control flow, not of the tool exchanges, and the record says which it was for every step. That is the whole of what can be known from a recording that holds no tool results, and reporting it as anything more would be a number you could not check.

The recorded outcome of the run is left exactly as it was. Overwriting it would silently change what every evaluator that reads an outcome would say, and the replay would then be measuring the sandbox.

Experiments — comparing two configurations on the same cases

An experiment compares two or more configurations of your agent on ONE frozen dataset version, scored by the same evaluators at the same exact versions. One configuration is the baseline; every other is compared against it.

One experiment can compare the five operational dimensions together: quality, safety, reliability, latency, and cost. Declare one exact evaluator version for each dimension. The comparison output is then one paired result per candidate and evaluator; a dimension appears only when both arms produced real scores for the same cases.

Because each configuration is measured on the same case, the comparison is paired — which removes the case-to-case variation that is usually far larger than the difference you are looking for. Comparing two configurations’ averages over different cases asks a question that one hard case can answer wrongly.

What you supply. The platform records what your agent did; it does not run your agent. So for each case you tell it which recorded run each configuration produced — a trial. Leaving the run out means “score the frozen case itself”, which is the fully repeatable arm: frozen cases cannot change and need no original run to still exist.

What you get back, per configuration and per evaluator:

Field	Meaning
paired_items	How many cases were scored under both configurations. Only these are compared.
baseline_only_items, candidate_only_items	Cases one side has and the other does not. Excluded, and reported — a comparison resting on a third of your dataset looks exactly like one resting on all of it.
mean_difference	The average per-case difference.
ci_low, ci_high, confidence_level, ci_method	The interval for that average, and the procedure that produced it.
p_value, p_value_method	Before the multiple-comparison correction.
adjusted_p_value, significant	After it, at the alpha you declared.
small_sample	Fewer than eight paired cases — the interval is reported, and is not worth much.
degenerate	Every case differed by the same amount, so the interval has zero width for an arithmetic reason.
unevaluated (on the response)	Declared trials with no score yet.

You must say how multiple comparisons are handled

Comparing several configurations, or several evaluators, makes it more likely that something looks significant by chance. So an experiment cannot be created without saying what to do about that:

correction	What it controls
NONE	Nothing. The p-values are uncorrected, and you have said so deliberately.
BONFERRONI	The chance of any false positive across the whole comparison.
HOLM	The same thing, less conservatively. The better of the two if you want that guarantee.
BENJAMINI_HOCHBERG	The expected proportion of false positives among the findings. A weaker, different guarantee.

There is no default. Every comparison you read back states the method, the alpha, how many comparisons it covered, and — in words — what it actually controls, so a number is never quotable without what it claims.

The correction is fixed when the experiment is created and cannot be changed afterwards, along with the dataset version, the evaluator versions and the candidate set. A choice made after the numbers are in is not a correction.

Running one

Running an experiment scores the outstanding cases and is deliberately resumable: one call does a bounded amount of work and tells you how many cases remain. Call it again while remaining is above zero. Repeating a call never re-scores a case that is already done, and never charges you twice for one.

Repeatability

Re-running an experiment over frozen cases with the deterministic evaluators reproduces every number exactly. Two things are outside that promise, and are worth knowing before you rely on it: a trial pointing at a live recorded run depends on that run still being there and unchanged, and an LLM-judge evaluator varies from run to run on its own.

An experiment is limited to 8 configurations, 8 evaluators, and 20 000 total case-scores. The limit is checked when you create it, so an experiment too large to compare is refused up front rather than after you have paid for the scoring.

Release gates — deciding whether a release may ship

A release gate binds a candidate release to evaluation evidence and produces a pass or a refusal with the reason, kept for ever. It is the thing that turns a comparison into a decision.

A gate is fixed when you create it — its evidence, its objectives, its confidence level. A changed objective is a new gate, because a gate whose bar can move under it is not a gate.

The two kinds of evidence, which are different designs

evidence_kind	What it compares	Paired?
EXPERIMENT	One arm of a declared experiment against that experiment’s own declared baseline, on the same frozen cases.	Yes — the same case is scored under both configurations.
CANARY	Two windows of live traffic that you declare: one before the candidate shipped, one after.	No. The runs in the two windows are different runs, so the interval carries the full case-to-case variance.

You do not name the comparison arm for experiment evidence — the experiment’s own declared baseline is used. Choosing the arm afterwards is how a flattering pair gets picked.

For a canary, the window boundary is your declaration of when the candidate shipped. Everything else that changed at that boundary is confounded with the release, and every stored canary comparison says so in words alongside its numbers, because a narrow interval and a causal claim look identical on a dashboard.

Objectives

Each objective names an evaluator at an exact version, the metric it must produce, what to compare, and the line to clear:

basis	The number compared against the threshold
CANDIDATE_MEAN	The candidate’s own average. An absolute floor or ceiling; needs no baseline.
DIFFERENCE_MEAN	The point estimate of candidate minus baseline.
DIFFERENCE_INTERVAL	The conservative bound of the interval around that difference — the low bound for a floor, the high bound for a ceiling. The strong form: the whole interval must clear the line, so a candidate that passes on noise does not.

min_samples is required and zero is refused. A gate that will pass on a single observation is something you have to ask for out loud.

min_judge_agreement is optional and puts a floor on how far the objective’s evaluator must agree with human judgments — see Calibration. It is not a way to opt out: an objective bound to a model judge with no calibration at all is inadmissible whatever this field says, and a gate whose only evidence comes from that judge answers INADMISSIBLE rather than passing. Leaving it at zero asks only that a calibration exist.

The rule applies to model judges alone. A deterministic or reference-trajectory evaluator is reproducible from its own definition and is calibrated against nobody, so nothing is asked of it.

Calibration is resolved when the gate is evaluated, not when it is created. A gate written before its judge was calibrated passes once the judge is; a judge whose calibration was superseded by an edited judgment set stops passing until it is recomputed.

An objective is checked against your evaluator registry when the gate is created, including that the evaluator you named actually produces the metric you named — so a gate that could only ever measure nothing is refused before it exists rather than after a release waited on it.

Verdicts: only PASS passes

Verdict	What it means	Where to look
PASS	Every objective was measured on admissible evidence and met.	—
FAIL	At least one objective was measured and missed.	The objective’s numbers.
PENDING	Created, never evaluated. Not a pass.	Evaluate it.
NO_EVIDENCE	An objective had nothing admissible to measure, or less than it declared it needs.	Your sampling — nothing ran.
INADMISSIBLE	Results exist for an objective and none of them may be used — incomplete provenance, an execution record that cannot say how the score was produced, a job that did not complete, or a model judge that has not been calibrated.	Your evaluator reference — things ran, and cannot be used.
RELEASE_MISMATCH	Every objective met, but the candidate runs did not all observably declare the release the gate names. Not a pass.	The decision’s release_binding: it distinguishes another release, no declared release, partial instrumentation, and an unmeasured subject shape.

The last two are kept apart deliberately: “nothing ran” and “what ran cannot be used” are different problems with different fixes, and one empty answer cannot tell them apart.

Per objective you also get observed_present, which is the only way to tell “the value was 0” from “there was no value”. A score of 0.0 is a legitimate measurement, so the number can never be that discriminator — never infer absence from observed_value. inadmissible_samples counts results that exist and may not be used; they are reported rather than filtered away, for the same reason.

Evaluating one

The first call reaches a verdict over the evidence available then and appends one permanent decision. You may supply an idempotency_key; repeating the same gate, caller and key returns that existing decision with replayed: true — even if new results arrived between the calls. If you omit the key, Anectico derives one stable default decision identity from the organization, project, gate and authenticated caller. Repeating the omitted-key call therefore replays that first decision too; it never silently regains at-least-once append behavior. The original decided_at, sample counts and outcomes show which evidence snapshot you received.

Use a new explicit key when you deliberately want to evaluate newly arrived evidence. That creates a new decision beside the first, and the whole history stays readable. Evidence is intentionally not part of the retry identity: otherwise a response-loss retry could silently become a contradictory second decision just because a result landed in between.

A gate never runs an evaluator and never spends your evaluation budget. That is deliberate — a gate that produced its own evidence on demand could never report NO_EVIDENCE, which is the one answer it exists to make reachable. Score first, then gate.

If the evidence store cannot be read, you get an error and nothing is stored. “We could not look” is never recorded as “we looked and found nothing”.

A release label is a claim that the evidence must prove

release is the label you declare when creating the gate. Evaluation reconciles it against the release actually carried by every admissible candidate run, deduplicated across objectives; baseline runs are expected to belong to the earlier comparison arm and do not participate in this check.

Every decision stores release_binding with the observed release names, released, unreleased, and unmeasured subject counts, a reason, and one of MATCHED, MISMATCH, UNDECLARED, PARTIAL, UNMEASURED, or NO_EVIDENCE. Only MATCHED may accompany PASS. If the objective numbers meet but the label does not, the verdict is RELEASE_MISMATCH rather than a green decision carrying a warning. If an objective already missed, the verdict remains the more direct FAIL, but the same binding discrepancy is still recorded.

“No release declared” is a first-class answer. A gate evaluated over those runs returns binding UNDECLARED and cannot pass; it does not error, silently discard those runs, or pretend they belong to the gate’s label.

Permissions

Scope	Grants
evals:read	List and read evaluators, sampling rules, the budget, results, the ordered observed-release series, datasets, experiments — including an experiment’s comparison — and release gates with their full decision history. Also replay one frozen case in a sandbox: it stores nothing, and it returns strictly less than reading the case does.
evals:write	Create evaluators and versions, manage sampling rules, set the budget, evaluate on demand, create datasets and promote reviewed cases or remove cases, create experiments and their trials, run an experiment, and create a release gate or evaluate one. Evaluating a gate is a write because it permanently records whether a release is admitted or held — a key minted only to look at gates must not be able to create it.
agents:content:read	Additionally see the explanation field on results, when the project’s content policy also allows it.
scores:write	Record or replace your own human judgment of a run. It is a member capability rather than an administrative one, deliberately: rating a run is something anyone reviewing the product does.
scores:read	Read the human judgments recorded for runs.

A project-scoped key can only reach its own project; a request naming another project returns 404 Not Found, the same answer an evaluator that does not exist returns.

Endpoints

Method	Path
GET	/api/v1/evaluation/evaluators
POST	/api/v1/evaluation/evaluators
GET	/api/v1/evaluation/evaluators/{evaluatorId}
POST	/api/v1/evaluation/evaluators/{evaluatorId}/versions
GET	/api/v1/evaluation/evaluators/{evaluatorId}/versions/{version}
GET	/api/v1/evaluation/sampling-rules
PUT	/api/v1/evaluation/sampling-rules
DELETE	/api/v1/evaluation/sampling-rules/{ruleId}
GET	/api/v1/evaluation/budget
PUT	/api/v1/evaluation/budget
POST	/api/v1/evaluation/evaluate
GET	/api/v1/evaluation/results
GET	/api/v1/evaluation/release-series
POST	/api/v1/agent-runs/{runId}/annotations
GET	/api/v1/agent-runs/{runId}/annotations
GET	/api/v1/evaluation/annotations
GET	/api/v1/evaluation/datasets
POST	/api/v1/evaluation/datasets
GET	/api/v1/evaluation/datasets/{datasetId}
GET	/api/v1/evaluation/datasets/{datasetId}/versions/{version}/items
POST	/api/v1/evaluation/datasets/{datasetId}/promotions
POST	/api/v1/evaluation/datasets/{datasetId}/removals
POST	/api/v1/evaluation/datasets/{datasetId}/versions/{version}/items/{itemId}/replays
GET	/api/v1/evaluation/experiments
POST	/api/v1/evaluation/experiments
GET	/api/v1/evaluation/experiments/{experimentId}
POST	/api/v1/evaluation/experiments/{experimentId}/trials
POST	/api/v1/evaluation/experiments/{experimentId}/run
GET	/api/v1/evaluation/experiments/{experimentId}/comparison
GET	/api/v1/evaluation/gates
POST	/api/v1/evaluation/gates
GET	/api/v1/evaluation/gates/{gateId}
POST	/api/v1/evaluation/gates/{gateId}/evaluations

Promoting cases and removing them are both POSTs that seal a new version, rather than a PUT or a DELETE on an existing one — nothing about a sealed version ever changes. Evaluating a gate is a POST onto /evaluations for the same reason: a new explicit idempotency key appends a decision rather than editing one. Its optional JSON body is { "idempotency_key": "your-stable-decision-key" }; an empty body selects the stable default decision intent.

Replaying a case is a POST for a different reason: it carries the sandbox plan, not because anything is written. A replay stores nothing and reproduces its answer every time, which is why it needs only evals:read.

GET /api/v1/evaluation/gates accepts an optional ?release= to narrow to one release label.

All accept an optional ?project_id= query parameter; a project-scoped key is pinned to its own project regardless. /budget is the exception: it is org-wide and reads no project.

From the command line, and from an agent

The API is the complete surface. The CLI mirrors it; MCP deliberately exposes a narrower subset.

The CLI mirrors the endpoints one-for-one under anectico evals:

anectico evals datasets list
anectico evals datasets items refunds --version 3
anectico evals experiments comparison exp-refunds
anectico evals release-series --evaluator task_completion --evaluator-version 1 --metric task_completion
anectico evals gates list --release 2026.08.3
anectico evals gates evaluate gate-1 --idempotency-key release-2026-08-26
anectico evals datasets replay refunds case-17 --version 3 --file plan.json
anectico evals annotations add $RUN_ID --metric helpfulness --label good --comment "answered the question"
anectico evals annotations list $RUN_A $RUN_B $RUN_C

anectico evals annotations add omits the number unless you pass --value, so a --label-only judgment is recorded as “not scored” rather than as a zero. list names every run you give it in one request.

Structured declarations — an experiment, a gate, a promotion — are supplied as JSON with --body or --file (--file - reads standard input), and are sent to the API exactly as you wrote them. Reads print a table on a terminal and JSON when piped; --output json forces it. anectico evals also covers evaluators, rules, budget, evaluate and results. Run anectico evals --help, or see the CLI reference.

An MCP agent can read this surface through the read gateway: list_eval_datasets, get_eval_dataset, list_eval_experiments, get_eval_experiment, compare_eval_experiment, list_release_gates and get_release_gate, all under evals:read. They are reached through list_read_actions and execute_read_action rather than appearing as tools of their own.

Three writes are available through list_write_actions and execute_internal_action. create_release_gate and evaluate_release_gate require evals:write; both preview first because gates and decisions are permanent and have no delete, void, or correction path. Creation requires a caller-owned gate_id, so its retry is refused by the unique ID instead of minting a duplicate. Evaluation through MCP requires idempotency_key; reusing it safely returns the original decision, while a new key asks for a fresh evidence snapshot. The API and CLI also accept omission and use the server-derived default intent described above. Both actions are gatewayed and add no first-class host tool.

submit_annotation, under scores:write, is also reached through list_write_actions and execute_internal_action. It previews first and applies on a second call with the returned confirm_token, because re-judging replaces the caller’s previous judgment of that run. Its receipt reports the metric, the number and the label; it does not repeat the rationale back.

Three things an agent cannot do over MCP, deliberately: it cannot read a dataset version’s frozen cases — those can carry recorded content, and the read that returns them is available over the API and the CLI only — it cannot read the ordered release series yet, and it cannot change the evaluation configuration. Creating a dataset, promoting cases, and declaring or running an experiment remain API and CLI operations.