Evaluation results and human judgments

Start with your first evaluation for a runnable workflow. See API and permissions for endpoint and credential requirements.

Reading results

curl "https://app.anectico.com/api/v1/evaluation/results?metric_name=task_completion&start=2026-08-01T00:00:00Z&limit=100" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

Filters: metric_name, evaluator_id, subject_ref, subject_refs, mode, start, end, limit. Omitting mode returns every mode; each result states its own, so comparing across them is something you do deliberately rather than by accident.

subject_refs asks about many runs at once — repeat it (?subject_refs=a&subject_refs=b) or pass a comma list (?subject_refs=a,b); both work, and it combines with the singular subject_ref as one set. Use it to draw a quality badge for a whole page of runs in one request instead of one request per row. At most 200 runs may be named, counting the two parameters together and counting a run named twice once; a longer list is refused rather than shortened, because a silently trimmed list comes back looking complete while whole runs are missing from it and you would then draw "not evaluated" for a run that was.

limit means different things on the two shapes, and the difference exists for that same reason:

• Naming at most one run, it is the page size — default 100, maximum 500.
• Naming several, it bounds each run's share of the page — default and maximum 20 — and the page is that share times the runs you named. A single total over many runs would be spent by whichever runs were evaluated most recently, and the runs at the far end would come back empty behind a 200 OK.

The response also carries truncated and truncated_subject_refs. truncated is true when a cap cut the page and there are more results matching your filters; it is exact rather than a guess from a full-looking page, so a page of exactly the cap still tells you truthfully whether anything was left behind. truncated_subject_refs names which runs were cut — one busy run sets the page-level flag for all 200, so if you are drawing a badge per run, treat exactly the runs listed there as unknown rather than as not evaluated, and read those again with a narrower window or a tighter metric_name. A run you asked about that is not in that list returned everything stored for it.

This read is a history, not a current value. Every evaluator version a run was measured under keeps its own result, on purpose: that is what makes "this run scored 0.2 under version 3 and 0.9 under version 4" answerable at all, and it is what a release gate compares. So a badge or an average built by summing everything this endpoint returns counts a run's old verdicts alongside its current ones — a run that was fixed reads as half-fixed for ever. When you want the current picture, keep the highest provenance.evaluator.version for each (provenance.evaluator.evaluator_id, metric_name) pair and summarize those.

Every result carries:

Field	Meaning
numeric_value, label	The score.
provenance.evaluator	The evaluator id and the exact version.
provenance.mode	How it was produced.
provenance.subject_kind, provenance.subject_ref	What was scored.
subject_release, release_declaration	The release observed on that subject, or the explicit answer that it declared none.
occurred_at	When the run happened.
evaluated_at	When the score was produced.
explanation	The evaluator's reasoning.
provenance.execution	What the evaluation actually did — see below.

occurred_at and evaluated_at are separate on purpose. They differ by however long sampling and queueing took, and a trend plotted on the wrong one attributes a regression to the day it was measured rather than the day it shipped.

What the evaluation actually did

provenance.execution records how each score came to exist: why the subject was picked, what the evaluation cost, which model answered, and whether the job finished. It is stored on the result rather than looked up later, so it stays true even after the rule that selected the subject is edited or the evaluator is retired.

Field	Meaning
sampling	SAMPLED (an online rule picked this subject), NOT_SAMPLED (you asked for this subject by name) or WHOLE_DATASET (a batch run measured every case).
sampling_rule_id, sampling_rate	The rule that fired and the rate it was applying at that moment. Present only for SAMPLED.
cost_state	CHARGED, FREE, UNPRICED or UNREPORTED.
charged_cost_nano_usd	The charge, in billionths of a US dollar. Meaningful only when cost_state is CHARGED.
input_tokens, output_tokens	What the model call consumed. Zero for an evaluator that calls no model.
judge_model_state	RECORDED, NOT_APPLICABLE (not a model judge) or UNREPORTED.
judge_model	The model that actually answered. Present only for RECORDED.
job_status	COMPLETED, PARTIAL, FAILED or CANCELLED.
job_failure_class	Names what was missing, for every status except COMPLETED.

Read cost_state before the charge. A charge of zero has two unrelated meanings — an evaluator that calls no model and costs nothing, and a model whose price is unknown — and the number alone cannot tell you which. UNPRICED is the second: tokens were spent, the price is not known, and the zero is an absence rather than a bill. It is never reported as a charge of zero.

judge_model is not the model the evaluator was configured with. The evaluator version records what was requested; this records what answered, and the two differ whenever a provider resolves a floating model name to a dated one or falls back to another model. A score attributed to the requested model would be attributed to a model that never saw the run.

PARTIAL still carries a real number. It means the job finished and something else about it did not: your content policy kept the judge's reasoning out of stored results, or the score was delivered by a retry after an earlier attempt did not finish. The score itself is exactly the one the evaluation produced, and a release gate accepts it. FAILED and CANCELLED mean the score's storage was still owed when the status was written — you can only see them on the immediate reply to a one-off evaluation, never on a stored result — and a release gate refuses them.

An objective is an evaluator at a version, and another version's results are not its evidence. A gate bound to version 2 of an evaluator whose results were all produced by version 1 reads NO_EVIDENCE — "nothing has scored what you bound" — never INADMISSIBLE: those rows are no more relevant to it than another evaluator's would be, so there is nothing to refuse.

A score that cannot answer these questions cannot gate a release. A result written before this record existed returns empty values here; it is still readable, and a release gate refuses it for the same reason it refuses a result with no evaluator version. An unexplained number is not evidence.

Quality feedback recorded before evaluators existed

Some results predate the evaluator model on this page. They were recorded when quality feedback was just a number and a label attached to a run — a reviewer's rating, or an automatic judgment with no versioned configuration behind it. They are returned by the same endpoint, so nothing you recorded earlier disappeared, and two fields mark them:

Field	Meaning
provenance.legacy_source	Non-empty, and it is the original kind of the judgment (for example human). Every result recorded since evaluators existed leaves this empty.
provenance.evaluator.version	0. Every other result carries an exact, non-zero version.

A result marked this way is readable and never usable as evidence. A release gate refuses it, and an SLO rule cannot select it — both bind an exact evaluator version, and 0 is reserved so that no exact reference can ever match one. That is deliberate: the whole point of an exact version is that the same question is being asked each time, and these results cannot say what question was asked.

Their evaluator_id reads as legacy/<author>/<metric>. It is not an evaluator you can fetch, version or run — it exists so that two people's separate ratings of the same run stay two separate results.

One caveat worth stating plainly: occurred_at on these results is when the feedback was recorded, not when the run happened, because the original record never captured the run's own time. It is close enough to find a result by its subject and wrong for a trend — which is another reason nothing computes one from them.

Finding the release where a metric changed

Ask for one metric produced by one exact evaluator version. Keeping the version exact is essential: otherwise changing the judge can look like changing the agent.

anectico evals release-series \
  --evaluator task_completion --evaluator-version 1 --metric task_completion \
  --start 2026-08-01T00:00:00Z --end 2026-09-01T00:00:00Z -o json

The response returns declared releases oldest first. Each point has its sample count, mean, first and last observed run time, and—after the first point—the arithmetic change from the previous release. That change is deliberately neutral: a negative completion delta may be a regression, while a negative latency delta may be an improvement.

If every release name is semantic versioning, semantic precedence orders the series. If even one is an arbitrary label, the whole series uses first-observed run time, with the label only breaking a tie; ordering_strategy tells you which rule was applied. This whole-catalog fallback avoids a contradictory order made from comparing some pairs by version and other pairs by time.

Runs that explicitly declared no release are returned as a separate unreleased aggregate with unreleased_present: true. They are never dropped and are never invented as a release with a place in the order. The equivalent REST request is:

curl "https://app.anectico.com/api/v1/evaluation/release-series?evaluator_id=task_completion&evaluator_version=1&metric_name=task_completion&start=2026-08-01T00:00:00Z&end=2026-09-01T00:00:00Z" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

Human judgments

An evaluator's result answers "what did the judge say". A human judgment answers "what did a person think", and the two are recorded separately and read separately.

For a terminal workflow, use your member session rather than a machine key:

unset ANECTICO_API_KEY
anectico login
anectico evals annotations add "$RUN_ID" --metric helpfulness --value 0.8 \
  --label good --comment "answered the actual question"

In the REST examples below, ANECTICO_USER_TOKEN means a short-lived access token from an authenticated user OAuth session, with scores:write for annotation writes and evals:write for dataset promotion. It is not an an_... API key or a refresh token. If you do not already manage user OAuth tokens in an integration, use the signed-in CLI or dashboard instead of extracting a browser cookie. Keep user tokens out of source control, logs, and agent prompts.

# Record or replace YOUR judgment of one run.
curl -X POST "https://app.anectico.com/api/v1/agent-runs/$RUN_ID/annotations" \
  -H "Authorization: Bearer $ANECTICO_USER_TOKEN" -H 'Content-Type: application/json' \
  -d '{"metric_name":"helpfulness","numeric_value":0.8,"label":"good","comment":"answered the actual question"}'

# Read every reviewer's judgments of one run.
curl "https://app.anectico.com/api/v1/agent-runs/$RUN_ID/annotations" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

# Read judgments for many runs at once, for a badge grid.
curl "https://app.anectico.com/api/v1/evaluation/annotations?subject_refs=$RUN_A&subject_refs=$RUN_B" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

Five things about this surface are deliberate and worth knowing before you use it.

You are always the author. There is no author field to send: the judgment is attributed to whoever the credential belongs to. A credential with no person behind it — an API key — cannot record one at all, and is told so rather than filing the judgment under nobody. Re-judging the same run on the same metric replaces your judgment and nobody else's.

An absent number is not a zero. numeric_value is optional, and leaving it out records "not scored" — which is what a thumbs-down with no number means. Sending 0 records a score of zero, which is a different statement about the run. At least one of numeric_value and label is required: a judgment that states neither says nothing.

metric_name is free text. It is not a reference to a registered evaluator, because a person may judge a dimension nothing automated measures yet — and requiring an evaluator first would make the human loop wait on the thing it exists to calibrate. It is lower-cased and trimmed.

A judgment is never evidence. Release gates and SLO objectives read evaluator results and cannot reach human judgments at all. That is the point of keeping them apart: comparing what a person said with what a judge said is only a well-posed question while the two are recorded separately.

Reading one needs no content scope, and neither does writing one. A reviewer's comment is their own sentence about a run rather than a restatement of what the run contained, so it is returned to anyone who may read judgments. And recording a judgment needs only the write scope — nothing here checks whether the run you named exists, so pointing at a run you may not read tells you nothing about it.

Reading judgments accepts metric_name, author_id and limit. author_id narrows the answer and is not a permission boundary: everyone who may read judgments in a project sees every colleague's, because a rating nobody else can see cannot be disagreed with. The batch read takes subject_refs on the same terms as results — at most 200 runs, refused rather than shortened, limit bounding each run's share — and its response carries the same exact truncated and truncated_subject_refs.

Historical note: quality feedback recorded before this surface existed is in results, not here. See the section above — nothing was lost, but a report that compares people with judges has to read both.

Calibration — how far a judge agrees with your people

A model judge's score is a claim, and until somebody checks it against a person it is a claim nobody has tested. Calibration measures that agreement over subjects you have both judged, and records it on the exact evaluator version.

curl -X POST \
  "https://app.anectico.com/api/v1/evaluation/evaluators/$EVALUATOR_ID/versions/4/calibrations" \
  -H "Authorization: Bearer $ANECTICO_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"subject_refs":["'"$RUN_A"'","'"$RUN_B"'"],"min_labels":20}'

curl "https://app.anectico.com/api/v1/evaluation/evaluators/$EVALUATOR_ID/calibrations" \
  -H "Authorization: Bearer $ANECTICO_API_KEY"

Or from the CLI:

anectico evals evaluators calibrate quality 4 --min-labels 20 --subject run-a --subject run-b
anectico evals evaluators calibrations quality

Which statistic you get depends on your labels. Where reviewers recorded a number, agreement is Lin's concordance coefficient between their ratings and the judge's. Where they recorded a verdict, it is Cohen's kappa — chance-corrected agreement, not the raw match rate. That distinction is the point: a judge answering "good" to everything matches an 80%-good corpus 80% of the time and has established nothing. The raw rate is still reported as agreement_rate because it is a fact; it is never the calibration.

The same reasoning is why the numeric statistic measures agreement rather than correlation. A judge that rates every response exactly half as highly as your reviewers do tracks them perfectly — plain correlation scores that a flawless 1.0, and it would clear any threshold you set. But its numbers are on a scale nobody agreed to, so every absolute threshold you write against them is wrong. Concordance penalises that offset: the same judge scores about 0.57. If your judge's agreement drops after you upgrade, this is the likely reason, and the fix is to correct the judge's scale rather than to lower the gate.

Every state except one carries no number, and that is deliberate — an agreement figure nobody measured must never read as one.

State	What it means
CALIBRATED	Agreement was computed. This is the only state carrying agreement.
UNCALIBRATED	A model judge nobody has compared to a person.
INSUFFICIENT_LABELS	Fewer comparable judgments than min_labels. Not a weak number — no number.
UNDEFINED_STATISTIC	Enough judgments, and a statistic that cannot be computed from them: everyone used one and the same verdict, or both sides gave the identical unvarying rating. Note that only both sides being constant is undefined — if your reviewers varied and the judge answered the same number every time, that is measurable disagreement, and it scores zero rather than landing here.
NOT_APPLICABLE	The evaluator is not a model judge. A deterministic or reference-trajectory evaluator is reproducible from its own definition.

min_labels is required and zero is refused, on the same terms as a gate's min_samples: a calibration you will act on over four judgments is one to ask for out loud.

A record is immutable per (version, label set). Recomputing over unchanged judgments returns the existing record with created: false; a reviewer changing their mind, or the judge being re-run, produces a new record beside it, so the history shows whether agreement moved when you changed the rubric. Each record names the exact judgment ids it was computed from.

Judgments about subjects the judge never scored are excluded and counted rather than treated as disagreements — a sampling gap is not a bad judge — and a judge that could not produce a verdict at all is excluded on the same terms. pair_count is what was compared and excluded_count is what was not.

Calibration reads your judgments and the judge's results and does arithmetic. It calls no model and spends no evaluation budget.

The posture appears in three places, so a number's credibility travels with it: on the evaluator version, on every result that version produced, and as the rule a release gate applies before it will treat a judge's score as evidence.

The explanation is treated as content

An explanation describes the run it scored, so it can restate what the run contained. Two independent controls decide whether you receive it, and both have to permit it:

• Your credential must hold agents:content:read. Without that scope the field is empty.
• Your project's content policy must allow model transcripts and tool arguments to be read. An explanation is a single piece of prose that may quote either, so it is disclosed only when both are allowed — no scope overrides that, and the same rule governs whether the reasoning is written down in the first place. If your project's policy cannot be read at the moment, the field is empty until it can be; nothing is lost, and reading again later returns it.

The score and the label are not withheld from anyone with read access, whatever the policy says. They are the answer, and they are what quality badges are built from — which is also why a label is held to a short verdict word: a model that answers with a paragraph gets scored instead, and its number is unaffected.

One exception, and it is narrow: quality feedback recorded before this surface existed and migrated into results carries the reviewer's own comment in explanation. A comment a colleague typed is served on the same terms as a human judgment's comment field — to anyone who may read judgments — because it is a person's sentence about a run rather than a machine's restatement of one. Every migrated machine comment stays behind the two controls above.