Evaluator configuration

Start with your first evaluation for a runnable workflow. See API and permissions for endpoint and credential requirements.

The three pieces

Piece	What it is
Evaluator	A named, versioned rule that turns a run into a number and a label.
Sampling rule	Which runs an evaluator sees, at what rate, with what per-tick ceiling.
Result	One score, with the full record of what produced it.

Evaluator versions are exact, and never float

An evaluator has an id and a list of versions. A version is immutable: changing a rule creates a new version and leaves the old one exactly as it was.

Every reference must name an existing version explicitly; there is no "latest" alias. References are validated when written, so a missing version fails the request immediately. Pin the same version when comparing releases.

Creating an evaluator

curl -X POST https://app.anectico.com/api/v1/evaluation/evaluators \
  -H "X-Anectico-API-Key: $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator_id": "task_completion",
    "display_name": "Task completion",
    "metric_name": "task_completion",
    "kind": "EVALUATOR_KIND_DETERMINISTIC",
    "config_json": "{\"aggregation\":\"all\",\"assertions\":[{\"field\":\"run.status\",\"op\":\"equals\",\"value\":\"AGENT_RUN_STATUS_COMPLETED\"},{\"field\":\"run.error_steps\",\"op\":\"lte\",\"number\":0}]}"
  }'

The response carries the evaluator and its version 1. Append a new version with POST /api/v1/evaluation/evaluators/{evaluatorId}/versions; there is no update or patch, by design.

Evaluator kinds

There are three: EVALUATOR_KIND_DETERMINISTIC, EVALUATOR_KIND_REFERENCE_TRAJECTORY and EVALUATOR_KIND_DECISION_SCORE. Only the last asks a model, and it answers with a score and a level name, never with prose. The former EVALUATOR_KIND_LLM_JUDGE no longer exists, and creating one is refused.

EVALUATOR_KIND_DETERMINISTIC — declarative assertions over a run's observable fields.

{
  "aggregation": "all",
  "assertions": [
    {"field": "run.status",       "op": "equals",   "value": "AGENT_RUN_STATUS_COMPLETED"},
    {"field": "run.error_steps",  "op": "lte",      "number": 0},
    {"field": "steps.tool_names", "op": "contains", "value": "search_docs"}
  ]
}

aggregation is all (a gate: every assertion must hold, the score is 1 or 0) or fraction (the share of assertions that held).

Field	Type	Operators
run.status	one value	equals, not_equals
run.error_steps, run.step_count, run.tool_calls, run.model_calls, run.duration_ms, run.cost_nanos	number	lte, gte
safety.error_steps	number	lte, gte
steps.outcomes, steps.tool_names, steps.canonical_operations	set	contains, not_contains
session.turn_count, session.run_count, session.duration_ms, session.cost_nanos	number	lte, gte

run.cost_nanos and session.cost_nanos read as unmeasured rather than zero when the run or session was never priced, and safety.error_steps reads as unmeasured rather than zero when the run carries no safety step — an assertion against an unmeasured field does not treat the absence as 0.

The session.* fields describe a whole conversation and are answerable only for SUBJECT_KIND_AGENT_SESSION — see What you can name as the subject.

A deterministic evaluator can instead return one native measurement. A measurement config contains measurement and no assertions or aggregation:

{"measurement":"run.cost_nanos"}

Measurement	Unit	When it is measurable
run.duration_ms	milliseconds	The recorded run reports a duration.
run.cost_nanos	billionths of a US dollar	The run has client-reported or calculated model cost. A missing or unpriced cost is not zero and produces no score.
safety.error_steps	count	The run contains explicitly identified safety-check steps with reported outcomes. No safety check, or an unknown outcome, is not a safe zero and produces no score.
session.turn_count	count	The session's turns carry turn ids. A session with none is not a session of nought turns; it produces no score.
session.run_count	count	Always measurable for a session that exists.
session.duration_ms	milliseconds	The time between the session's first and last recorded activity, including fractional milliseconds for measurements and numeric assertions. A session with one moment of activity measures 0, which is a real answer.
session.cost_nanos	billionths of a US dollar	The session's runs reported model cost. Nothing on a session says whether an unpriced conversation was free, so a zero is treated as unmeasured and produces no score.

Use the assertion form for reliability (for example, completed status and zero error steps), a reference-trajectory or decision-score evaluator for quality, and the native measurements for safety, latency, and cost. Experiments and release gates treat each as the evaluator's real numeric result; they do not fill missing dimensions with defaults.

EVALUATOR_KIND_REFERENCE_TRAJECTORY — did the agent do the right things in the right order.

{
  "step_key": "canonical_operation",
  "match": "ordered_subsequence",
  "reference": [
    "CANONICAL_OPERATION_INVOKE_AGENT",
    "CANONICAL_OPERATION_RETRIEVAL",
    "CANONICAL_OPERATION_CHAT"
  ]
}

ordered_subsequence (the default) scores the longest in-order match divided by the reference length; extra intervening steps do not reduce the score. exact requires an identical sequence. step_key may be canonical_operation or tool_name.

EVALUATOR_KIND_DECISION_SCORE — a decision model places the run on an ordered scale you write.

Use it for questions such as groundedness or relevance that cannot be expressed as deterministic assertions. The question is "which of these situations is this run in": you declare levels, lowest first, each with a name and a description of the situation. The model reads the evidence you select and answers with a position on that scale and a probability for every level, rather than prose.

{
  "instructions": "Judge only from the evidence shown. Prefer the lower level when you are unsure.",
  "levels": [
    {"name": "ungrounded", "description": "claims contradict, or are absent from, the retrieved context"},
    {"name": "partial",    "description": "some claims trace to the retrieved context"},
    {"name": "grounded",   "description": "every claim traces to the retrieved context"}
  ],
  "evidence": ["run_summary", "model_transcript"],
  "max_evidence_chars": 12000
}

Field	Requirement	Meaning
levels	required, 2–10	Ordered lowest to highest. Each needs a name (one line, at most 32 characters, unique within the evaluator) and a description.
evidence	required	What the model is shown. See below.
instructions	optional	A preamble for the scoring instruction.
max_evidence_chars	optional, 500–40000	Evidence budget. Defaults to 12000.

The configuration is validated when you write it, and unknown fields are rejected, not ignored. There are no weighted criteria and no model field: the scale you declare is the whole question.

Create it like any other evaluator, with the configuration as a JSON string:

curl -X POST https://app.anectico.com/api/v1/evaluation/evaluators \
  -H "X-Anectico-API-Key: $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator_id": "answer_grounding",
    "display_name": "Answer grounding",
    "metric_name": "answer_grounding",
    "kind": "EVALUATOR_KIND_DECISION_SCORE",
    "config_json": "{\"instructions\":\"Judge only from the evidence shown. Prefer the lower level when you are unsure.\",\"levels\":[{\"name\":\"ungrounded\",\"description\":\"claims contradict, or are absent from, the retrieved context\"},{\"name\":\"partial\",\"description\":\"some claims trace to the retrieved context\"},{\"name\":\"grounded\",\"description\":\"every claim traces to the retrieved context\"}],\"evidence\":[\"run_summary\",\"model_transcript\"],\"max_evidence_chars\":12000}"
  }'

What the result carries. numeric_value is the run's position on the scale, normalized to a number between 0 (the lowest level) and 1 (the highest). The position is continuous, so it usually falls between two levels rather than on one. label is the name of the most probable level, always one of the names you declared. The result carries no explanation: the model produces a distribution, not reasoning, and none is invented. To see why a run landed where it did, read the run itself.

What the model is shown

evidence is a declaration, and it is the one that decides what leaves your project:

Selector	What the model sees	Content
run_summary	Status, step counts, duration.	—
step_sequence	The ordered operations, outcomes and tool names.	—
model_transcript	Recorded prompt and response bodies.	model_transcript
tool_calls	Recorded tool and MCP call arguments and results.	tool_arguments

The first two carry no message bodies at all — a tool name is a symbol, not the arguments it was called with — so an evaluator built only from them shows a model nothing you recorded.

Your content policy decides whether the evidence may be sent

Sending recorded content to an evaluation model is a transfer, and it is governed by your project's content policy at the judge_transfer boundary. The policy is consulted before anything is sent, and before any budget is charged, never after.

Three properties are worth stating exactly, because all are easy to get wrong and none is visible from the outside:

• The question asked of your policy is derived from what the evidence actually contains, not from what the evaluator might want. An evaluator that selects tool_calls, running against a run that recorded no tool arguments, asks nothing about tool_arguments — so a project that permits transcripts and refuses tool arguments is evaluated normally on those runs.
• But a kind of content your policy actually withheld is never quietly dropped. When the content your evaluator asked for was removed by the policy, the transfer is put to the policy again for exactly that kind, and a refusal is a refusal for the whole evaluation — 403 Forbidden — even when the evaluator's other selectors still had something to show. Scoring "was the answer grounded in the transcript" over a run summary alone is scoring something you did not ask about. If the removal was not your policy's decision (a narrower credential, for instance), the evaluation proceeds and the model is told which kinds of content were withheld, by name, so it cannot read an absence as evidence that nothing was there.
• A refusal is terminal and it is recorded. The run is not sent, is never scored 0, is not charged against your budget and is not retried in a loop. For an automatic check it is recorded as withheld, naming the kind of content, the boundary, and which of three things decided: your policy said so, the project has no policy yet, or the policy could not be read. "Nobody judged this" and "nobody sampled this" stay different facts. On an on-demand evaluation the answer is 403 Forbidden, and its message names the kind of content and judge_transfer, never the content itself.

The read and judge_transfer cells are independent for automatic quality checks. A run may be hidden from product reads and still evaluated when judge_transfer allows its content; denying judge_transfer still prevents the transfer whatever the read cell says.

Why some rules are refused

Evaluator configuration is validated when you write it, and some rules are refused even though they are well-formed. Each refusal names its reason. The reason is always the same one:

A field that nothing fills is not a field you can measure.

A run's operation, outcome and similar values carry a placeholder when your instrumentation did not report them. A rule written against that placeholder does not measure your agent — it measures how completely you are instrumented, and reports the answer as a quality score. Worse, a trajectory written against it scores higher the less instrumented the run is, inverting the metric entirely.

So you cannot assert on OUTCOME_UNKNOWN, on CANONICAL_OPERATION_UNKNOWN, or on fields no instrumentation populates. If you need a value that is refused, the fix is to emit it — see Instrumenting LLM and agent calls.

The same principle applies at run time. If a run carries nothing the evaluator can measure, it is recorded as unmeasured and retried later, never scored zero. A zero would say your agent failed; the truth is that nobody measured it. A decision-score evaluator is held to the same rule twice over: a run carrying none of the evidence the evaluator selected is not sent to a model at all, and a model that comes back with something unusable is retried rather than recorded as a failing score.

Sampling production traffic

curl -X PUT https://app.anectico.com/api/v1/evaluation/sampling-rules \
  -H "X-Anectico-API-Key: $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator": {"evaluator_id": "task_completion", "version": 1},
    "subject_kind": "SUBJECT_KIND_AGENT_RUN",
    "sampling_rate": 0.1,
    "per_tick_limit": 50,
    "enabled": true
  }'

• sampling_rate (0–1) is a true cost cap. Selection is derived from a stable hash of the run, so the same run gets the same answer every time and the unselected share is never evaluated at all. It is not a per-cycle coin flip, which would eventually select everything.
• per_tick_limit caps how many evaluations start per cycle. Zero disables the rule, as does enabled: false or a rate of 0.

A rule is evaluated only where you create one. Nothing is evaluated by default.

Evaluating one run on demand

curl -X POST https://app.anectico.com/api/v1/evaluation/evaluate \
  -H "X-Anectico-API-Key: $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluator": {"evaluator_id": "task_completion", "version": 1},
    "subject_kind": "SUBJECT_KIND_AGENT_RUN",
    "subject_ref": "0af7651916cd43dd8448eb211c80319c"
  }'

This is the same engine, with sampling skipped — naming a run is selecting it. Asking twice for the same run and evaluator version returns the stored answer with already_evaluated: true; it does not re-run and does not spend again.

Set your client's timeout above 30 seconds

The call is synchronous: it reads the subject, and for a decision-score evaluator it waits for the model. Such a call is allowed up to five minutes. Clients whose default request timeout is 30 seconds — which is most of them — can hang up on a call that is still working, and the evaluation is then abandoned rather than stored.

Give this one endpoint its own timeout of six minutes — a little above the five the server allows, so that what you read is the server's answer rather than your own client giving up — or use a deterministic evaluator (which answers in under a second) when you need a fast answer.

If the five minutes do run out, you get 504 Gateway Timeout with a message, not a dropped connection. The anectico CLI already applies its own longer timeout to anectico evals evaluate and anectico evals experiments run; every other command keeps the short default.

What you can name as the subject

subject_kind says what subject_ref is, and it is also the label the stored result carries. Three kinds can be evaluated today:

subject_kind	subject_ref is	Notes
SUBJECT_KIND_AGENT_RUN	the run (trace) id	The whole run. This is what sampling rules select.
SUBJECT_KIND_AGENT_STEP	"<run_id>/<span_id>"	One step of a run — the span_id is the one the run graph reports for that step. The score is about that step: the evaluator sees only it, not its siblings.
SUBJECT_KIND_AGENT_SESSION	the session key	The whole conversation: every run that shares that session key. Only the session.* fields can score it, and only an evaluator that asks for no recorded content can be run over it — see below.

SUBJECT_KIND_AGENT_TURN is part of the result vocabulary — a stored result can carry it — but cannot be submitted here: a turn cannot be looked up from its id alone. SUBJECT_KIND_DATASET_ITEM is a batch subject and is evaluated by running an experiment over a dataset version, not through this endpoint. Naming either returns 501 Not Implemented. A result is never labelled with a subject that was not read: an unsupported kind is refused rather than scored from the run behind it.

Which fields an evaluator can assert on depends on what you named, and each family is answerable for its own subject only:

• run.* describe the whole run and are answerable for SUBJECT_KIND_AGENT_RUN and for dataset items. Asking one about a step or a session returns 400 Bad Request ("carries nothing this evaluator can measure") rather than reporting the run's number under another subject's name.
• steps.* and safety.error_steps are answerable for a run and for a step, and for a step subject they describe that step alone.
• session.* describe a whole conversation and are answerable only for SUBJECT_KIND_AGENT_SESSION. Asking one about a single run of that session returns 400 Bad Request for the same reason in the other direction — a run's own duration or cost is not the conversation's.

What a decision-score evaluator reads for each subject

An evaluator that asked for model_transcript or tool_calls has to be shown recorded content, and what that content is depends on the subject:

• A run — that run's own recorded content.
• A step — the content of the run the step belongs to, with the step itself as the subject the score is filed under.
• A session — nothing. A conversation is a set of runs and has no transcript of its own, and there is no answer to "show the model the session" that would not mean choosing runs on your behalf. Rather than pick one, a content-asking evaluator over a session is refused with 400 Bad Request naming the kind. An evaluator built only from run_summary and step_sequence is not refused for content — it reads none — but a session rollup carries neither a run summary nor a step sequence, so it has nothing to show a model either and answers "carries nothing this evaluator can measure". Score a conversation with the session.* fields, which read the rollup itself.

Response	Meaning
404 Not Found	Nothing here matches what you named — the run, the step, or the evaluator version. Look the identifier up again; retrying it unchanged will not help. A run that was just produced may not be readable for a few seconds, and the same call succeeds once it is.
504 Gateway Timeout	Either the subject could not be read within its own budget, or the whole evaluation ran past the five minutes the call is allowed. This is not a 404: it says the subject could not be read, never that it does not exist, so retry the identifier you already have rather than looking for a new one. No result is stored. A model call that had already started may still have been charged, and the retry is free once a result exists — asking again for the same run and evaluator version returns the stored answer.
501 Not Implemented	This subject_kind cannot be evaluated on demand. See the table above; retrying will not help.
400 Bad Request	The subject_ref cannot name a subject of that kind — a step reference must be "<run_id>/<span_id>".
409 Conflict	Already being evaluated. Retry shortly.
429 Too Many Requests	One of two daily limits is spent, and the message says which: your organization's daily evaluation budget, or your plan's daily typed-decision spend limit (see plans and limits). Nothing was scored. The plan limit resets at 00:00 UTC.
400 Bad Request	The run carries nothing this evaluator can measure.
403 Forbidden	Your content policy withholds content your evaluator asked for from the model. The message names the kind of content and judge_transfer, never the content itself. Nothing was charged and no model was asked.
400 Bad Request	A decision-score evaluator that asks for recorded content cannot be run over this subject_kind. Today that is a session: a conversation has no single transcript to show, and the answer names the kind rather than reading one of its runs. Score the session with session.* fields, or evaluate its runs individually.
400 Bad Request	Typed evaluation is not available on this deployment, or the model refused this evaluation's request and will refuse it again. The message says so and ends "nothing was scored". Retrying unchanged will not help.
503 Service Unavailable	The model is temporarily unavailable because of a timeout, throttle, transport failure, or upstream outage. The message ends "nothing was scored, retry shortly". Retry is appropriate.

None of these answers stores a result: an evaluation the model did not answer is never recorded as a score, and never as a zero. A sampled check that meets the same conditions is released and retried on a later pass, and one your content policy refuses ends as withheld.

The spend budget

A decision-score evaluator calls a model, which costs money, so the daily ceiling is a hard limit rather than a throttle: it is enforced transactionally, and it does not open up when anything is degraded.

curl -X PUT https://app.anectico.com/api/v1/evaluation/budget \
  -H "X-Anectico-API-Key: $ANECTICO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"daily_paid_evaluation_cap": 500}'

GET /api/v1/evaluation/budget returns the cap and today's consumption.

daily_paid_evaluation_cap accepts integers from 0 through 4294967295, inclusive. It is a count of paid evaluations per UTC day, not a currency amount. Setting it to 0 disables new paid evaluations. Updating the cap does not reset consumed_today; lowering it below the amount already consumed leaves no budget for new paid evaluations that day.

Three things to know:

• An organization with no configured budget runs no paid evaluations. The safe state is the default; you opt in to spending. A decision-score evaluator in an organization with no budget never runs.
• Evaluators that do not call a model never consume budget. EVALUATOR_KIND_DETERMINISTIC and EVALUATOR_KIND_REFERENCE_TRAJECTORY are in that category, so they keep working after a paid budget is exhausted. EVALUATOR_KIND_DECISION_SCORE is not: it is charged one unit per run evaluated, once — a retried delivery of a score that was already produced never charges again.
• A second, dollar-denominated limit applies on top. A decision-score evaluation also counts against the daily typed-decision spend limit of your plan. When that limit is reached, the evaluation is not scored: a sampled check waits and is retried after the reset, and an on-demand call answers 429 with "nothing was scored".

The budget is charged after your content policy permits the transfer and before the model is asked, so a run your policy withholds costs nothing.

Setup request fields

Creating an evaluator supplies evaluator_id, display_name, kind, metric_name and config_json, and seals version 1. Adding a version supplies config_json only; the evaluator keeps its kind and metric. A sampling rule uses sampling_rate and per_tick_limit, alongside its exact evaluator, subject_kind, rule_id and enabled state. It has no filters field.