Evaluation release gates

Start with your first evaluation for a runnable workflow. See API and permissions for endpoint and credential requirements.

Release gates — deciding whether a release may ship

A release gate binds a candidate release to evaluation evidence and produces a pass or a refusal with the reason, kept for ever. It is the thing that turns a comparison into a decision.

A gate is fixed when you create it — its evidence, its objectives, its confidence level. A changed objective is a new gate, because a gate whose bar can move under it is not a gate.

The two kinds of evidence, which are different designs

evidence_kind	What it compares	Paired?
GATE_EVIDENCE_KIND_EXPERIMENT	One arm of a declared experiment against that experiment's own declared baseline, on the same frozen cases.	Yes — the same case is scored under both configurations.
GATE_EVIDENCE_KIND_CANARY	Two windows of live traffic that you declare: one before the candidate shipped, one after.	No. The runs in the two windows are different runs, so the interval carries the full case-to-case variance.

You do not name the comparison arm for experiment evidence — the experiment's own declared baseline is used. Choosing the arm afterwards is how a flattering pair gets picked.

For a canary, the window boundary is your declaration of when the candidate shipped. Everything else that changed at that boundary is confounded with the release, and every stored canary comparison says so in words alongside its numbers, because a narrow interval and a causal claim look identical on a chart.

Each canary window includes its start and excludes its end: [start, end). Selection uses the subject's occurrence time, including windows within one second. Saved window bounds retain microsecond precision; result occurrence times retain nanosecond precision.

Objectives

Each objective names an evaluator at an exact version, the metric it must produce, what to compare, and the line to clear:

basis	The number compared against the threshold
GATE_BASIS_CANDIDATE_MEAN	The candidate's own average. An absolute floor or ceiling; needs no baseline.
GATE_BASIS_DIFFERENCE_MEAN	The point estimate of candidate minus baseline.
GATE_BASIS_DIFFERENCE_INTERVAL	The conservative bound of the interval around that difference — the low bound for a floor, the high bound for a ceiling. The strong form: the whole interval must clear the line, so a candidate that passes on noise does not.

comparator is the direction the threshold is checked in:

comparator	Passes when the basis value is
GATE_COMPARATOR_GREATER_OR_EQUAL	>= threshold
GATE_COMPARATOR_GREATER	> threshold
GATE_COMPARATOR_LESS_OR_EQUAL	<= threshold
GATE_COMPARATOR_LESS	< threshold

threshold is the line itself — the number the basis value is compared to using comparator.

min_samples is required and zero is refused. A gate that will pass on a single observation is something you have to ask for out loud.

min_judge_agreement is optional and puts a floor on how far the objective's evaluator must agree with human judgments — see Calibration. It is not a way to opt out: an objective bound to a decision-score evaluator with no calibration at all is inadmissible whatever this field says, and a gate whose only evidence comes from that evaluator answers GATE_VERDICT_INADMISSIBLE rather than passing. Leaving it at zero asks only that a calibration exist.

The rule applies to decision-score evaluators alone, the one kind a model answers. A deterministic or reference-trajectory evaluator is reproducible from its own definition and is calibrated against nobody, so nothing is asked of it.

Calibration is resolved when the gate is evaluated, not when it is created. A gate written before its evaluator was calibrated passes once the evaluator is; an evaluator whose calibration was superseded by an edited judgment set stops passing until it is recomputed.

An objective is checked against your evaluator registry when the gate is created, including that the evaluator you named actually produces the metric you named — so a gate that could only ever measure nothing is refused before it exists rather than after a release waited on it.

Verdicts: only GATE_VERDICT_PASS passes

Verdict	What it means	Where to look
GATE_VERDICT_PASS	Every objective was measured on admissible evidence and met.	—
GATE_VERDICT_FAIL	At least one objective was measured and missed.	The objective's numbers.
GATE_VERDICT_PENDING	Created, never evaluated. Not a pass.	Evaluate it.
GATE_VERDICT_NO_EVIDENCE	An objective had nothing admissible to measure, or less than it declared it needs.	Your sampling — nothing ran.
GATE_VERDICT_INADMISSIBLE	Results exist for an objective and none of them may be used — incomplete provenance, an execution record that cannot say how the score was produced, a job that did not complete, or a decision-score evaluator that has not been calibrated.	Your evaluator reference — things ran, and cannot be used.
GATE_VERDICT_RELEASE_MISMATCH	Every objective met, but the candidate runs did not all observably declare the release the gate names. Not a pass.	The decision's release_binding: it distinguishes another release, no declared release, partial instrumentation, and an unmeasured subject shape.

The last two are kept apart deliberately: "nothing ran" and "what ran cannot be used" are different problems with different fixes, and one empty answer cannot tell them apart.

Per objective you also get observed_present, which is the only way to tell "the value was 0" from "there was no value". A score of 0.0 is a legitimate measurement, so the number can never be that discriminator — never infer absence from observed_value. inadmissible_samples counts results that exist and may not be used; they are reported rather than filtered away, for the same reason.

Evaluating one

The first request computes a complete preview of the verdict, objective outcomes, sample counts and release binding, without storing a decision. A stable idempotency_key is required through REST, MCP or anectico evals gates evaluate --idempotency-key <key>. Review the complete current and proposed state, then repeat identical arguments with the returned confirm_token (CLI --confirm-token). If the evidence changed, obtain and review a new preview.

Confirmation appends one permanent decision. Reusing its key returns that original decision, even if new results arrive later. Its decided_at, counts and outcomes describe the original evidence snapshot. Use a new explicit key for a fresh decision; it is stored beside the first. Gate reads return one page of decision history; follow pagination.next_cursor with unchanged filters.

A gate never runs an evaluator and never spends your evaluation budget. That is deliberate — a gate that produced its own evidence on demand could never report GATE_VERDICT_NO_EVIDENCE, which is the one answer it exists to make reachable. Score first, then gate.

If the evidence store cannot be read, you get an error and nothing is stored. "We could not look" is never recorded as "we looked and found nothing".

A release label is a claim that the evidence must prove

release is the label you declare when creating the gate. Evaluation reconciles it against the release actually carried by every admissible candidate run, deduplicated across objectives; baseline runs are expected to belong to the earlier comparison arm and do not participate in this check.

Every decision stores release_binding with the observed release names, released, unreleased, and unmeasured subject counts, a reason, and one of GATE_RELEASE_BINDING_STATE_MATCHED, GATE_RELEASE_BINDING_STATE_MISMATCH, GATE_RELEASE_BINDING_STATE_UNDECLARED, GATE_RELEASE_BINDING_STATE_PARTIAL, GATE_RELEASE_BINDING_STATE_UNMEASURED, or GATE_RELEASE_BINDING_STATE_NO_EVIDENCE. Only GATE_RELEASE_BINDING_STATE_MATCHED may accompany GATE_VERDICT_PASS. If the objective numbers meet but the label does not, the verdict is GATE_VERDICT_RELEASE_MISMATCH rather than a green decision carrying a warning. If an objective already missed, the verdict remains the more direct GATE_VERDICT_FAIL, but the same binding discrepancy is still recorded.

“No release declared” is a first-class answer. A gate evaluated over those runs returns binding GATE_RELEASE_BINDING_STATE_UNDECLARED and cannot pass; it does not error, silently discard those runs, or pretend they belong to the gate's label.