Evaluation experiments

Start with your first evaluation for a runnable workflow. See API and permissions for endpoint and credential requirements.

Experiments — comparing two configurations on the same cases

An experiment compares two or more configurations of your agent on ONE frozen dataset version, scored by the same evaluators at the same exact versions. One configuration is the baseline; every other is compared against it.

One experiment can compare the five operational dimensions together: quality, safety, reliability, latency, and cost. Declare one exact evaluator version for each dimension. The comparison output is then one paired result per candidate and evaluator; a dimension appears only when both arms produced real scores for the same cases.

Because each configuration is measured on the same case, the comparison is paired — which removes the case-to-case variation that is usually far larger than the difference you are looking for. Comparing two configurations' averages over different cases asks a question that one hard case can answer wrongly.

What you supply. The platform records what your agent did; it does not run your agent. So for each case you tell it which recorded run each configuration produced — a trial. Leaving the run out means "score the frozen case itself", which is the fully repeatable arm: frozen cases cannot change and need no original run to still exist.

What you get back, per configuration and per evaluator:

Field	Meaning
paired_items	How many cases were scored under both configurations. Only these are compared.
baseline_only_items, candidate_only_items	Cases one side has and the other does not. Excluded, and reported — a comparison resting on a third of your dataset looks exactly like one resting on all of it.
mean_difference	The average per-case difference.
ci_low, ci_high, confidence_level, ci_method	The interval for that average, and the procedure that produced it.
p_value, p_value_method	Before the multiple-comparison correction.
adjusted_p_value, significant	After it, at the alpha you declared.
small_sample	Fewer than eight paired cases — the interval is reported, and is not worth much.
degenerate	Every case differed by the same amount, so the interval has zero width for an arithmetic reason.
unevaluated (on the response)	Declared trials with no score yet.

You must say how multiple comparisons are handled

Comparing several configurations, or several evaluators, makes it more likely that something looks significant by chance. So an experiment cannot be created without saying what to do about that:

correction	What it controls
NONE	Nothing. The p-values are uncorrected, and you have said so deliberately.
BONFERRONI	The chance of any false positive across the whole comparison.
HOLM	The same thing, less conservatively. The better of the two if you want that guarantee.
BENJAMINI_HOCHBERG	The expected proportion of false positives among the findings. A weaker, different guarantee.

There is no default. Every comparison you read back states the method, the alpha, how many comparisons it covered, and — in words — what it actually controls, so a number is never quotable without what it claims.

The correction is fixed when the experiment is created and cannot be changed afterwards, along with the dataset version, the evaluator versions and the candidate set. A choice made after the numbers are in is not a correction.

Running one

Running an experiment scores the outstanding cases and is deliberately resumable: one call does a bounded amount of work and tells you how many cases remain. Call it again while remaining is above zero. Repeating a call never re-scores a case that is already done, and never charges you twice for one.

Repeatability

Re-running an experiment over frozen cases with the deterministic evaluators reproduces every number exactly. Two things are outside that promise, and are worth knowing before you rely on it: a trial pointing at a live recorded run depends on that run still being there and unchanged, and an LLM-judge evaluator varies from run to run on its own.

An experiment is limited to 8 configurations, 8 evaluators, and 20 000 total case-scores. The limit is checked when you create it, so an experiment too large to compare is refused up front rather than after you have paid for the scoring.