# Create useful alerts

> Notify the right people about customer-impacting failures without creating noise.

Canonical page: https://anectico.com/docs/respond/alerts/


An alert should describe a condition someone can act on. Start with customer symptoms and critical
journeys, then add service-level alerts only where they support that response.

## Add a destination

Open **Settings → Notifications** and create a notification channel for direct rule delivery. Send a
test notification before attaching it to a production rule. Add personal contact methods only when
the rule routes through an escalation policy or on-call schedule.

Available destinations depend on the connections enabled for your organization. Webhooks can cover
destinations without a dedicated integration.

## Create a rule

Open **Alerts** and create a rule. Define:

- the project and environment;
- the signal or customer being watched;
- the condition type, its specific settings, and the evaluation window;
- how long the condition must remain true and then remain clear; and
- the notification destination.

The selected project is an evaluation boundary, not only an organizational label:
the rule reads telemetry from that project and never combines matching data from another
project. If a watched customer/account scope or grouped query cannot be measured completely,
Anectico leaves that evaluation unmeasured rather than treating missing results as zero.

For a cohort target, each evaluation reads one exact membership generation from Analytics.
The generation must be synchronized and its source evidence must remain readable. Anectico checks
again after reading the scoped telemetry. If the cohort changes, source content is erased, or
membership becomes unavailable, that evaluation fails without triggering or clearing alerts or
changing their unmeasured markers. An unavailable cohort is never treated as an empty one. Refresh
the cohort to produce a new valid generation before retrying; restoring old source rows alone does
not make an erased generation usable again.

Cohort notifications retain the exact generation used for their rendered observation, including
escalation steps and retries. Before each channel or personal-contact send, Anectico checks that
original generation again. Missing, pruned or erased evidence stops delivery without a provider
attempt. A temporarily unavailable evidence service delays delivery without consuming an attempt.
Refreshing the cohort does not authorize an older queued notification. A complete later generation
that removes a watched person starts the rule's normal recovery handling for that person's alert.


Use a sustained firing window for noisy metrics and a recovery window when one healthy evaluation
should not close an incident. A breach during recovery restarts the healthy window. Use immediate
notification and recovery only for conditions where one event is decisive, such as a critical
customer error.

For a metric rule, select `metrics`, add a `metric_name` filter, and use the same aggregation and
window you already validated in **Signals → Metrics** or a dashboard. A threshold is the clearest
choice for a hard service limit. An anomaly condition can preview an expected band when
`metric_name` is present, but it remains an approximate signal rather than proof of a problem.

### Query fields

Rule filters are exact, case-sensitive string matches. Choose a field from the dashboard picker or
use one of the API fields below. Anectico validates and normalizes the query before saving the rule, so
a typo or a field from the wrong signal returns a validation error instead of creating a rule that
fails later. `service` is the portable field for every signal and maps to the stored
`service_name` column. The canonical `service_name` spelling is accepted too. Logs also accept
`level` for `severity`; traces accept `operation` for `operation_name` and `status` for
`status_code`.

| Signal | Exact-match filter fields |
| --- | --- |
| Errors | `service`, `service_name`, `environment`, `error_id`, `error_group_id`, `group_id`, `fingerprint`, `error_type`, `type`, `message`, `release`, `dist`, `user_id`, `user_email`, `user_ip_address`, `trace_id`, `span_id`, `session_id`, `distinct_id`, `request_method`, `request_url`, `platform`, `sdk_name`, `sdk_version`, `project_id` |
| Logs | `service`, `service_name`, `level`, `severity`, `severity_text`, `message`, `environment`, `trace_id`, `span_id`, `logger_name`, `host`, `pod_name`, `source_file`, `distinct_id`, `session_id`, `project_id` |
| Metrics | `service`, `service_name`, `metric_name`, `name`, `metric_type`, `type`, `environment`, `project_id`, `unit`, `description`, `aggregation_temporality` |
| Traces | `service`, `service_name`, `operation`, `operation_name`, `status`, `status_code`, `status_message`, `span_kind`, `environment`, `trace_id`, `span_id`, `parent_span_id`, `distinct_id`, `session_id`, `project_id`, `gen_ai_provider_name`, `gen_ai_model`, `source_operation`, `gen_ai_cost_source`, `agent_session_key`, `gen_ai_agent_name`, `gen_ai_agent_id`, `gen_ai_tool_name`, `gen_ai_response_id`, `gen_ai_finish_reason` |
| Agent steps | `category`, `operation`, `canonical_operation`, `source_operation`, `source_kind`, `agent`, `agent_observation_key`, `environment`, `trace_id`, `span_id`, `session_key`, `turn_id`, `distinct_id`, `source_schema`, `project_id` |

`count` can omit its field. Numeric reducers (`sum`, `avg`, `min`, `max`, `p50`, `p95`, and
`p99`) accept `value` for metrics, `duration_ns` and the numeric `gen_ai_*` token/cost fields for
traces, or `source_line` for logs. Errors and agent steps support `count` only. Grouping is also
signal-specific; the rule editor shows the supported dimensions. Filter objects, arrays, nulls, comparison
operators, arbitrary free-form expressions, and unknown identifiers are rejected before the rule
is persisted.

### Alert on what an agent did

The **Agent steps** signal (`agent_events` in the API) is one row per step an agent took inside a
run — a guardrail decision, a tool call, a memory write. Use it for questions a run-level view cannot
answer: how often a guardrail is denying, how much a given agent is calling a particular tool, or
whether an agent has stopped taking steps at all.

Agent-step rules are created through the API and the CLI today; the rule editor's signal picker does
not offer them yet. Everything below applies to a rule created either way.

Agent steps support `count` and no other reducer, and every other reducer is rejected when you save
the rule rather than silently returning zero. The reason is worth stating plainly: a step does not
yet report its own duration, so an average over it would compare `0` against your threshold on every
evaluation and page you forever while nothing was wrong. Those reducers become available in the same
release that starts reporting the numbers behind them.

Two shapes cover most of what this signal is for:

- **A rate ceiling.** Count steps in a window, filter to `category` (and optionally `agent`), and set
  a `>` threshold — "more than 10 guardrail denials in 15 minutes".
- **A silent fleet.** Count steps and set `<` with a threshold of `1`, or use an absence condition.
  Either fires when the window contains no agent activity at all.

Because the reducer is `count`, the result field on the condition is the alias you gave the count.
Naming a raw field there instead is rejected when you save.

If you group an agent-step rule by `distinct_id`, that label is treated as person-identifying, the
same way `user_email` is on an error rule: it is kept out of the delivered notification entirely, and
reading it back from the API needs permission to read agent runs, not just alerts.

For a heartbeat, scheduled job, or other expected signal, choose an absence condition. It checks
whether the filtered query matched any source rows during `query.time_range`, which is the sole
lookback. Raw, grouped, and aggregate queries are supported; a real aggregate value of zero still
counts as present data. The condition does not use an operator, threshold, result field, or its own
`time_window`.

### Alert on an evaluation objective

An **SLO condition** alerts on the scores an evaluator produces, rather than on raw telemetry. Use
it to hold a quality objective — "faithfulness stays at or above 0.9", "task completion does not
fall below 0.95" — the same way a threshold holds a latency or error-rate objective.

SLO rules are created through the API and the CLI today; the rule editor's signal picker does not
offer them yet. Everything below applies to a rule created either way.

An SLO condition names three things:

| Setting | Meaning |
| --- | --- |
| Evaluator | The evaluator whose scores the objective is measured on. |
| Version | An exact evaluator version. Required, and never "latest". |
| Mode | Which kind of run counts: `production`, `experiment`, or `simulation`. |

Set the operator and threshold to the objective itself, aggregate `numeric_value` over the window,
and filter or group on `evaluator_id`, `evaluator_version`, `mode`, `metric_name`, `subject_kind`,
`label`, `environment`, or `project_id`.

Three rules follow from what an SLO is for, and each one is enforced when you save:

- **The version is exact.** An objective bound to a moving evaluator is not an objective: the same
  release would pass on Monday and fail on Tuesday with nothing in the record explaining why. A
  version of `0` or a missing version is rejected.
- **The evaluator must already exist, and must be reproducible.** Saving the rule checks that the
  evaluator and version are registered in the project, that the evaluator names the dimension it
  scores, and that its configuration is fingerprinted so a score can be traced back to exactly the
  rule that produced it. An evaluator that cannot supply that is refused when you save the alert,
  naming what is missing — not at 3am when the alert fires on numbers nobody can reproduce. If the
  check cannot be completed, the rule is not saved and you can retry; it is never stored unchecked.
- **One objective per rule, on its own.** An SLO condition cannot be combined with another
  condition, and evaluation scores cannot be read by any other condition type. This keeps the
  numbers an alert compares scoped to the exact evaluator, version and mode it declared.

Because an evaluator belongs to a project, an SLO rule must be scoped to a project rather than to
the whole organization. Creating one also requires permission to read evaluations.

The exact evaluator reference is checked again immediately before every scheduled evaluation. The
rule's `evaluator_status` reports `valid`, `invalid`, or `unavailable`, with
`evaluator_checked_at` and a safe `evaluator_error` when something is wrong. An invalid reference
does not run the score query and is never represented as a score of zero. Updating the rule clears
the previous status until the new definition is checked.

The objective is evaluated on when each score was **produced**, so a score always lands in a window
the alert has yet to check, however long the evaluation itself took.

The evaluator's written `explanation` is recorded content, not an alert dimension. SLO evaluation
selects only the declared score metadata and numeric fields, so an explanation is never copied into
an alert label, alert read, or Slack/email/webhook notification. Read the evaluation result itself
with the separate content permission and current project policy when you need that reasoning.

## Test before enabling

Use the rule test action and confirm the message contains enough context to begin an investigation:
the condition, severity, project, environment, time, and a link back to Anectico.

## Respond to an alert

1. **Acknowledge** when someone takes ownership.
2. Open the linked customer, Issue, trace, or service evidence.
3. Create an incident when coordination or a durable case file is needed.
4. **Resolve** after the condition has cleared and follow-up is recorded.

Resolving is final for that alert. It records who resolved it and when, and nothing reopens it — if
the condition breaches again you get a new alert with its own history, rather than the old one
returning to firing. This holds even when a check of the same rule is already running as you
resolve.

Use a silence for planned maintenance or a known temporary condition. Give every silence an owner
and expiration time. Matching alerts show `silenced` with that provenance; acknowledging one keeps
the responder-owned state until the acknowledgment is cleared. If maintenance ends while the
condition still breaches, the alert returns to firing and notifies once on its next evaluation.

## Tune based on outcomes

Review alerts that fired without action and alerts that should have fired but did not. Change the
condition, window, or routing rather than training the team to ignore noise.

- [Manage incidents and on-call](/docs/respond/incidents-and-on-call)
- [Explore and compare metrics](/docs/investigate/metrics)
- [Alerts or integrations are not delivering](/docs/help/alerts-and-connections)