Browse documentation

Guide

Create useful alerts

Notify the right people about customer-impacting failures without creating noise.

On this page

An alert should describe a condition someone can act on. Start with customer symptoms and critical journeys, then add service-level alerts only where they support that response.

An invalid rule test returns status: error over MCP and leads with invalid: <reason>. Read the complete validation message before creating the rule. The CLI prints that result and exits 1 when invalid, including JSON output. A routing preview that cannot page anyone also reports a negative outcome and its reasons; it does not mean the service is covered.

Your agent creates and tunes alert rules, silences and escalation through MCP or the CLI. You decide where alerts are sent: notification channels are set in the Console. When an alert becomes an incident, you open its proof page to check the case.

Choose a response workflow

Metric watches measure saved product questions.
Account request objectives cover paused proposals, the current activation limit and evidence reads.
Event delivery to your agent sends signed wake events to your endpoint and exposes retained delivery receipts.
Incidents and on-call coordinate responders and the case file.
Connections and notifications configure ordinary notification channels and their delivery reads.

Ask your agent

"Create an alert rule that pages the platform team when checkout errors stay above 10 a minute for 5 minutes. Test it first and show me what it would send."

"What fired in the last 24 hours? Acknowledge the checkout alert and silence the staging rules until 18:00."

Job	MCP tool or action	CLI command
Check a rule without saving or running it	`test_alert_rule`	`anectico alerts rules test <rule-id>`
Create a rule	`create_alert_rule`	`anectico alerts rules create`
List rules and silences, or read one rule	`list_alert_rules`, `get_alert_rule`	`anectico alerts rules list`, `anectico alerts rules get <rule-id>`
Change, pause or delete a rule	`update_alert_rule`, `delete_alert_rule`	`anectico alerts rules update <rule-id>`, `anectico alerts rules delete <rule-id>`
Evaluate a rule now (not a dry run)	`evaluate_alert_rule`	`anectico alerts rules evaluate <rule-id> --yes`
Preview or review a metric watch	`list_metric_watches`, `get_metric_watch`, `preview_metric_watch`	`anectico alerts rules preview-metric`, `anectico alerts rules history-metric <watch-id>`
See what is firing now	`get_active_alerts`	`anectico alerts list`
Search alert history in every state	`search_alert_history`	`anectico alerts history`
Acknowledge, withdraw an acknowledgment, resolve	`acknowledge_alert`, `unacknowledge_alert`, `resolve_alert`	`anectico alerts ack <alert-id>`, `anectico alerts unack <alert-id> --yes`, `anectico alerts resolve <alert-id>`
Read who acknowledged an alert	`list_alert_acknowledgments`	`anectico alerts acknowledgments <alert-id>`
Silence matching alerts	`create_silence`, `get_silence`, `update_silence`, `delete_silence`	`anectico alerts silences create`, `get`, `update`, `delete`
Open an incident from an alert	`create_incident`	`anectico incidents create`
Read an SLO rule's error-budget burn	`get_slo_burn`	`anectico slo burn <rule-id>`

Confirmation-gated writes preview first in both MCP and the CLI. Review the proposed change, required scopes and any notifications, then repeat the identical arguments with confirm_token (CLI: --confirm-token). --yes only answers a local prompt; it cannot apply a server preview. If a version conflict or expired token requires a new preview, review it again. See MCP tools. The key needs alerts:read to see alerts and alerts:rules:read to read a rule. It needs alerts:rules:write to create, change, test or evaluate a rule, alerts:rules:delete to delete one, alerts:acknowledge and alerts:resolve to respond, and silences:read, silences:write and silences:delete for silences. Creating or testing a rule also needs evals:read.

For anectico alerts ack, supply a stable --idempotency-key. The first call returns the server preview. Repeat the same alert, note and retry key with --confirm-token from that preview to record the acknowledgement. The key also needs mcp:write.

Watch one canonical customer

Use an ordinary query rule with one canonical person UUID in one exact project. The server expands every current alias; an agent must not turn a displayed alias sample into the rule. A known UUID does not require profile access. Optional persons:read permits an identity lookup; use a person proof link only when an authorized lookup supplies it.

Replace the three placeholder UUIDs below. Choose the threshold from the customer's measured traffic, and choose a notification channel explicitly. This definition counts error occurrences; it is separate from a saved-product-metric watch.

{
  "project_id": "PROJECT_UUID",
  "kind": "query",
  "name": "customer-error-count",
  "description": "Error count for the selected canonical customer",
  "watch_target": {"kind": "person", "mode": "aggregate", "person_id": "PERSON_UUID"},
  "query": {
    "data_type": "errors",
    "time_range": "5m",
    "no_data_policy": "evaluate_zero",
    "filters": {},
    "aggregations": [{"function": "count", "field": "*", "alias": "error_count"}],
    "group_by": []
  },
  "conditions": [{"type": "threshold", "operator": ">=", "threshold": 3, "field": "error_count"}],
  "evaluation_interval": "60s",
  "evaluation_timeout": "30s",
  "for_duration": "0s",
  "recovery_duration": "60s",
  "severity": "warning",
  "is_active": true,
  "is_paused": false,
  "notification_channels": ["CHANNEL_UUID"]
}

project_id confines identity and telemetry. kind: "query" selects scheduled query evaluation. The technical name and description describe the operation without copying profile data. watch_target.kind, mode and person_id select one canonical customer, aggregate over their current aliases, and allow no other target selector.

data_type: "errors" chooses error occurrences. time_range: "5m" is the only lookback. filters: {} adds no further predicates; the server still applies the target and project. count(*) counts visible matching rows and alias: "error_count" names that measurement. group_by: [] produces one aggregate. The threshold compares that alias with >= 3, so exactly three breaches too. Multiple conditions are ANDed. Omit unused condition time_window and params; they do not define a second lookback. no_data_policy: "evaluate_zero" explicitly compares a completely resolved empty count population as zero. An unknown or erased identity is not such a population.

evaluation_interval requests a tick every 60 seconds; evaluation_timeout bounds a tick to 30 seconds. for_duration: "0s" fires without sustained-breach confirmation. recovery_duration: "60s" requires a measured clear condition for 60 seconds; a breach or unmeasured interruption breaks healthy confirmation. severity: "warning" sets urgency. is_active: true enables the rule and is_paused: false permits scheduling. Only the UUIDs in notification_channels receive direct notifications. Existing validators remain authoritative. Omit server-owned IDs, organization, version, author, clocks and evaluation and budget state on create.

Use MCP test_alert_rule with the full draft first. For creation, discover the door with list_write_actions, then preview create_alert_rule with that full rule, selected project and a stable idempotency_key. Present the exact query, target, admitted schedule and channels before approval. Confirm identical arguments with the returned confirm_token.

CLI uses the same proposal:

anectico alerts rules create --project PROJECT_UUID --file rule.json --idempotency-key CREATE_KEY -o json
anectico alerts rules create --project PROJECT_UUID --file rule.json --idempotency-key CREATE_KEY --confirm-token TOKEN -o json
anectico alerts rules get RULE_UUID --project PROJECT_UUID -o json
anectico alerts rules test RULE_UUID --project PROJECT_UUID -o json

Keep the creation key for a lost reply: replay returns the original stored rule. The mutation receipt's complete-object change contains that stored rule as untrusted JSON. Read it back with get_alert_rule, compare watch_target, version, full schedule, query, conditions and channels. For an update use the exact current version: MCP update_alert_rule takes expected_version and explicit fields; the CLI patch file contains version and changed fields. Both preview, confirm and refuse a stale version. Validation and creation prove configuration only. Verify scheduled evaluation and channel delivery separately; no rule proof page exists.

The errors recipe uses alerts:rules:write, alerts:rules:read, conservative catalog evals:read, errors:read, channels:read and alerts:read for lifecycle evidence. MCP also uses mcp:read/mcp:write. A known person UUID adds no persons:profile:read requirement. Alternative telemetry sources require their actual traces:read, logs:read or metrics:read.

Unknown, erased, unavailable, empty or oversized identity is unmeasured. It never drops the alias predicate, queries another customer, reports a healthy zero or advances recovery. Identity is rechecked after the telemetry read; a changed alias set withholds the verdict. Counts concern retained, visible data; upstream completeness stays unknown.

Person erasure defines removal of person-watch configuration, original replies, measurements and linked delivery copies, including its limits for in-flight requests and external recipients. Person export defines the separate archive boundary.

What your agent gets back

A rule read returns the query, conditions, timing and routing. It also returns kind (query or external) and, when a rule is bound to an evaluator or a source, evaluator_status and external_source. An evaluation result carries unmeasured and unmeasured_reason when the measurement could not be made. The sections below explain each field. An alert read gives its severity, state, start time and the identifiers that acknowledge and resolve use.

A message about one incident, one issue or one watched person carries a link to that proof page. Other alerts carry no link.

Open the proof

An alert has no proof page of its own. When you open an incident from an alert, the incident has one at /view/incident/:id. It is read-only, it names its project, and it has one action, Copy link. See Proof pages and Manage incidents and on-call.

Add a destination

In the Console, open Connections and notifications (/connections), choose the Notifications tab, and create a notification channel for direct rule delivery. Your agent can also create one with create_notification_channel (CLI: anectico channels create). Send a test notification before attaching it to a production rule. Add personal contact methods only when the rule routes through an escalation policy or on-call schedule. See Connect tools and notification delivery.

Available destinations depend on the connections enabled for your organization. Webhooks can cover destinations without a dedicated integration.

Create a rule

Ask your agent to build the rule definition, test it, and then create it. The definition sets:

the project and environment;
the signal or customer being watched;
the condition type, its specific settings, and the evaluation window;
how long the condition must remain true and then remain clear; and
the notification destination.

Optional evaluation settings and message templates are fields of the same definition. Ask the agent to read the rule back with get_alert_rule and to explain it in plain words before you rely on it.

If saving fails because of the selected project, see alert project errors.

The selected project is an evaluation boundary, not only an organizational label: the rule reads telemetry from that project and never combines matching data from another project. If a watched customer or account scope or grouped query cannot be measured completely, Anectico leaves that evaluation unmeasured rather than treating missing results as zero.

For a cohort target, each evaluation reads one exact membership generation of that cohort. The generation must be synchronized and its source evidence must remain readable. Anectico checks again after reading the scoped telemetry. If the cohort changes, source content is erased, or membership becomes unavailable, that evaluation fails without triggering or clearing alerts or changing their unmeasured markers. An unavailable cohort is never treated as an empty one. Refresh the cohort to produce a new valid generation before retrying; restoring old source rows alone does not make an erased generation usable again.

Cohort notifications retain the exact generation used for their rendered observation, including escalation steps and retries. Before each channel or personal-contact send, Anectico checks that original generation again. Missing, pruned or erased evidence stops delivery without a provider attempt. A temporarily unavailable evidence service delays delivery without consuming an attempt. Refreshing the cohort does not authorize an older queued notification. A complete later generation that removes a watched person starts the rule's normal recovery handling for that person's alert.

Use a sustained firing window for noisy metrics and a recovery window when one healthy evaluation should not close an incident. A breach during recovery restarts the healthy window. Use immediate notification and recovery only for conditions where one event is decisive, such as a critical customer error.

For a metric rule, select metrics, add a metric_name filter, and use the same aggregation and window you already validated with query_metrics (CLI: anectico metrics query <metric-name>) or in a saved dashboard. Metric rules and anomaly baselines count a redelivered point once and retain its first received value, just as metric queries do. Match the filters, grouping, reduction, and effective window when comparing an evaluation with a query. A threshold is the clearest choice for a hard service limit. An anomaly condition can preview an expected band when metric_name is present, but it remains an approximate signal rather than proof of a problem.

Evaluation timing

evaluation_interval sets the cadence. evaluation_timeout bounds one complete evaluation, including scheduled budget checks, target resolution, its query, historical comparisons and per-entity measurements. Scheduled and manual evaluations use the same bound. Set it between one second and the evaluation interval. If omitted, it defaults to half the interval with a minimum of five seconds: a 60-second cadence defaults to a 30-second timeout. A shorter client deadline and query safety limits still apply.

If measurement times out, the result carries unmeasured: true, unmeasured_reason: "evaluation_timeout" and null condition observations. An unfinished measurement supplies no firing or recovery verdict; active alerts gain an unmeasured since marker. A later complete measurement must establish recovery again. REST, MCP evaluate_alert_rule and anectico alerts rules evaluate preserve this reason when the caller can receive the result. A timeout after an alert write begins remains an error because the write may already have taken effect. Read the rule and active alerts after an interrupted request before considering a retry.

test_alert_rule and anectico alerts rules test validate the definition without running its query. A passing test does not establish that real evaluation finishes within its timeout.

Numeric measurements and no data

Numeric threshold, change and slo conditions require an explicit aggregation. A raw span or metric row has no declared ordering or population meaning, so that rule is refused at create, update and test. For “a span exceeded 2 seconds”, declare max(duration_ns) and compare its alias with 2000000000; an average or percentile answers a different question. Entity watches also require a declared reducer.

Set query.no_data_policy for these comparisons:

Policy	Meaning
`retain`	Missing numeric input leaves the evaluation unmeasured, with a `no_data` reason and `observed: null`. Existing alerts retain their last measured state; missing input cannot start or complete recovery.
`query.no_data_policy: "evaluate_zero"`	An empty count or sum population is explicitly compared as zero. This can fire a heartbeat or clear a count threshold. It describes the event population, not proof that the service is healthy.

query.no_data_policy: "evaluate_zero" requires only count or sum aggregations and conditions that select their aliases. Partial reducers such as average, maximum and percentile have no empty-input value and must use retain. Percentage-change and mixed baseline comparisons also use retain. A missing or unknown policy is refused; the comparison operator never chooses it. Absence, anomaly-only, issue-lifecycle and saved-metric watches have their own presence contracts and leave this field empty. Use an absence condition when silence itself should alert.

For a grouped numeric rule, retain withholds the whole tick if a previously active group has no current input. For query.no_data_policy: "evaluate_zero", only known active groups receive their declared zero; a missing identity never creates a new group. Entity watches with retain withhold all changes if any watched entity lacks input. A complete membership change can still remove a departed cohort member through the documented membership recovery contract.

Manual evaluation reports the normalized reducers, selected fields, stored rule version, policy and frozen window in result.query_result, alongside condition observations and row counts. MCP preserves this evidence in its mutation receipt; REST and CLI retain the native result. conditions_result_scope distinguishes a scalar, a representative group and the first watch entity. Entity-watch metadata includes every entity observation by index, without disclosing its identifying labels. A null observation is different from a measured zero. get_active_alerts also reports unmeasured since, so a retained breach is not mistaken for a fresh observation. Manual evaluation sends notifications and needs confirmation.

Grouped query rules

With query.group_by, each result group evaluates the complete set of conditions independently. For example, an error count grouped by service_name can fire for both checkout and payments in the same evaluation. Every condition must hold within the same group; conditions met by different groups do not combine into a breach. Each breaching group refreshes one alert whose identity uses the declared grouping dimensions. A changed count or rule name preserves that identity.

Change conditions match the same grouping dimensions in the current and previous windows. Anomaly baselines also stay within their group. If a present group has a missing, non-finite, or undefined required measurement, the whole evaluation is unmeasured and existing alerts stay open.

An evaluation supports up to 10,000 groups in each current or historical query. Exceeding that limit leaves the evaluation unmeasured. Narrow the query filters or choose fewer grouping dimensions so the complete result can be evaluated.

Issue-spike rules apply this bound independently to their current and prior issue-group sets. If either set exceeds 10,000 groups, the result is unmeasured with incomplete_group_result; it neither fires nor clears alerts. Existing alerts keep their last measured state and an unmeasured since marker until a complete evaluation returns. Narrow the filters before interpreting the result as recovery or a new spike.

Every issue_lifecycle rule uses query.data_type: "errors", including spike, new and regressed rules. Other sources are refused when creating, updating or testing a rule; an invalid stored definition cannot evaluate another signal as errors.

Spike rules apply every accepted error equality filter to both windows, including release, error_type, error_group_id and their supported aliases. An empty string is an exact match, not a wildcard. New or regressed issue-event rules support only project, issue-group, service and environment filters: their event payloads do not contain release or error type, so other filters are refused when creating, updating or testing those rules.

A numeric group absent from a complete result follows the declared no-data policy above. Without an expected-series inventory, an absence condition checks whether the whole query received data. Issue-spike rules retain their complete-set occurrence and recovery contract. Change and anomaly comparisons remain unmeasured when their required measurement is missing. Use a customer watch when absence must be checked for each known customer separately.

Detect one reporter going silent

For metrics, declare the expected identities in query.expected_series. Each identity names exactly the group_by dimensions. This example expects two hosts; a fleet of 90 uses 90 entries. Use one absence condition and one count() aggregation. Metric values, including zero, do not affect whether a reporter is present.

{
  "data_type": "metrics",
  "time_range": "5m",
  "evaluation_delay": "30s",
  "filters": {"metric_name": "system.cpu.utilization", "environment": "production"},
  "aggregations": [{"function": "count", "alias": "samples"}],
  "group_by": ["host"],
  "expected_series": {
    "lookback": "1d",
    "identities": [{"host": "host-01"}, {"host": "host-02"}]
  }
}

Put that query in a complete rule with conditions: [{"type":"absence"}], the project, schedule, severity and chosen notification channels. Create or update it with the same confirmed MCP actions and CLI commands above. Read the saved definition back before relying on it.

Metrics support host, pod_name and service_instance_id for grouping and equality filters. They read host.name, k8s.pod.name and service.instance.id, respectively; those attribute names are also accepted as field aliases. A point attribute overrides the same resource attribute, including an explicit empty value. Other metric attributes are not grouping fields.

The inventory supports 1–1,000 unique identities, each with exactly the declared dimensions and nonempty values of at most 256 bytes. Maintain the inventory when reporters are intentionally added or removed. A project_id dimension must match the rule's project. The inventory is explicit; observing a new reporter does not add it to the expected set. lookback must be at least time_range and at most seven days. A history query over more than 10,000 groups is unmeasured; narrow its filters.

Each missing reporter gets its own alert. query_result.series_presence reports expected_count, present_count, missing_count, unexpected_count, all_missing and per-reporter observations with labels, status, last_seen_at and last_seen_state. One missing out of 90 has missing_count: 1, all_missing: false; all 90 missing has missing_count: 90, all_missing: true. Unexpected reporters never substitute for an expected identity. A timestamp is the most recent observation within the bounded lookback. When none exists, last_seen_at is null and last_seen_state is not_observed_in_lookback; it does not mean the reporter never existed.

An unavailable or incomplete presence query returns unmeasured: true and presence status unknown, with null missing counts. It cannot fire or recover an alert. Observing a missing reporter again starts its normal recovery policy. Alerts also carry series_last_seen_at when known and series_last_seen_state. MCP evaluation receipts preserve the population evidence in their /query_result change; CLI JSON returns it under result.query_result.

Allow late observations and inspect current status

Ordinary windowed rules can set query.evaluation_delay, such as "30s" (default zero, maximum "168h"). The evaluation window ends that far before the attempt time. Comparison and anomaly baselines and the intake assessment use the same delayed boundary. This gives late data time to arrive; it is not proof that an exporter sent everything. Intake completeness is still disclosed, and known incomplete intake withholds the verdict.

Issue-lifecycle rules and current review signals refuse this setting. Product metric watches use their own evaluation_delay_ms; account objectives retain their own completeness contract.

Rule reads expose last_evaluation for the current definition, even if no alert has ever fired: state is clear, breaching or unknown, with reason, evaluated_at, rule_version, the effective window, delay and population counts when available. breaching describes the measured condition; pending and recovery durations still govern alert state. An edit clears this status until the new definition is evaluated. Always check its time: pausing a rule stops new evaluations. Timeouts remain unknown with reason evaluation_timeout. New or regressed issue-event rules are driven by their events and do not publish a query-evaluation status. Use get_alert_rule or list_alert_rules, or anectico alerts rules get RULE --output json.

For a measured evaluation, the response includes every breaching group in triggered_alerts. Its scalar conditions_result describes the first triggered group in that response, or a deterministic first group when none breach. Source permissions still govern access to identifying group labels.

Anomaly conditions

Each measured breach updates the open alert's severity, observed value, explanation, expected band, and anomaly score together. Severity can increase or decrease while the alert keeps its identity and history. If a firing duration is configured, the first notification uses the observation that completes confirmation. A severity change on an already-firing alert does not send another notification by itself. If you change a grouped rule from anomaly detection to a threshold, the same group's open alert clears its previous expected band and anomaly score on the next breach.

Renaming an anomaly rule updates its open alert's displayed name on the next measured breach. The alert keeps its ID, history, acknowledgment, and firing confirmation window; the rename does not indicate recovery or trigger another notification. A silence that matches the rule's name follows the updated name; match the rule ID to keep it independent of renames.

Anomaly detection needs a finite current measurement and at least one finite earlier point. Missing current data, insufficient history, or non-finite baseline values leave the evaluation unmeasured. A failed baseline query is also reported as an evaluation error. For metric rules, the current measurement comes from the newest completed interval; an interval still in progress is excluded, and an older point cannot replace a missing newest measurement.

For telemetry metric anomaly rules, params.interval accepts 1m, 5m, 15m, 30m, 1h, 6h, 12h, or 1d. 24h is equivalent to 1d; both use UTC day boundaries. Omitting the interval selects 1h. The anomaly condition scores the latest completed bucket and excludes the bucket still being collected.

Create, update, and test refuse unsupported metric intervals such as 2h or 10m, and explicit empty, malformed, zero, negative, or non-string intervals. If an older rule has an invalid interval, evaluation reports an error and leaves open alerts in their current state until the condition is corrected. Other query sources can sample any positive integer duration in seconds (s), minutes (m), hours (h), days (d), or weeks (w).

An unmeasured anomaly does not trigger, re-notify, or clear alerts. Open alerts keep their last measured state and an unmeasured since marker. Firing and recovery confirmation restart with the next appropriate measured observation. A measured non-anomalous result can recover normally, including a genuine zero count. In a rule combining several conditions, an unmeasured anomaly prevents the whole rule from changing alert state regardless of condition order. This also applies to aggregate customer watches.

The evaluation response sets unmeasured and an explanatory unmeasured_reason, leaves the anomaly condition's observed as null, and preserves the primary window's rows_scanned.

Percentage-change conditions

A generic query rule's change condition compares the current window with the earlier window selected by comparison_period. Its threshold is a percentage, not a fraction: (current - previous) / previous * 100. The sign of the previous value is preserved. Both windows read the condition's selected field. When field is omitted, a query with one aggregation uses that aggregation's output alias, including a generated or custom alias. Raw numeric comparisons are refused. Queries with several aggregations require an explicit field when saving or testing the rule. Both comparison windows keep the same selected field. For example, 100 to 120 is +20%, 100 to 0 is -100%, and either 100 to -100 or -100 to 100 is -200%. This is the contract for generic query rules, not saved-metric watches.

Both windows need a finite measured value, and the previous value must be nonzero. A zero previous value, including zero to zero, makes the percentage undefined. A missing measurement or a percentage too large to represent also leaves the evaluation unmeasured, not healthy and not breaching. A failed comparison query is reported as an evaluation error.

An unmeasured evaluation does not trigger, re-notify, or clear an alert. Open alerts retain an unmeasured since marker until measurement returns. Firing and recovery confirmation restart with the next appropriate measured observation; an undefined interval does not count as sustained evidence. In a rule combining several conditions, an undefined change prevents the whole rule from changing alert state, regardless of the conditions' order. These rules also apply to aggregate customer watches.

Threshold and SLO conditions also leave an evaluation unmeasured if their selected value is missing, NULL, nonnumeric, or nonfinite. A missing measurement cannot clear an open alert, and the evaluator never substitutes another aggregation's value.

The evaluation response sets unmeasured and an explanatory unmeasured_reason, leaves the change condition's observed as null, and preserves the current window's rows_scanned. A genuine measured zero percentage remains observed: 0. When a change fires, the condition's value and observed contain the percentage; the triggered alert's value remains the current signal value. Messages retain small nonzero values and use compact scientific notation for extreme magnitudes.

Query fields

Rule filters are exact, case-sensitive string matches. Use one of the fields below. Anectico validates and normalizes the query before saving the rule, so a typo or a field from the wrong signal returns a validation error instead of creating a rule that fails later. service is the portable field for every signal and maps to the stored service_name column. The canonical service_name spelling is accepted too. Logs also accept level for severity; traces accept operation for operation_name and status for status_code.

Signal	Exact-match filter fields
Errors	`service`, `service_name`, `environment`, `error_id`, `error_group_id`, `group_id`, `fingerprint`, `error_type`, `type`, `message`, `release`, `dist`, `user_id`, `user_email`, `user_ip_address`, `trace_id`, `span_id`, `session_id`, `distinct_id`, `request_method`, `request_url`, `platform`, `sdk_name`, `sdk_version`, `project_id`
Logs	`service`, `service_name`, `level`, `severity`, `severity_text`, `message`, `environment`, `trace_id`, `span_id`, `logger_name`, `host`, `pod_name`, `source_file`, `distinct_id`, `session_id`, `project_id`
Metrics	`service`, `service_name`, `metric_name`, `name`, `metric_type`, `type`, `environment`, `project_id`, `unit`, `description`, `aggregation_temporality`, `host` / `host.name`, `pod_name` / `k8s.pod.name`, `service_instance_id` / `service.instance.id`
Traces	`service`, `service_name`, `operation`, `operation_name`, `status`, `status_code`, `status_message`, `span_kind`, `environment`, `trace_id`, `span_id`, `parent_span_id`, `distinct_id`, `session_id`, `project_id`, `gen_ai_provider_name`, `gen_ai_model`, `source_operation`, `gen_ai_cost_source`, `agent_session_key`, `gen_ai_agent_name`, `gen_ai_agent_id`, `gen_ai_tool_name`, `gen_ai_response_id`, `gen_ai_finish_reason`
Agent steps	`category`, `operation`, `canonical_operation`, `source_operation`, `source_kind`, `agent`, `agent_observation_key`, `environment`, `trace_id`, `span_id`, `session_key`, `turn_id`, `distinct_id`, `source_schema`, `project_id`

count can omit its field. Numeric reducers (sum, avg, min, max, p50, p95, and p99) accept value for metrics, duration_ns and the numeric gen_ai_* token and cost fields for traces, or source_line for logs. Errors and agent steps support count only. Grouping is also signal-specific; a rule with a grouping dimension its signal does not support is refused when you test or save it. Filter objects, arrays, nulls, comparison operators, arbitrary free-form expressions, and unknown identifiers are rejected before the rule is persisted.

Alert on what an agent did

The Agent steps signal (agent_events in the API) is one row per step an agent took inside a run — a guardrail decision, a tool call, a memory write. Use it for questions a run-level view cannot answer: how often a guardrail is denying, how much a given agent is calling a particular tool, or whether an agent has stopped taking steps at all.

Agent steps support count and no other reducer, and every other reducer is rejected when you save the rule rather than silently returning zero. The reason is worth stating plainly: a step does not yet report its own duration, so an average over it would compare 0 against your threshold on every evaluation and page you forever while nothing was wrong. Those reducers become available in the same release that starts reporting the numbers behind them.

Two shapes cover most of what this signal is for:

A rate ceiling. Count steps in a window, filter to category (and optionally agent), and set a > threshold — "more than 10 guardrail denials in 15 minutes".
A silent fleet. Count steps and set < with a threshold of 1, or use an absence condition. Either fires when the window contains no agent activity at all.

Because the reducer is count, the result field on the condition is the alias you gave the count. Naming a raw field there instead is rejected when you save.

If you group an agent-step rule by distinct_id, that label is treated as person-identifying, the same way user_email is on an error rule: it is kept out of the delivered notification entirely, and reading it back from the API needs permission to read agent runs, not just alerts.

For a heartbeat, scheduled job, or other expected signal, choose an absence condition. It checks whether the filtered query matched any source rows during query.time_range, which is the sole lookback. Raw, grouped, and aggregate queries are supported; a real aggregate value of zero still counts as present data. The condition does not use an operator, threshold, result field, or its own time_window.

Alert on an evaluation objective

An SLO condition alerts on the scores an evaluator produces, rather than on raw telemetry. Use it to hold a quality objective — "faithfulness stays at or above 0.9", "task completion does not fall below 0.95" — the same way a threshold holds a latency or error-rate objective.

An SLO condition names three things:

Setting	Meaning
Evaluator	The evaluator whose scores the objective is measured on.
Version	An exact evaluator version. Required, and never "latest".
Mode	Which kind of run counts: `production`, `experiment`, or `simulation`.

Set the operator and threshold to the objective itself, aggregate numeric_value over the window, and filter or group on evaluator_id, evaluator_version, mode, metric_name, subject_kind, label, environment, or project_id.

Three rules follow from what an SLO is for, and each one is enforced when you save:

The version is exact. An objective bound to a moving evaluator is not an objective: the same release would pass on Monday and fail on Tuesday with nothing in the record explaining why. A version of 0 or a missing version is rejected.
The evaluator must already exist, and must be reproducible. Saving the rule checks that the evaluator and version are registered in the project, that the evaluator names the dimension it scores, and that its configuration is fingerprinted so a score can be traced back to exactly the rule that produced it. An evaluator that cannot supply that is refused when you save the alert, naming what is missing — not at 3am when the alert fires on numbers nobody can reproduce. If the check cannot be completed, the rule is not saved and you can retry; it is never stored unchecked.
One objective per rule, on its own. An SLO condition cannot be combined with another condition, and evaluation scores cannot be read by any other condition type. This keeps the numbers an alert compares scoped to the exact evaluator, version and mode it declared.

Because an evaluator belongs to a project, an SLO rule must be scoped to a project rather than to the whole organization. Creating one also requires permission to read evaluations.

The exact evaluator reference is checked again immediately before every scheduled evaluation. The rule's evaluator_status reports valid, invalid, or unavailable, with evaluator_checked_at and a safe evaluator_error when something is wrong. An invalid reference does not run the score query and is never represented as a score of zero. Updating the rule clears the previous status until the new definition is checked.

The objective is evaluated on when each score was produced, so a score always lands in a window the alert has yet to check, however long the evaluation itself took.

The evaluator's written explanation is recorded content, not an alert dimension. SLO evaluation selects only the declared score metadata and numeric fields, so an explanation is never copied into an alert label, alert read, or Slack, email, and webhook notification. Read the evaluation result itself with the separate content permission and current project policy when you need that reasoning.

Rules fed by an external source

Most rules are query rules: Anectico evaluates their query on a schedule. A rule can instead be an external rule, bound to exactly one source such as a monitor. An external rule has no query, conditions or evaluation schedule; its source reports each failure and recovery, and every failure episode becomes one alert however many times the source repeats it. Rule reads return kind (query or external) and, for an external rule, its external_source.

external_source.concurrent_occurrences says how episodes of one source relate. A monitor has one failure episode at a time (false): a new failure episode replaces the open alert. A service in your response catalog receives many independent incoming alerts at once (true): each distinct alert stays open as its own alert, and its recovery closes only that alert.

An external rule always pages through an escalation policy. Creating one, pointing it at another policy, or re-activating it is refused unless the policy exists in the same project, is active and has at least one level. Rule reads also return routing_coverage for an external rule, so a policy that is later deleted, deactivated or emptied shows as missing_policy, policy_inactive or policy_has_no_levels instead of silently paging nobody. Query, condition and schedule fields cannot be edited on an external rule, and it cannot be tested or evaluated on demand.

Test before enabling

Ask your agent to test the rule and show you the message it would send. Confirm the message contains enough context to begin an investigation: the condition, severity, project, environment and time. A message about one incident, one issue or one watched person in a project also carries a link to that proof page. Other alerts carry no link.

Respond to an alert

Ask your agent what is firing. get_active_alerts (CLI: anectico alerts list) lists the alerts that are open now. search_alert_history (CLI: anectico alerts history) includes resolved alerts and takes a window, such as --since 7d. Name a rule to see how often it fired, or one alert to read just that alert. These reads cover all environments in the project; use a rule's query filters when evaluation should cover a particular environment. A list is cursor-paged: the agent passes next_cursor back unchanged to read the next page, and a count of loaded rows is not a total of all alerts.

An alert read shows its observed value, lifecycle state, environment, labels, and current rule. The current rule may have changed since the alert fired.

For active anomaly alerts, the read shows the actual value, expected interval, and labelled detector score from the latest measured observation. Labels and annotations are custom metadata; annotations do not override detector evidence. If the rule changes to a plain threshold, the anomaly detail disappears on the next read.

When a rule cannot obtain a required measurement, the alert read says the current measurement is unavailable and gives an unmeasured since time, which is the first consecutive measurement gap. The retained value, expected interval, and detector score are last measured evidence. The alert stays open and its response actions remain available. The warning clears when measurement returns, including a genuine zero value.

Detector scores are specific to the detector, rather than a universal statistical unit, probability, or root-cause diagnosis. The Likely real marker remains a promotion signal; findings without it are not dismissed.

Acknowledge when someone takes ownership (acknowledge_alert).
Open the linked customer, Issue, trace, or service evidence.
Create an incident when coordination or a durable case file is needed (create_incident).
Resolve after the condition has cleared and follow-up is recorded (resolve_alert).

Resolving is final for that alert. It records who resolved it and when, and nothing reopens it — if the condition breaches again you get a new alert with its own history, rather than the old one returning to firing. This holds even when a check of the same rule is already running as you resolve.

Automatic query recovery records resolution: recovered in alert history when the configured healthy window completes. It preserves any acknowledgment and records no human resolver.

Acknowledging or resolving an alert stops its escalation and closes its linked incident, whichever surface you use: the API, MCP, the CLI or Slack. A manual resolution is recorded as resolution: manual: it ends the response, and says nothing about whether the watched source has recovered. For an external rule, further failures of that same episode do not reopen it; a new failure after the source recovers opens a new alert. Every page — a first notification, a retry or the next escalation step — is checked against the alert's current state just before it is sent, so an alert that has been acknowledged, resolved or silenced sends nothing more. A page already handed to the delivery provider when you acknowledge may still arrive.

Use a silence for planned maintenance or a known temporary condition. Give every silence an owner and expiration time. Matching alerts show silenced with that provenance; acknowledging one keeps the responder-owned state until the acknowledgment is cleared. If maintenance ends while the condition still breaches, the alert returns to firing and notifies once on its next evaluation. An external rule's alert resumes within seconds of the silence ending or being deleted early, because its source does not re-evaluate. A resumed alert that had already paged notifies once and does not open a second incident; one that fired straight into the silence pages for the first time. The same holds for a scheduled silence that starts after an alert fired but before its first page went out: the page is held and sent once when the silence ends. An acknowledgment or recovery that happens first wins, and nothing resumes.

External monitors have a second, source-side suppression: a maintenance window covering a monitor stops a new failure episode from opening at all, rather than silencing an alert that already exists. The check still runs and every round is still recorded; when the window ends, a monitor that is still failing pages once. Use a silence when you want an already-firing alert to stop paging; use a maintenance window on the monitor itself when you know it is about to fail (a deploy, a planned outage) and do not want the first failure to open an episode in the first place.

Deleting a rule keeps its alert history and acknowledgments readable: the rule disappears from rule lists and stops being evaluated, its name can be reused at once, and any of its alerts still active are resolved with resolution: rule_deleted, which stops their paging and closes their incidents. An alert whose source stopped reporting it without recovering — a failing monitor owned by a service that was deleted or moved to another route — is resolved with resolution: withdrawn, which likewise stops its paging and closes its incident without claiming the check recovered.

A silence has matchers on labels (exact match or full-match regular expression), a start time and an end time. The MCP tool previews the matching scope before it creates the silence; read the preview before you confirm. Silences apply across the organization wherever all labels match; the selected project alone does not restrict them. Pausing notifications does not itself resolve an alert.

Tune based on outcomes

Review alerts that fired without action and alerts that should have fired but did not. Change the condition, window, or routing rather than training the team to ignore noise.

Scheduled work budgets

get_alert_rule and anectico alerts rules get show the rule's budget origin and latest evaluation budget state. Work changes move the origin to the updater; a rename or pause keeps it. A missing credential moves new evaluations to scheduled work in the workspace.

Each accepted scheduled evaluation counts one query operation, including all its source queries. Retries do not add a charge. Budget limits never stop an alert evaluation. It takes no concurrency slot. Usage marks any overage as over budget by scheduled alerting. Unavailable limits are marked unchecked. See Scheduled work.

Alert-rule and silence commands show complete server proposals before applying. Silence proposals include every matcher and the exact window. Creates use stable retry keys and return their original result on an identical retry. Edits refuse a changed before-state. Rule retirement resolves active alerts and can resolve linked incidents and send recovery notifications to the previewed audience; queued delivery selects recipients at send time.

A metric watch automatically pauses when its admitted key or source permission is lost. Read pause_reason and paused_at in its definition and the control-state conclusion. The workspace receives a control notice; a queued notice is not proof of delivery. Restore authority and explicitly preview and confirm resume. An operator can pause without source access. Stopping a watch leaves an already active alert unresolved.

Resolving an alert can close linked incidents and send recovery notices. Both MCP and the CLI show the full current alert, requested note and recovery audience first. After approval repeat identical CLI arguments with --confirm-token. An unreadable active state refuses a confirmation token. Resolution has no inverse.

Account objective evidence and privacy

The objective guide owns the source-lifetime, current-access and erasure rules for account measurements and original triggers.

Metric watch transition meaning

Product metric watches require source_loss_grace_periods: 2. A breach and a recovery require measured confirmation; unavailable evidence interrupts it. An exact removed source or sustained explicit unavailable coverage is source loss, including before any alert fired. Restored availability alone is not recovery. A silence suppresses breach delivery without changing that measured fact; its expiry, acknowledgment, manual closure and resume are not new measured transitions. Receive these transitions through agent event delivery, and use its authorized watch-event reads to follow an occurrence.

Exact account request objectives

Use Account request objectives to prepare a paused account proposal, read its current activation limit, inspect fresh burn evidence or follow an original trigger. The request population is defined in Observed customer request reliability.

Versions and approval

Deleting or evaluating a saved rule requires its current integer expected_version; saved-rule validation requires it too. Use --expected-version on the CLI. Editing or deleting a silence requires expected_version equal to its current updated_at. A stale version returns a conflict and changes nothing; read the current object and obtain a new preview. An admitted test evaluates the selected definition rather than following later edits.