Skip to content
Console
Browse documentation
Guide

Incoming alerts

Let Prometheus Alertmanager or your own signed sender page a service in your response catalog.

On this page

An incoming-alert source lets an external system report alerts for a service in your response catalog — Prometheus Alertmanager, or anything that can sign a JSON request. Each source belongs to one project and pages through whichever service its events name; a service still pages exactly the way any other alert does, through its escalation policy.

Ask your agent

"Add an Alertmanager incoming-alert source called prod-alertmanager. Use checkout as the fallback service key."

"Show the last deliveries of prod-alertmanager and tell me why the latest one did not page."

Job MCP tool or action CLI command
Add a source create_incoming_alert_source anectico incoming-alerts create --provider alertmanager --name <name>
List sources or read one list_incoming_alert_sources, get_incoming_alert_source anectico incoming-alerts list, anectico incoming-alerts get <source-id>
Change the name, fallback service key or service label update_incoming_alert_source anectico incoming-alerts update <source-id>
Read the last delivery outcomes list_incoming_alert_deliveries anectico incoming-alerts deliveries <source-id>
Replace the signing secret rotate_incoming_alert_secret anectico incoming-alerts rotate-secret <source-id> --yes
Revoke a source revoke_incoming_alert_source anectico incoming-alerts revoke <source-id> --yes
Check where a service key would page preview_alert_routing anectico service-catalog preview --service-key <key>

Reads need connections:read. Adding, changing and rotating need connections:write. Revoking needs connections:delete. The MCP write tools preview first; see MCP tools for the confirm step.

Add a source

Choose the provider (alertmanager or generic_json), give the source a name, and optionally set a fallback service key: the service an event pages when it names no service, or one that does not exist. Your agent does this with the tools above. You can also do it yourself: in the Console, open Connections and notifications (/connections) and use the incoming-alert source card on the Connections tab. See the Console guide.

A source has one signing secret at a time. The secret is shown exactly once, in the result of the create call (or the rotate call). Anectico stores it encrypted to verify deliveries. Reads do not return it. If the agent made the source, tell it to put the secret straight into the sender's configuration or your secret store, and not to repeat it in chat or logs. If the secret is lost, rotate it to issue a new one. Rotating replaces the secret immediately: the old one stops verifying deliveries, and anything still configured with it starts failing until you update it.

The delivery endpoint is POST https://<your Anectico host>/api/v1/incoming-alerts?source=<source id> — one URL per source, with the source's id in the query string (it identifies the source, not a secret; the signature is). The source id is in the create result and in every read of the source.

What your agent gets back

A source read returns the provider, the project, the status, the fallback service key and the last delivery. It never returns the secret. The create and rotate calls return the secret once. A delivery read returns outcomes only, not the request body.

Open the proof

Incoming-alert sources have no proof page. When a delivery opens an alert and an incident, the incident has one: see Proof pages.

Prometheus Alertmanager

Point an Alertmanager receiver at your source's URL with the secret as a bearer credential:

receivers:
- name: anectico
  webhook_configs:
  - url: https://<your Anectico host>/api/v1/incoming-alerts?source=<source id>
    http_config:
      authorization: {type: Bearer, credentials: <secret>}

Anectico accepts Alertmanager's webhook payload version 4 only. Each alert's fingerprint is its dedup key, status (firing or resolved) is its state, and the label named by service label (default service) supplies the service key — set a different label name on the source if your alerts carry it under another name. The alert's summary or description annotation becomes the incident summary, falling back to alertname; generatorURL and a runbook_url annotation become reference links. A notification carries 1-100 alerts, each applied in order.

Generic JSON

Send one JSON object per request, signed with HMAC-SHA256 over "<unix-timestamp>.<request body>" using the source's secret, as two headers:

X-Anectico-Timestamp: 1717000000
X-Anectico-Signature: sha256=<hex-encoded HMAC>

The timestamp must be within 5 minutes of when Anectico receives the request — a replayed or clock-skewed request is refused, signature or not. The body accepts exactly these fields; an unrecognized field is refused rather than silently ignored:

{
  "dedup_key": "db-cpu-checkout",
  "state": "firing",
  "service": "checkout",
  "severity": "critical",
  "summary": "CPU above 95% for 5 minutes",
  "started_at": "2026-09-25T10:15:00Z",
  "observed_at": "2026-09-25T10:20:00Z",
  "links": ["https://your-dashboard.example.com/d/cpu"]
}

dedup_key identifies one continuing problem; started_at marks when it began. A resolved event needs resolved_at instead of observed_at, at or after started_at. links is optional, at most 10 absolute http:// or https:// URLs. service names the service key this event pages (falling back to the source's fallback key when empty or unknown); severity and summary are free text carried onto the alert.

Re-firing after recovery

A dedup_key that fires again with a new started_at after its previous occurrence recovered is a new, independent alert — a genuine recurrence pages again rather than silently reopening the old one. While one occurrence is still open, a repeated firing update refreshes it, and its own recovery only ever closes that occurrence: an older, already-superseded recovery arriving late changes nothing. A service can have many distinct alerts open at once, each following this same rule independently — one incoming source firing for several unrelated problems does not make one alert supersede another.

What happens to a delivery

Outcome Meaning
accepted A new alert opened.
updated An existing open alert was refreshed.
resolved The named alert's occurrence closed.
duplicate An identical retry of an already-applied event; nothing changed.
stale Older than what this occurrence already recorded; nothing changed.
unknown_service No live service has this key, and no fallback applies. Nothing paged.
service_not_routable The service exists but its alert rule is not fully set up yet. Nothing paged.

Each source keeps its last 100 delivery outcomes — never the request body — so you can confirm deliveries are arriving and see why one did not page. Ask your agent to read them with list_incoming_alert_deliveries (CLI: anectico incoming-alerts deliveries <source-id>).

Refusals

Response Meaning
401 Unauthorized The source is unknown, its signature does not verify, or it was revoked. These read identically on purpose, so a caller cannot use the response to guess which source ids exist.
413 Payload Too Large The request body exceeds 256 KiB.
429 Too Many Requests More than 120 requests in the last minute from this source. Retry after a pause.
400 Bad Request The body could not be read as the documented shape for this provider.
503 Service Unavailable A transient failure; safe to retry the identical request.

A 401, 413 or 400 is not retried automatically by Anectico — fix the request and resend it.