# Manage incidents and on-call

> Turn an active customer-impacting problem into an owned, traceable response.

Canonical page: https://anectico.com/docs/respond/incidents-and-on-call/


Use an incident when a problem needs coordination, explicit ownership, or a durable timeline. A
single Issue can often be handled directly without declaring an incident.

## Prepare on-call

Before relying on escalation:

1. add personal contact methods in **Settings → Notifications**;
2. create an on-call schedule;
3. add rotations and any temporary overrides;
4. create an escalation policy; and
5. send a test through the complete route.

Keep schedule ownership current. An escalation policy cannot compensate for stale contact methods.

## Build an on-call schedule

Open **Respond → On-Call** and create a schedule with the timezone where its handoffs should be
interpreted. Add members to a layer in rotation order, choose the rotation length and start time, and
use active-window restrictions when a layer covers only particular days or hours.
Rotation lengths must be positive whole seconds representable by the server; fractional or
overflowing protobuf durations are rejected. A layer has no on-call member before its start time.

Schedules and overrides belong to one project. Anectico accepts only current members of the same
organization in rotation layers and overrides; if membership cannot be verified, the change fails
closed. An inactive schedule never resolves an on-call person and cannot page through escalation.

Layers stack: a later active layer overrides an earlier one. Review the preview, then check the live
**on call:** badge because the preview does not apply restrictions or temporary overrides.

Rotation anchors, override boundaries, and the boundary inspector interpret `datetime-local`
values in the schedule or layer's selected IANA timezone—not the browser timezone. Anectico rejects a
wall time that does not exist during a spring-forward gap or occurs twice during a fall-back fold;
choose an unambiguous time instead. The UI shows the timezone name and numeric offset alongside the
exact UTC instant, and the boundary inspector accepts a fixed instant for repeatable handoff checks.
Switching projects clears the previous project's rows while the selected project refetches.

A new layer leaves its layer timezone blank and follows the schedule timezone. Changing that
schedule timezone preserves the wall-clock value and recalculates its UTC instant. Entering an
explicit layer timezone later does the same reinterpretation; it never resets a rotation start that
you already entered. Moving from the rotation-start field commits its visible value before the
timezone change is applied.

Use an override for a dated coverage change such as leave or a shift swap. Set both start and end;
the underlying rotation resumes when the override expires. Override windows are half-open
(`[start, end)`) and may touch, but overlapping windows on the same schedule are rejected so
coverage never depends on an ambiguous winner.

## Create escalation policies

Open **Respond → Escalation Policies** and create ordered levels. Each level waits its configured
delay, then notifies one or more channels, members, or on-call schedules. Every level needs at least
one target.

Use later levels to broaden the audience when an alert remains unacknowledged. Repeating escalation
requires a repeat interval and a maximum escalation count. Attach the policy to an alert rule and
test the complete route. If a policy is deleted, rules using it fall back to their own notification
channels. Once an alert starts escalating, its levels, delays, repeat settings, and targets are
frozen for that run; editing, shrinking, deactivating, or deleting the policy affects only future
alerts. Acknowledgement and resolution are terminal for the running escalation.

Escalation policies belong to the selected project. Policy lists, details, edits, deletes, alert-rule
pickers, and on-call schedule targets are scoped to that same project; a policy ID from another
project is treated as not found. The editor disables saving when members, channels, or schedules
cannot be verified rather than accepting a free-text target ID.

## Declare an incident

When an Issue triggers the response, open that Issue and choose **Declare incident**. Anectico carries the
Issue link into the new case file. For a manual incident, open **Respond → Incidents**, choose
**Declare incident**, and enter a short customer-facing symptom as the title.

The dashboard's manual declaration starts with the title and default severity. Use the CLI when you
need to set the description, severity, and primary signal in one operation:

```bash
anectico incidents create "Checkout unavailable" \
  --description "Customers cannot complete payment" \
  --severity 1 \
  --link-type error_group \
  --link-ref <issue-id>
```

Supported incident links are `alert`, `error_group` (Issue), and `anomaly`. Add another supported
signal with `anectico incidents link <incident-id> --type <type> --ref <signal-id>`. Use the linked Issue
to reach affected customers, traces, replays, releases, and external tickets.

Anomaly findings are promoted automatically. The case file preserves the detector explanation,
expected band, observed value, score, non-content labels, and detection time even if the original
finding is later removed. Labels named after recorded-content conventions and the detector's opaque
series key are not copied into the case title or durable snapshot. Opening the same finding from the
UI takes you to that case instead of creating a duplicate.

## Work from the case file

Record material changes as timeline comments:

- what customers are experiencing;
- current impact and affected scope;
- evidence supporting the working cause;
- mitigation and rollback actions; and
- decisions that change the response.

Use Customer Detective to gather cited evidence, then link the relevant underlying objects rather
than pasting an unsupported summary.

Assign an owner, attach evidence (notes, URLs, logs, traces, metrics, replays, screenshots, or
documents), and keep the retrospective in the case file. Comments and evidence from teammates are
append-only, so simultaneous responders do not overwrite each other. If somebody changes incident
metadata or lifecycle state while your editor is open, Anectico reports a conflict and asks you to
refresh instead of silently replacing their update. The notification receipts section records each
durable incident email/Slack/webhook delivery attempt and its final status.

The CLI exposes the same strict case file. Read it immediately before a mutation, then reuse the
returned version and one stable retry identity:

```bash
anectico incidents view <incident-id>
anectico incidents watch <incident-id> --timeout 30m --output json
anectico incidents acknowledge <incident-id> \
  --expected-version <version> \
  --idempotency-key ack-<incident-id>-v<version>
anectico deliveries list --incident <incident-id>
```

The incident watcher emits a current snapshot followed by newly appended
timeline events as newline-delimited JSON. Each record carries an opaque resume
cursor. After interruption, restart with `--cursor <last-processed-cursor>`;
Anectico verifies the cursor against the same project, incident, event ID, and
database sequence before emitting anything newer. It fails if that resume point
is absent or changed rather than silently skipping chronology. Temporary
transport, HTTP 408/429, and 5xx failures reconnect with bounded exponential
backoff and put degraded/recovered state on stderr, leaving stdout safe for an
NDJSON consumer. Existing timeline entries are skipped unless you pass
`--include-existing`; that fresh-history option cannot be combined with a
resume cursor.

Keep the same idempotency key when retrying the exact same request. This holds for
acknowledge, resolve, and reopen, and equally for adding a comment or attaching evidence.
Anectico returns the first successful response and does not add a second timeline,
evidence, or notification event.

Requests are compared by value, not by the exact bytes you sent, so re-serializing an
identical retry is safe: a different key order, `1e2` where you first sent `100`, `2.0`
where you first sent `2`, or different whitespace all still replay. A number sent as a
string is a real change, and so is any change to a value a reader of the case file would
see.

Change anything about the request and reuse the key, and you get `409 Conflict` with
nothing written — a different comment body, a different evidence artifact, different
transition fields, or the same key sent by a different user or API key. A new key with an
old version conflicts too. Nothing is silently replaced, and a retry can neither hide
another responder's change nor hand you their original response. If you get this 409,
your write did not happen: read the incident back and retry from its current state. Which
key to use depends on which conflict you hit — reuse of a key with changed content needs a
new key, while a stale version needs the same key with the current version. The message
says which.

For an organization-level credential, set `--project` or `ANECTICO_PROJECT`; project-scoped credentials
remain pinned to their signed project. MCP clients can use the `get_incident` tool for the
equivalent strict read, and reach `resolve_oncall` and `list_notification_deliveries` through
`execute_read_action`.

Evidence links can be Anectico root-relative paths or absolute HTTP(S) URLs. Anectico rejects
protocol-relative links and executable schemes such as `javascript:` and `data:`.

## Close deliberately

Resolve the incident when customer impact has ended and monitoring confirms recovery. Capture any
remaining follow-up work in the linked ticket system. Reopen the incident if the same impact returns
before the response is complete.

## Verify the response path

Schedule a safe test window, confirm the expected member is on call, trigger a test alert, and verify
the correct escalation levels stop after acknowledgement. Test again after changing rotations,
provider connections, or contact methods.

- [Connect tools and notification delivery](/docs/manage/connections-and-notifications)
- [Alerts or integrations are not delivering](/docs/help/alerts-and-connections)

Escalation level delays and repeat intervals must be valid protobuf durations representable by the
server. Severity filters and weekly restriction types are normalized before storage so accepted
case and surrounding whitespace do not change matching behavior.
