Concepts
Five things to understand, and what an alert does from arrival to resolution.
Services
A service is a thing that can break: an API, a worker, a database. It is the unit alerts are addressed to, and it points at one schedule, which decides who gets woken.
Integrations
An integration is a way in. Each one has its own URL containing its own token, so you can give a different one to each monitoring tool and revoke it without touching the others.
Every service gets a default integration when you create it. The token is shown when it is created and when you rotate it. If you did not keep it, rotate to get a new one. We cannot show you the old one again.
Schedules
A schedule is who is on call. It has a timezone and one or more levels:
- Within a level, several people means a rotation: one person at a time, taking turns.
- Between levels means escalation: if level 1 does not acknowledge in time, level 2 is paged next.
Those are the only two shapes today. Paging two people simultaneously is not yet expressible. Adding a second person to a level alternates between them rather than alerting both.
An override replaces whoever the rotation would have picked, for a window you choose. That is the mechanism for cover, holidays and swaps.
Priority and urgency
Two separate axes, deliberately.
| Axis | Values | What it decides |
|---|---|---|
| Priority | critical, high, medium, low, info | How bad it is. A label for humans, and how alerts sort. |
| Urgency | high, low | Whether it may break through a silenced phone. |
They are separate because "serious" and "wake someone up now" are not the same claim. A critical problem found on a Sunday that can wait until Monday is high priority and low urgency. A low priority certificate expiring in an hour is the reverse.
Leave urgency unset and the priority decides it.
The life of an alert
| State | What it means |
|---|---|
notifying | Live, and we are paging whoever is on call. Reminders repeat until somebody acknowledges. |
acked | Somebody has it. Reminders stop, escalation stops. |
resolved | Over. Either a person resolved it or your monitoring sent a recovery. |
unreachable | We ran out of people. Every level was paged, the chain repeated, and nobody acknowledged. |
unreachable is the outcome to design against. It does not mean the alert was unimportant. It means the alert was raised, delivered, and nobody answered it. If you see them, the schedule is wrong, not the alerting.
The ledger
Every state change writes a row to an append-only ledger, in the same database transaction as the change itself. Nothing edits or deletes those rows, so "what actually happened, and when" has one answer and it cannot be quietly rewritten afterwards.
You can read yours on any alert page, or under Audit. It is where troubleshooting starts.