From measurement to alarm

Written for whoever carries the pager. A failure travels from a single check to a ringing phone through a fixed sequence of steps, and each step also filters out noise, so the phone rings for confirmed outages.

The chain

check → region verdict → alarm quorum → incident → escalation → acknowledgement
                                                                      ↓
                                        recovery quorum ← re-alerting if unresolved

1. Detection

The eleven regional probe types run on a schedule from the regions in your plan. Agent and heartbeat monitors report directly and skip the regional vote. Your plan sets the shortest interval, because check volume drives the cost:

Plan	Shortest interval
Free	300 s
Pulse	60 s
Sentinel	30 s
Command	15 s
Enterprise	10 s

The web form offers 30 seconds to 1 hour. The REST API and MCP set every interval down to the floor, including 15 s and 10 s.

A single failed check is one region’s signal on the public internet. The next two steps turn signals into a confirmed outage.

2. Region verdict: the first quorum

Each region can hold more than one node. When it does, a region verdict needs 75% of that region’s fresh votes, rounded up, and at least one. The current topology runs one active node per region, so this layer presently resolves as 1/1. Additional nodes per region are planned.

Nodes voting	Needed to agree
1	1
2	2
4	3
6	5
10	8

This keeps one flaky node in Frankfurt from speaking for Europe. The rule is computed per region as a fixed property of how a region forms its verdict, separate from the policy you configure.

3. Alarm quorum: the second quorum

Regions with a verdict are then counted against your alert profile. This is the quorum you configure, separate from the region verdict above:

• a mode and a value: how many regions must agree before an alarm is confirmed,
• a consecutive count: how many passes in a row must hold, so a single bad minute pages nobody.

Below the threshold, the interface shows the service as degraded. Regions are reporting, and the quorum has not confirmed anything yet. That distinction is the point of the quorum: a suspicion is not an outage.

Recovery has its own mode, value, and consecutive count. You usually want more evidence before you declare a service healthy again than before you declare it broken, and separate settings give you exactly that.

4. Incident

Once the alarm quorum holds, an incident opens. It carries the first region that reported, the time it did, and every state change after that. Everything downstream (status page, availability figures, post-mortem) hangs off this one record.

5. Escalation and on-call

On-call rotations and the escalation ladder are included from Sentinel. On every plan, incidents open and notify your team.

The escalation runs on a tick, in stages with configurable delays (stage 1, then stage 2). Each stage widens the circle of people contacted. Two behaviors are worth knowing in advance:

• A coverage gap is itself a condition. If nobody is on duty when an incident opens, Perstat records the gap, and the end of the chain applies immediately.
• Escalation follows the current plan. The tick re-checks the on-call entitlement per organization on every run. After a downgrade below Sentinel, running escalations stop advancing, and existing incidents stay open and visible.

Personal SMS is included from Pulse, on-call SMS and voice from Sentinel. Successful personal and on-call deliveries share these quotas:

Plan	Shared SMS + voice / 24 h	Shared SMS + voice / month
Free	0	0
Pulse	20	100
Sentinel	50	500
Command	200	2 000
Enterprise	unlimited	unlimited

6. Acknowledgment: taking ownership

Acknowledging takes ownership. It does two things at once:

1. It stops the escalation. No further stage fires for this incident.
2. It puts your name on it. The incident records who acknowledged and when, and the team sees it.

That is the whole design: the escalation ladder exists to find someone, and it stops the moment someone takes responsibility. Acknowledge the incidents you are going to work on, because an acknowledgment silences the mechanism that would otherwise find somebody else.

Acknowledging leaves the incident open. The service is still down, and the team now knows that a person is on it.

7. Re-alerting

An acknowledged incident that stays open comes back. Perstat reminds the owner, because an owner can get pulled into something else while the outage carries on.

8. Recovery and closure

When the recovery quorum holds, the incident closes, and the status page and availability figures follow. Incidents resolved by other means (manually resolved, discarded, monitor disabled) have their escalations cleaned up on the same tick, so paging always ends with the incident.

What this chain guarantees

• Pages come from confirmed failures. Two quorums stand between one node’s signal and your phone.
• An outage is a measured failure. A monitor without fresh measurements waits for data instead of alerting.
• Acknowledgment stops the escalation, and the incident comes back if it stays open.

Next

• Quorum and regions: how regions are chosen and what each one measures.
• Incidents and post-mortems: what happens to an incident after it closes.
• Plans and limits: the full capability matrix.