From measurement to alarm

This page is for whoever gets woken up. It describes the path a failure takes from a single check to a phone ringing, and - just as important - where the system deliberately stays quiet.

Everything here describes what ships today.

The chain

check → region verdict → alarm quorum → incident → escalation → acknowledgement
                                                                      ↓
                                        recovery quorum ← re-alerting if unresolved

1. Detection

This detection path applies to the eleven regional probe types. They run on a schedule from the regions included in the plan. Agent and heartbeat monitors report directly and do not use regional voting. The shortest interval you can set for scheduled probes depends on your plan, because check volume is the cost driver:

Plan	Shortest interval
Free	300 s
Pulse	60 s
Sentinel	30 s
Command	15 s
Enterprise	10 s

A single failed regional check does not by itself confirm an outage. It is one region’s signal on the public internet.

2. Region verdict - the first quorum

Each region can hold more than one node. When it does, a region verdict needs 75% of that region’s fresh votes, rounded up, and at least one. The current topology has one active node per region, so this layer is presently 1/1 rather than independent corroboration inside a region.

Nodes voting	Needed to agree
1	1
2	2
4	3
6	5
10	8

This is what keeps one flaky node in Frankfurt from speaking for Europe. It is computed per region and is not configurable - it is a property of how a region forms an opinion, not a policy knob.

3. Alarm quorum - the second quorum

Regions with a verdict are then counted against your alert profile. This is the quorum you configure, and it is a different thing from the one above:

• a mode and a value - how many regions must agree before an alarm is confirmed,
• a consecutive count - how many passes in a row must hold, which is what stops a single bad minute from paging anyone.

Below the threshold the interface shows the service as degraded, not down. Regions are reporting, but nothing is confirmed. That distinction is the entire point of the quorum: a suspicion is not an outage.

Recovery has its own mode, value and consecutive count, deliberately separate. Coming back is not the mirror image of going down - you usually want more evidence before you declare a service healthy again than before you declare it broken.

4. Incident

Once the alarm quorum holds, an incident opens. It carries the first region that reported, the time it did, and every state change after that. Everything downstream - status page, availability figures, post-mortem - hangs off this one record.

5. Escalation and on-call

On-call is a paid capability: available from Sentinel upwards. Without it, incidents still open and still notify, but there is no rotation and no escalation ladder.

The escalation runs on a tick, in stages with configurable delays (stage 1, then stage 2). Each stage widens the circle of people contacted. Two behaviours are worth knowing because they are easy to be surprised by:

• A coverage gap is itself a condition. If nobody is on duty when an incident opens, that is recorded, not silently skipped.
• Downgrading stops escalation. The tick re-checks the on-call entitlement per organization on every run. If a plan no longer includes on-call, running escalations stop advancing. Existing incidents stay open and visible.

Personal SMS starts with Pulse; on-call SMS and voice start with Sentinel. Successful personal and on-call deliveries share these quotas:

Plan	Shared SMS + voice / 24 h	Shared SMS + voice / month
Free	0	0
Pulse	20	100
Sentinel	50	500
Command	200	2 000
Enterprise	unlimited	unlimited

6. Acknowledgement - taking ownership

Acknowledging is not a “mark as read”. It does two things at once:

1. It stops the escalation. No further stage fires for this incident.
2. It puts your name on it. The incident records who acknowledged and when, and the team sees it.

That is the whole design: the escalation ladder exists to find someone, and it stops the moment someone takes responsibility. Acknowledging an incident you are not going to work on is worse than not acknowledging it, because it silences the mechanism that would have found somebody else.

Acknowledging does not resolve the incident. The service is still down; you have only said that a human is on it.

7. Re-alerting

An acknowledged incident that never closes comes back. Ownership without resolution is exactly the case where a quiet system fails you - somebody took it, then got pulled into something else, and the outage carries on unattended.

8. Recovery and closure

When the recovery quorum holds, the incident closes and the status page and availability figures follow. Incidents that were resolved by other means (manually resolved, discarded, monitor disabled) have their escalations cleaned up on the same tick, so nothing keeps paging for something that no longer exists.

What this chain will not do

• It will not page you for one node’s opinion. Two quorums sit in the way.
• It will not treat missing data as an outage. A monitor with no fresh measurements is not the same as a monitor that measured a failure.
• It will not keep escalating after acknowledgement - but it will come back if the incident stays open.

Next

• Quorum and regions - how regions are chosen and what each one measures.
• Incidents and post-mortems - what happens to an incident after it closes.
• Plans and limits - the full capability matrix.