How monitoring works
This page is the mechanics. If you want to know what to monitor, start with Monitors and check types; if you want to know when an alarm fires, read From measurement to alarm. This one explains the machinery underneath, because knowing it makes the rest predictable.
One scheduler, however many servers
Section titled “One scheduler, however many servers”The control plane can run on more than one process at a time. Only one of them schedules.
Leadership is held through a database lock that lives and dies with the connection holding it. If the leader disappears - crash, restart, deploy - the lock is released by the database itself and another instance picks up scheduling. There is no timeout to tune and no split brain to reconcile: either you hold the lock or you do not.
This is why a deploy does not double-schedule your checks, and why nothing stops scheduling when one instance goes away.
The probe nodes pull their work
Section titled “The probe nodes pull their work”A check does not get pushed to a region. Each probe node asks:
/node/heartbeat- the node reports that it is alive and available./node/due- the node asks what is due for it right now./node/result- the node reports what it measured.
The consequence is worth stating plainly: the probe nodes need no inbound connectivity. Nothing has to reach into a region for it to participate. A node that loses its uplink simply stops asking, and stops being counted - it does not become a source of false results.
It also means a check has a real, honest owner at every moment. The node that asked is the node that answers, and its result carries its region.
A monitor disabled mid-flight
Section titled “A monitor disabled mid-flight”There is a gap between “this check is due” and “here is the result”. If you pause or archive a monitor inside that gap, a result for it can still arrive. The ingest recognises that case and drops the result rather than recording a measurement for something you switched off. Turning a monitor off is immediate from your side.
From result to region verdict to incident
Section titled “From result to region verdict to incident”A single result is not a verdict, and a verdict is not an incident:
| Step | Who decides | What it takes |
|---|---|---|
| Result | one probe node | one check |
| Region verdict | the nodes of that region | 75 % agreement, rounded up |
| Incident | the regions | your alert profile’s quorum, two regions by default |
Each step exists to throw away a class of false alarm: a bad node, a bad region, a bad minute. Quorum and regions covers the rule; From measurement to alarm covers the whole path including escalation and recovery.
How often
Section titled “How often”The interval you can choose has a floor that depends on your plan - 300 s on Free down to 10 s on Enterprise. The floor exists because check volume is what the fleet actually costs; see Plans and limits.
Two things follow from intervals that surprise people:
- A change you make is not instant everywhere. Server-side settings reach agents on their next config poll, which is why a freshly created agent monitor reads “no data yet” for a few minutes rather than failing.
- A result that is too old does not count. Stale results are not folded into a current verdict; the freshness window is part of the rule, not an optimisation.
What this design refuses to do
Section titled “What this design refuses to do”- No pushing into customer networks. The direction of every connection is outbound from the node.
- No single scheduler as a single point of failure, and equally no second scheduler quietly running in parallel.
- No measurement without a named region. Every result knows where it came from, which is what makes a regional quorum possible at all.
- Monitors and check types - the 13 types and their settings.
- Quorum and regions - the decision rule.
- From measurement to alarm - the full path to a page.