How monitoring works
Knowing the machinery makes every result predictable. Monitors and check types covers what to monitor, and From measurement to alarm covers when an alarm fires. Here is what runs underneath.
One scheduler, however many servers
Section titled “One scheduler, however many servers”The control plane can run several processes at once, and exactly one of them schedules.
Leadership is a database lock that lives and dies with the connection holding it. If the leader disappears (a crash, a restart, a deploy), the database releases the lock and another instance takes over scheduling. The rule is binary: either you hold the lock or you do not. There is no timeout to tune and no split brain to reconcile.
So a deploy never schedules a check twice, and scheduling continues when an instance goes away.
The probe nodes pull their work
Section titled “The probe nodes pull their work”Each probe node fetches its own work in three calls:
/node/heartbeat: the node reports that it is alive and available./node/due: the node asks what is due for it right now./node/result: the node reports what it measured.
The result: the probe nodes need no inbound connectivity. A region takes part without anything reaching into it. A node that loses its uplink stops asking and stops being counted, so it never becomes a source of false results.
Every check also has a clear owner at every moment. The node that asked is the node that answers, and its result carries its region.
Pausing takes effect at once
Section titled “Pausing takes effect at once”Turning a monitor off is immediate from your side. A check can already be in flight when you pause or archive the monitor, and its result may still arrive. The ingest recognizes that case and discards the result, so nothing is recorded for a monitor you switched off.
From result to region verdict to incident
Section titled “From result to region verdict to incident”Each step confirms the one before it:
| Step | Who decides | What it takes |
|---|---|---|
| Result | one probe node | one check |
| Region verdict | the nodes of that region | 75 % agreement, rounded up |
| Incident | the regions | your alert profile’s quorum, two regions by default |
Each step filters out one class of false alarm: a bad node, a bad region, a bad minute. Quorum and regions covers the rule, and From measurement to alarm covers the whole path, including escalation and recovery.
How often
Section titled “How often”Your plan sets the shortest interval, from 300 s on Free down to 10 s on Enterprise, because check volume is what the probe fleet costs to run. Plans and limits lists every floor.
Two consequences are worth knowing:
- Agents pick up changes on their next config poll. A freshly created agent monitor therefore reads “no data yet” for a few minutes instead of failing.
- Only fresh results count. The freshness window is part of the rule, so a current verdict is built from current results.
What this design guarantees
Section titled “What this design guarantees”- Every connection is outbound. Each one starts at the node, and nothing ever pushes into a customer network.
- Exactly one active scheduler. Another instance takes over the moment the leader goes away, and a second scheduler never runs in parallel.
- Every measurement names its region. Each result knows where it came from, and that is what makes a regional quorum possible.
- Monitors and check types: the 13 types and their settings.
- Quorum and regions: the decision rule.
- From measurement to alarm: the full path to a page.