Agents on a fleet

The host agent page covers one machine. This one covers many, where the questions change: not how to install it, but how to roll it out repeatably, what a sensible set of monitors per host looks like, and how to keep the picture clean when hardware comes and goes.

Nothing here is specific to a kind of workload. A rack of database servers, a set of build machines and a spread of edge nodes all pose the same three problems: get the agent on, decide what is worth an incident, and clean up after a machine that is gone.

Rolling out

Two properties make this scriptable.

A reusable enrolment token. The default token is single use, which is right for one machine and wrong for twenty. When you create the token, switch it to reusable and give it a short lifetime. A token that lives an hour and covers the whole roll-out window leaks less badly than one that lives a month, and tokens cannot be withdrawn before they expire.

A re-runnable installer. The install command is safe to run repeatedly. On a host that is already enrolled it switches to upgrade mode: it refreshes the binary and the service, keeps the existing identity, and does not need a token at all. That is what makes it usable from configuration management, where the same line runs on every host on every pass.

The simplest form, once you have the command from the app:

TOKEN=psag_…
for host in $(cat hosts.txt); do
  ssh "$host" "PERSTAT_TOKEN=$TOKEN sh -c \"\$(curl -fsSL https://agent.perstat.io/install.sh)\""
done

Passing the token through the environment keeps it out of ordinary process arguments and ps; a literal command can still enter shell or automation logs. Inject it from your secret store and apply the history/log policy of your tooling. If you drive this from Ansible, Salt or Puppet, no special module is needed because the installer is re-runnable.

The installer always checks SHA-256 and verifies Ed25519 when an Ed25519-capable OpenSSL 3 is available. Otherwise it warns and falls back to TLS plus SHA-256. Since installer and key come from the same origin, first install still trusts that origin; the agent’s later self-update is fail-closed with a compiled key.

What to watch per host

One monitor watches one signal, so this is a deliberate choice rather than a default. A set that works for most fleets:

Monitor	Per host	Why
availability	always	The only agent monitor that fires when a host goes silent. Without it, a dead machine is invisible.
disk	almost always	Covers every mount point at once, since the threshold applies to the fullest filesystem.
service	one per critical process	The check that catches “the machine is up but the thing it exists for is not running”.
cpu, mem	rarely as an outage	On small machines a percentage threshold is either noisy or meaningless, and real memory pressure usually shows up as a dead process, which the service monitor already catches.

If you want CPU and memory for context rather than for paging, set "breach_severity": "degraded". That raises an in-app notice and opens no incident and sends no push.

Give every agent an availability monitor. It is the one that carries the fleet. When a host stops reporting, the other monitors are not evaluated at all, so they neither open nor resolve anything and keep displaying their last reading. That is why a dead host produces one incident instead of one per metric, and it is also why the availability monitor is the one to trust when a host is quiet.

How many monitors that is

Multiply: monitors per host times hosts. A host with availability, disk and four watched processes is six monitors, so twenty of them is a hundred and twenty.

Three rules decide whether that fits:

• Agent enrolments are capped by plan: 2 in Free, 10 in Pulse, 50 in Sentinel, 200 in Command, and unlimited in Enterprise. Outside Enterprise, one agent can carry at most 25 monitors.
• Archived monitors free their slot, paused ones do not. If you are at your limit and pausing monitors to make room, that will not work. Archive instead.
• The per-plan monitor limits are on plans and limits.

Work the multiplication out before the roll-out rather than after it, because hitting the limit halfway through leaves you with a partly monitored fleet, which is worse than an unmonitored one: it looks like coverage.

Naming

Monitor names are unique within a project, so a fleet needs a scheme rather than improvisation. Something that puts the host first and the signal second reads well in lists and sorts sensibly:

web-07 · availability
web-07 · disk
web-07 · nginx

The agent itself is identified by the hostname it reports. If you rebuild machines under the same hostname, keep in mind that the rebuilt machine enrols as a new agent, and the old one stays until you remove it.

Finding the right process names

service_name matches a process name, not a systemd unit, and on a fleet you want the real names rather than what you assume they are. Two ways:

Run ps -eo comm= | sort -u on a representative host.

Or switch on the process inventory for one agent, which is opt-in and off by default. It transmits process and executable names only, never command lines, paths, users or PIDs, and the monitor form then suggests those names. Turning it on for a single representative host is usually enough to configure the whole fleet, and you can switch it off again afterwards.

Watch for services that start as scripts. They appear as python3, sh or node and cannot be told apart by name, which matters more on a fleet because the mistake is then repeated on every host.

Replacing a machine

Two steps belong in the same decommissioning routine.

1. Remove the agent in the app. Its token stops working immediately; attached monitors are archived, their open incidents resolved, and their plan slots freed while history remains subject to the applicable retention rules.
2. Remove the software on the host with uninstall.sh, which also deregisters it. If the machine is already gone, step 1 is enough.

When several hosts fail at once

A correlated failure, a rack losing power or a network segment going away, opens one incident per affected monitor. Three things soften that.

Push notifications for an organization are collapsed into one message rather than one per incident, so a correlated failure does not empty your phone’s battery.

Snooze silences an incident for you without acknowledging it for the team, which is the right move when you already know and are working on it.

Dependent alarms let a monitor name the monitors it depends on. While a named origin is provably down, the dependent monitor holds its alarm chain, so the router going away does not page you once per machine behind it. It never suppresses the incident itself or changes a status page, only the alarm. It is opt-in per monitor, which means it is worth setting up deliberately for a fleet with a known topology.

What you can automate today

The REST API is read-only: its API keys read exactly seven endpoints. Programmatic writes to monitors run through MCP. So the second half of a roll-out, creating and managing the right monitors for each host, scripts through MCP, while REST supplies the read side.

Enrolling the agents themselves does not. Creating an enrolment token is done in the app, and the agent’s identity is created by the host when it enrols rather than by an API call. In practice that makes a fleet roll-out one manual step per window rather than one per machine: create one reusable token, then let configuration management do the rest.

The blind spot worth knowing

A machine that is dead and a machine that is healthy but cannot reach us produce the same signal, because the timestamp we work from is when the report arrived. On a fleet spread across sites or networks, that difference is often the first thing you want to know.

If it matters to you, pair each availability monitor with an external check against the same machine, a ping or tcp monitor on an address that reaches it directly. Agent silent plus external check passing means we lost sight of a working host. Agent silent plus external check failing means the host is gone.