Agents on a fleet

The host agent page covers one machine. On a fleet the questions change: how to roll the agent out repeatably, what a sensible set of monitors per host looks like, and how to keep the picture clean when hardware comes and goes.

The approach works for any workload. A rack of database servers, a set of build machines, and a spread of edge nodes share the same three tasks: get the agent on, decide what is worth an incident, and clean up after a machine that is gone.

Rolling out

Two properties make this scriptable.

A reusable enrollment token. The default token is single use, which is right for one machine and wrong for twenty. When you create the token, switch it to reusable and give it a short lifetime. A token that lives an hour and covers the roll-out window keeps the exposure short, and a token stays valid until it expires.

A re-runnable installer. The install command is safe to run repeatedly. On a host that is already enrolled, it switches to upgrade mode: it refreshes the binary and the service, keeps the existing identity, and needs no token at all. That is what makes it usable from configuration management, where the same line runs on every host on every pass.

The shortest form, once you have the command from the app:

TOKEN=psag_…
for host in $(cat hosts.txt); do
  ssh "$host" "PERSTAT_TOKEN=$TOKEN sh -c \"\$(curl -fsSL https://agent.perstat.io/install.sh)\""
done

Passing the token through the environment keeps it out of ordinary process arguments and ps. A literal command can still enter shell or automation logs, so inject the token from your secret store and apply the history and log policy of your tooling. Ansible, Salt, or Puppet run the same line as it is, because the installer is re-runnable.

The installer always checks SHA-256 and verifies Ed25519 when an Ed25519-capable OpenSSL 3 is available. On other systems, it warns and verifies through TLS plus SHA-256. Installer and key come from the same origin, so the first install trusts that origin. The agent’s later self-updates are fail-closed with a compiled key.

What to watch per host

One monitor watches one signal, so you choose the set per host. A set that works for most fleets:

Monitor	Per host	Why
availability	always	The agent monitor that fires when a host goes silent. It makes a dead machine visible.
disk	almost always	Covers every mount point at once, since the threshold applies to the fullest filesystem.
service	one per critical process	The check that catches “the machine is up but the thing it exists for is not running”.
cpu, mem	as context	On small machines a percentage threshold tends to be noisy, and real memory pressure usually shows up as a dead process, which the service monitor catches.

For CPU and memory as context rather than for paging, set "breach_severity": "degraded". That raises an in-app notice without an incident or a push.

Give every agent an availability monitor. It is the one that carries the fleet. When a host stops reporting, the other monitors pause their evaluation, so they neither open nor resolve anything and keep displaying their last reading. That is why a dead host produces one incident instead of one per metric, and it is also why the availability monitor is the one to trust when a host is quiet.

How many monitors that is

Multiply: monitors per host times hosts. A host with availability, disk, and four watched processes is six monitors, so twenty of them is a hundred and twenty.

Three rules decide whether that fits:

• Agent enrollments per plan: 2 on Free, 10 on Pulse, 50 on Sentinel, 200 on Command, and unlimited on Enterprise. Outside Enterprise, one agent carries up to 25 monitors.
• Archiving frees a slot, pausing keeps it. To make room at your limit, archive monitors.
• Packs grow both quotas without a plan change. The per-plan limits and the packs are on plans and limits.

Work out the multiplication before the roll-out, so every host gets its full set from the start.

Naming

Monitor names are unique within a project, so a fleet needs a scheme rather than improvisation. Something that puts the host first and the signal second reads well in lists and sorts sensibly:

web-07 · availability
web-07 · disk
web-07 · nginx

The agent itself is identified by the hostname it reports. If you rebuild machines under the same hostname, keep in mind that the rebuilt machine enrolls as a new agent, and the old one stays until you remove it.

Finding the right process names

service_name matches a process name, not a systemd unit, and on a fleet you want the real names rather than what you assume they are. Two ways:

Run ps -eo comm= | sort -u on a representative host.

Or switch on the process inventory for one agent, which is opt-in and off by default. It transmits process and executable names only, never command lines, paths, users, or PIDs, and the monitor form then suggests those names. Turning it on for a single representative host is usually enough to configure the whole fleet, and you can switch it off again afterwards.

Watch for services that start as scripts. They appear as python3, sh, or node and share a name, which matters more on a fleet, where a naming choice repeats on every host.

Replacing a machine

Two steps belong in the same decommissioning routine.

1. Remove the agent in the app. Its token stops working immediately. Attached monitors are archived, their open incidents resolved, and their plan slots freed, while history remains subject to the applicable retention rules.
2. Remove the software on the host with uninstall.sh, which also deregisters it. If the machine is already gone, step 1 is enough.

When several hosts fail at once

A correlated failure (a rack losing power, a network segment going away) opens one incident per affected monitor. Three features keep that manageable.

Push notifications for an organization are collapsed into one message rather than one per incident, so a correlated failure leaves your phone’s battery alone.

Snooze silences an incident for you without acknowledging it for the team, which is the right move when you already know and are working on it.

Dependent alarms let a monitor name the monitors it depends on. While a named origin is provably down, the dependent monitor holds its alarm chain, so the router going away pages you once instead of once per machine behind it. The incident itself and the status page continue as usual, and only the alarm waits. Set it up per monitor, which pays off on a fleet with a known topology.

Automate the roll-out

The REST API creates, updates, and deletes monitors with a scoped API key, and it lists and removes agents. MCP lets an AI agent create and manage the same monitors. The second half of a roll-out, creating the right monitors for each host, therefore runs from your scripts or from your agent.

Enrollment starts in the app: you create the token there, and each host creates its own identity when it enrolls. A fleet roll-out therefore takes one manual step per window instead of one per machine. Create one reusable token, and configuration management does the rest.

Tell a dead host from a lost connection

On a fleet spread across sites or networks, the first question during an outage is often whether a host died or lost its connection. From the agent’s side both look alike, because Perstat works from the arrival time of each report.

Pair each availability monitor with an external check against the same machine, a ping or tcp monitor on an address that reaches it directly. Agent silent and external check passing: the host works, and its connection to Perstat is interrupted. Agent silent and external check failing: the host is gone.

Next

• Rolling out host agents: the app, the REST API, and the timings.
• API: scoped keys for monitors and agents.