Skip to content

Host agent

Probes check your services from the outside. The agent checks the machine itself and shows what only the inside reveals: a full disk, a dead process, or a host that is up but no longer doing its job. It measures and reports, and it executes nothing, so you can run it on production hosts without reviewing what it might be told to do.

Every 15 seconds, on the host:

Signal Detail
CPU Total utilization, 0 to 100
Load average 1, 5 and 15 minutes (Unix only)
Memory and swap Used, total, percentage
Disk Every mounted filesystem, used and total and percentage
Uptime Seconds since boot
Services For each watched process: running, process count, CPU, resident memory

The table is the complete list: the agent collects these signals about the host and nothing else.

Create an enrollment token in the app under Host agents, then New agent. It hands you a ready-made command:

PERSTAT_TOKEN=psag_… sh -c "$(curl -fsSL https://agent.perstat.io/install.sh)"

The environment assignment keeps the token out of ordinary process arguments and ps output. A literal command pasted into an interactive shell can still land in shell history. Inject the token from your secret manager or through hidden input, and unset the variable afterwards.

The agent runs on Linux (x86_64 and aarch64, static musl builds without runtime dependencies) and on macOS as a universal binary. A Windows agent is planned.

The installer detects OS and architecture, downloads the matching binary, and checks SHA-256. With an Ed25519-capable OpenSSL 3, it also verifies the release signature against the key in the script. On systems without that capability, such as common macOS LibreSSL builds and older distributions, it warns and verifies through TLS transport plus SHA-256. Installer and key come from the same origin, so the first install trusts that origin. Every later self-update verifies against a key compiled into the agent and is fail-closed, as described below.

On Linux the installer sets up a systemd unit, on macOS a LaunchDaemon. Set PERSTAT_NO_SERVICE=1 to get the binary without a service.

Re-running is safe. If the host is already enrolled, the installer switches to upgrade mode: it refreshes the binary and the service, keeps the existing identity, and needs no token. Running the same line from your configuration management on every host, repeatedly, is the intended use. For more than a handful of machines, see agents on a fleet.

Property Default Range
Lifetime 24 hours 1 to 720 hours
Single use yes can be made reusable

The token is shown once and stored as a hash. Single use suits one machine, and a reusable token suits a fleet roll-out. A token stays valid until it expires, so choose a short lifetime.

The token itself is opaque. Which organization and project it belongs to is resolved on the server, never parsed by the agent.

An agent reports, and a monitor decides what is worth an incident. Create monitors of type agent and bind them to the agent. One monitor watches one signal:

metric Fields Fires when
availability grace_seconds (default 180, range 60 to 3600) The host stopped reporting for longer than the grace
cpu threshold (default 90) CPU above the threshold
mem threshold (default 90) Memory above the threshold
disk threshold (default 90) The fullest filesystem above the threshold
service service_name, grace_seconds (default 120, range 60 to 3600) The process has been absent for longer than the grace

Two settings apply to all of them. breach_severity is down (opens an incident, pages on-call, shows on status pages) or degraded (an in-app notice). notify_push toggles the push notification.

One disk monitor covers every mount point, because the threshold applies to the fullest filesystem. It watches filesystems filling up: a log file that adds 700 MB to a 25 GB disk moves the figure by three points, so a 90 percent threshold stays quiet. To alert on a single growing file or directory, use a heartbeat monitor.

Perstat evaluates agent monitors about every 30 seconds. They report directly, without check regions or quorum, so a single breaching evaluation opens the incident once the grace has passed.

service_name is a process name, not a systemd unit. The agent scans the process table and matches three ways: the exact process name, the exact basename of the executable (which is what makes postgres and postmaster both work), and the Linux 15-character name truncation. Matching is case sensitive on Linux.

Good to know before you configure it: a service that starts as a script shows up in the process table as python3, sh, or node, so two such services share a name. Compiled daemons and anything that sets its own process name are identified precisely.

To see the real names, either run ps -eo comm= | sort -u on the host, or switch on the process inventory for that agent. It is opt-in, off by default, and transmits process and executable names only, deduplicated and capped, never command lines, paths, users, or PIDs. With it on, the monitor form suggests names instead of asking you to type them.

The agent scans the process table only when at least one service monitor exists for it.

Outbound only. The agent never opens a listening socket. There is no inbound rule to write and no port to expose, and the systemd unit restricts address families and system calls to match.

Two destinations over HTTPS on port 443:

Host Purpose
api.perstat.io Enrollment, measurements, configuration polling
agent.perstat.io Installer, binaries, self-update downloads

Both need to resolve. On a machine that runs an authoritative nameserver rather than a resolver, make sure a usable resolver is configured.

Measurements every 15 seconds, sent in a batch every 60 seconds, configuration polled about every 5 minutes. The first measurement is sent immediately after enrollment, so a new host appears within seconds.

When a push fails, measurements queue in memory (about six hours’ worth) and are sent when the connection returns, so a short outage leaves your history complete. The queue lives in memory and starts fresh when the agent restarts.

One consequence of the 5-minute configuration poll: after you add a service monitor, it can take a few minutes before the agent is watching that process. Until real data arrives, the monitor stays pending rather than firing, so a newly created monitor never raises a false alarm.

The agent keeps itself current. Releases are ed25519-signed, the signing key stays offline, and the agent verifies the signature against a key compiled into its own binary before swapping anything. It is fail-closed: an unverifiable release is not installed, and a lower version never qualifies. The swap happens inside the agent’s own directory and needs no root helper.

Set DATARGO_AGENT_NO_SELFUPDATE=1 in the service environment if you would rather drive versions yourself.

Remove the identity with the trash icon on the agent. This revokes its token, archives the monitors attached to that agent, and resolves their open incidents. Archived monitors free their plan slots while their historical records remain subject to the applicable retention rules.

Remove the software on the host:

curl -fsSL https://agent.perstat.io/uninstall.sh | sudo sh

It stops and removes the service, deletes the binary, state, and service user, and deregisters the host. If the machine is already gone, removing the identity in the cockpit is sufficient.

Worth understanding precisely, because it decides what you see.

The availability monitor fires: silence past its grace period opens an incident. This is the monitor that tells you a host is gone, and it is the reason to give every agent one.

The other monitors pause their evaluation once the last report is older than 180 seconds. They write no checks, so they neither open nor resolve anything, which is why a dead host produces one incident rather than one per metric. Those tiles keep displaying their last reading, usually green, so read the availability monitor when a host is silent.

To tell a dead host from a healthy one that cannot reach Perstat, pair the availability monitor with an external check against the same machine. The agent’s signal is the arrival of its report, so both cases look alike from the agent’s side.

The agent never collects command lines, environment variables, or usernames. Process inventory is opt-in and limited to names. The agent measures the host and reports, and its whole scope fits in the table under What it measures.

It also executes nothing. There is no remote command channel, no script hook, and no way for anyone at Perstat to run something on your machine.

  • Service monitors check processes. The check answers “is a process of this name alive”. A failed unit with a lingering process, a wedged process, or restart flapping inside the grace window therefore reads as healthy. For critical services, add an external check or a heartbeat that tests what the service delivers.
  • Custom measurements run through heartbeats. To alert on a number only the host knows, use a heartbeat monitor: each one has a healthy endpoint (/ping/<token>) and a failure endpoint (/ping/<token>/fail), and a timer on the host decides which to call. The failure endpoint breaches at the next evaluation rather than waiting out the period. That covers what a percentage threshold cannot, such as a count that dropped or a directory that grew.
  • One monitor per signal. Watching four processes on a host means four monitors.