# Host metrics — tiered sampling (audit + proposed architecture)

Status: **implemented** (`hostmetrics/{tiers,fast,medium,slow,top}.rs`,
`hostmetrics/extended/`, `hostagent/sampler.rs`). §1 is the audit that motivated the split; §2–3
describe the shipped design (the *as-built* notes flag where it departs
from the original proposal). Companion to `hostagent.md` and
`metrics.md`.

## 1. Audit of `collect()` — the shape before the split

Historical: the whole-snapshot `collect()` this section measured is gone
from `hostmetrics.rs`, replaced by the tier collectors of §3. Back then
one `collect()` call gathered everything, every 10 s, on one blocking
thread. Its wall time (~200–700 ms) was dominated by two *waits*, not by
work, and the cheapest gauges (CPU %, RAM) were refreshed no faster than
the most expensive probe (NTP round trip, `nvidia-smi`).

Costs below are per call, measured on this M4 Max (macOS 15) for the
subprocesses and estimated from the syscalls involved elsewhere.
"Volatility" is how fast the value meaningfully changes.

| # | Collector (code) | What it feeds | Cost / hazard | Volatility |
|---|---|---|---|---|
| 1 | `sleep(MINIMUM_CPU_UPDATE_INTERVAL)` + `refresh_cpu_usage` | `global_cpu_percent`, `per_core_percent` | **200 ms fixed wait** — CPU% is a delta and `System` is rebuilt each call, so the warmup window is paid every time | high |
| 2 | `start/finish_cpu_sampling` (`/proc/stat` ×2, `systemstat` on macOS) | `cpu_states`, `kernel` | 2 × <1 ms reads, but rides on wait #1 | high |
| 3 | `sysinfo::System::new_all()` | `processes_count`, `process_states`, `cpu_brand`, cores, `mem_*`, `swap_*`, `cgroup_limits` | **Enumerates every process** (`/proc/<pid>/stat` walk, `proc_pidinfo` on macOS): 20–300 ms depending on process count; the only reason it is here is the process-state histogram | mem: high; processes: medium |
| 4 | `query_ntp_offset_ms` (thread) | `ntp_offset_ms` | UDP to `pool.ntp.org`, **up to 500 ms**, network egress every 10 s from every host of a fleet | very low (drift is minutes-scale) |
| 5 | `gpu::read_gpus` (thread) | `gpus` | `nvidia-smi` subprocess 20–100 ms (2 s deadline; also keeps an idle NVIDIA GPU out of its low-power state), `ioreg` ~30 ms, DRM sysfs <1 ms | gauges: medium; identity: static |
| 6 | `Disks::new_with_refreshed_list` + `statvfs` per mount | `disks` (space, inodes, read/write totals) | `getfsstat`/`/proc/mounts` + one `statvfs` per mount, ~1–5 ms; **a stalled NFS/FUSE mount blocks the whole collect** | space: low; I/O counters: high |
| 7 | `Networks::new_with_refreshed_list` | `networks` (counters, mac, ips) | `getifaddrs` + `/proc/net/dev`, ~1 ms | counters: high; mac/ips: static |
| 8 | `Components::new_with_refreshed_list` | `sensors` | `/sys/class/hwmon` walk (dozens of files) or SMC/IOKit on macOS, 5–50 ms | low |
| 9 | Linux `/proc` one-file readers: `stat`, `vmstat`, `meminfo`, `net/snmp`, `net/netstat`, `sys/fs/file-nr`, `pressure/*`, `net/sockstat`, `sys/net/netfilter/*`, `sys/fs/{dentry-state,inode-nr}`, `random/entropy_avail`, cgroup files | `paging`, `mem_detail`, `tcp_counters`, `file_handles`, `pressure`, `sockets`, `conntrack`, `vfs_cache`, `entropy_avail`, `cgroup` | <1 ms each, ~5 ms total | high |
| 10 | `read_tcp_states` (`/proc/net/tcp{,6}` full walk) / `listeners::get_all()` on macOS | `tcp_states` | **O(sockets)**: a busy server with 100 k connections is a multi-MB read under a kernel lock, 20–200 ms; macOS `listeners` enumerates every pid's fds | medium |
| 11 | `read_raid` (`/proc/mdstat`) | `raid` | <1 ms | very low |
| 12 | `block_devices_of` (`/proc/diskstats`) | `block_devices` | <1 ms | high |
| 13 | macOS `vm_stat`, `netstat -s -p tcp` subprocesses | `mem_compressor`, `tcp_counters` | fork+exec, ~15–20 ms each | medium |
| 14 | `System::host_name/name/long_os_version/kernel_version/cpu_arch/boot_time` | identity fields | `uname`, `/etc/os-release`, `sw_vers` plist parse: ~1–10 ms | **never changes** |

Findings:

- **F1 — the cheap, volatile core is hostage to the slow tail.** CPU %,
  load, RAM, PSI, NIC/disk counters cost ~5 ms combined but are only
  refreshed every 10 s because the same tick also waits for NTP,
  `nvidia-smi` and a process walk. Lowering `THREEPM_HOSTMETRICS_INTERVAL_MS`
  today drags the expensive probes down with it (the doc even warns that
  below ~1 s "the collection itself dominates").
- **F2 — the 200 ms CPU warmup is an artefact of statelessness.** A
  periodic sampler already *has* a window: the tick. Keeping the
  `sysinfo::System` / previous `CpuTicks` across ticks removes the sleep
  entirely (same trick `rates.rs` already applies to byte counters).
- **F3 — throughput resolution is 10 s.** `rx_bytes_per_s` on the `ls`
  line is an average over the last 10 s; pm2 users expect ~1 s.
- **F4 — a hung collector stalls every metric.** One deadlocked
  `nvidia-smi` (2 s deadline) or a stalled NFS `statvfs` (no deadline)
  delays CPU/RAM freshness with it. The blast wall isolates the daemon,
  not the metrics from each other.
- **F5 — fleet noise.** One NTP packet per host every 10 s is 8 640
  UDP probes/day/host for a value that drifts over hours.
- **F6 — static identity is re-derived 8 640 times a day** (#14, GPU
  model/driver, NIC mac/ips, disk inventory).

## 2. Proposed tiers

Assignment is by **cost first, volatility second**: anything under a few
ms with no hazard goes fast regardless of volatility (it is free); anything
that walks, forks, or touches the network goes slower.

| Tier | Default period | Env knob | Fields |
|---|---|---|---|
| **T0 identity** | rides the slow ticker (first tick at start) | — | `hostname`, `os_*`, `kernel_version`, `arch`, `boot_time_epoch_s`, `cpu_brand`, `physical_cores`, `logical_cpus`, `user` (the account the daemon runs as: `getuid` → passwd, `None` where there is no passwd database) |
| **T1 fast** | **1 s** | `THREEPM_HOSTMETRICS_INTERVAL_MS` (keeps its name: it *is* "the" sampling rate users tune) | `global_cpu_percent`, `per_core_percent`, `cpu_states`, `cpu_frequency_mhz`, `per_core_freq_mhz`, `kernel`, `load_avg`, `uptime_s`, `mem_*`, `swap_*`, `paging`, `mem_detail`, `pressure`, `file_handles`, `sockets`, `tcp_counters` (Linux `/proc` path only), `conntrack`, `vfs_cache`, `entropy_avail`, `cgroup`, NIC counters + `*_per_s`, `block_devices`, disk `read/written_bytes_total` + `*_per_s` |
| **T2 medium** | **10 s** | `THREEPM_HOSTMETRICS_MEDIUM_MS` | `processes_count`, `process_states` (the process walk — on Linux, since 2026-10, T6's walk when it is younger than a medium period: one walk of the table every 5 s instead of three every 10 s), `tcp_states` (socket table walk), `sensors`, `gpus` (identity + gauges — one tool invocation), disk rows: identity + `available_bytes`/`inodes_*` (statvfs), `raid`, `batteries` + `on_ac` (the `/sys/class/power_supply` walk, Linux; a handful of small sysfs reads per supply), macOS `mem_compressor` + `tcp_counters` (subprocesses), `sessions` (the accounts logged in — one utmp walk through libc, sorted, deduplicated, capped at 64; empty where there is no utmp) |
| **T3 slow** | **300 s** | `THREEPM_HOSTMETRICS_SLOW_MS` | `ntp_offset_ms`; T0 identity refresh; with Extended Monitoring on, the `extended` families `cpuidle`, `cpufreq`, `thermal` (§3.7) |
| **T4 3pm** | = medium | (rides `THREEPM_HOSTMETRICS_MEDIUM_MS`) | `3pmDaemon`: the daemon's and the agent's own cpu/RSS/uptime/fds/threads (two `sysinfo` process refreshes + a few `/proc` reads, ~1 ms) and the daemon's `Internals` over `rpc.sock` (one connect + ping + request, bounded 2 s; on the daemon side `fill_io_gauges` walks `/dev/fd` and stats three files — cheap at 10 s, noise at 1 s, hence never on the fast period). Its own loop (`hostagent/selfmetrics.rs`), so a daemon busy with a handoff never delays T2's walks, and never part of the first-snapshot gate |
| **T5 extended** | **10 s** | `THREEPM_HOSTMETRICS_EXTENDED_MS`; exists only with `[hostmetrics] extended` on (`THREEPM_HOSTMETRICS_EXTENDED`, default on) | `extended`: the kernel's own counters by family (§3.7), Linux; an empty map elsewhere. Never part of the first-snapshot gate |
| **T6 top** | **5 s** | `THREEPM_HOSTMETRICS_TOP_MS`; exists only with `[hostmetrics] top_processes` on (`THREEPM_HOSTMETRICS_TOP_PROCESSES`, default on) | `top_processes`: the 10 largest processes of the host by CPU, by memory (footprint on macOS, RSS on Linux) and by network rate, one union table sorted by pid (§3.8). A walk or a `ps`, plus one fork: ~35–45 ms a tick; macOS also keeps one `top` running for the footprint, ~1.8 % of a core at 5 s. Never part of the first-snapshot gate |

*As built:* NIC `mac`/`ips` stay in T1, but since 2026-10 on Linux
they are no longer re-read every tick: the counters come from one
`/proc/net/dev` read a tick, the addresses from a `sysinfo` refresh
(seven sysfs files per interface and two `getifaddrs`) every 10 s and
at once when an interface appears — an address change on an existing
interface shows within 10 s. Per-core frequencies are read from
`scaling_cur_freq` files the agent keeps open (one `pread` a core a
tick; the agent holds one descriptor per core). Block devices and the
disks' I/O counters come from one `/proc/diskstats` read a tick (it
was read twice, beside `/proc/mounts` and `/dev/disk/by-id`), the
mounts they are joined to being re-listed every 10 s — the period of
T2's disk rows. GPU identity
stays with the gauges in T2 (one `nvidia-smi`/`ioreg` run instead of
two). Disk rows are T2 (`statvfs` hazard) and receive their I/O
counters and rates from T1 by `mount_point` at assembly time.

Clamps: fast ≥ 200 ms (today's `MIN_INTERVAL_MS`), medium ≥ fast,
slow ≥ medium, extended ≥ fast, top ≥ fast; a knob set below its floor is raised with a warning in
`hostagent.log`, never rejected.

Effect on the `ls` host line (all its segments are T1 except the CPU
sensor temperature and GPU, which are T2): cpu/ram/load/throughput refresh
every second, throughput segments appear after 2 s instead of 20 s.

Budget: T1 at 1 s costs ~2–5 ms CPU per tick (no sleep, no fork) — 20–50 ms
per 10 s, i.e. the same order as today's single 10 s collect, for 10×
the freshness. T2 unchanged (~50–150 ms per 10 s). T3 removes 29 of every
30 NTP probes.

## 3. Architecture

### 3.1 Tier collectors

```
hostmetrics/
  mod.rs          the collectors' module; no whole-snapshot entry point (the agent assembles)
  tiers.rs        IdentityPart / FastPart / MediumPart / SlowPart / ExtendedPart / TopPart; Parts; assemble()
  fast.rs         FastCollector { sys, disks, nets: sysinfo handles; CpuWindow }  ← stateful
  medium.rs       MediumCollector { Linux: procwalk::Walker; elsewhere: sysinfo process table, tasks off }  ← stateful
  procwalk.rs     Linux: one `/proc/<pid>/stat` read per process into a kept buffer (T2's histogram, T6's table)
  slow.rs         fn collect_slow(extended) -> SlowPart, fn collect_identity() -> IdentityPart
  top.rs          TopCollector (two readings → the table), TopTier (reader + collector)  ← stateful (§3.8)
  top/            macos.rs (`ps`, `nettop`), linux.rs (`procwalk` walk, `ss` fallback), sockets.rs (`sock_diag`): readers + parsers
  extended/       Extended Monitoring (§3.7): parse.rs (shapes), table.rs (sources), mod.rs (collect),
                  ebpf.rs + ebpf/shortlived.bpf.{c,o} (short-lived processes, Linux, root)
  linux.rs / macos.rs / gpu.rs   unchanged pure parsers + readers
```

- `HostMetrics` (wire) stays the single flat struct — **no wire change for
  the shape**. Internally it is assembled from its parts:
  `HostMetrics = identity ⊕ fast ⊕ medium ⊕ slow ⊕ 3pm ⊕ extended ⊕ top`
  (the last three optional, never waited for) by a pure
  `assemble(&Parts) -> HostMetrics` in `tiers.rs`, unit-tested with
  partial inputs (a part still `None` → its fields absent/default,
  exactly like today's "absent, never faked").
- Cross-tier joins are by key: NIC by `name`, disk by `mount_point`, GPU
  by index within vendor — the same keys `rates.rs` already uses.
- `FastCollector` is **stateful** (F2): it owns a `sysinfo::System`
  refreshed with `refresh_specifics(cpu + memory)` only (never
  `new_all()`), and the previous `/proc/stat` ticks. CPU % and
  `cpu_states` are deltas over the tick window; the very first tick keeps a
  one-off 200 ms warmup so the first sample is never `None` on cpu (the
  `ls` line hides absent segments — an empty first line would look like a
  regression). Disk I/O counters on macOS come from
  `Disks::refresh_specifics(DiskRefreshKind::nothing().with_io_usage())`
  (no `statvfs`); on Linux from `/proc/diskstats` which T1 already reads.
- `rates.rs` generalises from `(prev: &HostMetrics, cur: &mut HostMetrics)`
  to `(prev: &FastPart, cur: &mut FastPart)` — same tests, window is now
  the fast tick.

### 3.2 Sampler: one ticker per tier, one cache

```
sampler::run(cache, Collectors, Intervals, token)
   ├─ task identity: interval(slow)   → spawn_blocking(collect_identity()) → cache.set_identity(part)
   ├─ task fast:     interval(fast)   → spawn_blocking(fast.collect())     → cache.set_fast(part)   (rates vs the previous fast part)
   ├─ task medium:   interval(medium) → spawn_blocking(medium.collect())   → cache.set_medium(part)
   ├─ task slow:     interval(slow)   → spawn_blocking(collect_slow(ext))  → cache.set_slow(part)
   ├─ task extended: interval(extended), only when Some → spawn_blocking(collect_extended() + ebpf::collect(shared)) → cache.set_extended(map)
   └─ task top:      interval(top), only when Some → spawn_blocking(TopTier::collect()) → Some: cache.set_top(part); None: nothing
every set_*: stamp the part, store it, bump the version; a read (`latest`) assembles(parts)
```

- `Cache` serves `latest` and `wait_first` to `server.rs`. *As built
  since 2026-10:* the snapshot is assembled when it is read, not when a
  part lands — an assembly deep-copies every part (the extended map is
  ~1 200 values) and the fast tier lands one a second, read or not. A
  `watch::Sender<u64>` version, bumped by each landing, is what readers
  wait on. The first snapshot is servable once **identity, fast and
  medium** have each landed (all first ticks fire immediately; the slow
  tier's NTP probe is not waited for). Collectors are `FnMut` boxes built by a
  factory so the fast one can keep state and be rebuilt after a panic.
- Each tier has its own `MissedTickBehavior::Delay` ticker and its own
  `spawn_blocking`; a wedged T2 (`nvidia-smi`, NFS) never delays T1 (F4).
  A panic in one tier keeps that tier's previous part and leaves the
  others live — extend
  `a_panicking_collector_keeps_the_previous_part_and_is_rebuilt`
  (`hostagent/sampler.rs`, and its twin in `hostagent/selfmetrics.rs`)
  per tier.
- Overlap guard: a tier whose collection is still running when its next
  tick fires skips that tick (Delay already gives this).

### 3.3 Freshness on the wire (additive, protocol v1)

`collected_at_epoch_ms` keeps meaning "age of the newest data" = the T1
stamp, so `3pm hostmetrics`'s `sampled` row and existing consumers are
unchanged. One optional field was added:

```json
"sampled_at_epoch_ms": { "fast": 1787490168000, "medium": 1787490160000, "slow": 1787489900000 }
```

(`Option<HostSampledAt>` whose members are each optional — a tier that
has not run yet is simply absent; an `extended` member dates the
Extended Monitoring tier when it is on (its slow families are the `slow`
stamp's); a `top` member dates the top-processes table (§3.8): when
the last table published landed in the cache — up to the network fork's
2 s after its reading — and, since a tick with nothing to publish leaves
it alone, how a reader tells a stale table; the whole object is absent from a
pre-tier daemon; frozen in `tests/fixtures/wire/v1/`.) `rate_window_ms`
stays and now reports the fast window (~1000). The `hostmetrics`
renderer adds `sampled (medium)` / `sampled (slow)` rows only when that
tier lags the fast stamp by more than 2 s.

### 3.4 Config surface

| Knob | Default | Floor |
|---|---|---|
| `THREEPM_HOSTMETRICS_INTERVAL_MS` | `1000` (was `10000`) | `200` |
| `THREEPM_HOSTMETRICS_MEDIUM_MS` | `10000` | `= fast` |
| `THREEPM_HOSTMETRICS_SLOW_MS` | `300000` | `= medium` |
| `THREEPM_HOSTMETRICS_EXTENDED_MS` | `10000` | `= fast` |
| `THREEPM_HOSTMETRICS_EXTENDED` / `[hostmetrics] extended` | on | — (off: no T5, no slow-tier walks, `extended` absent) |
| `THREEPM_HOSTMETRICS_TOP_MS` | `5000` | `= fast` |
| `THREEPM_HOSTMETRICS_TOP_PROCESSES` / `[hostmetrics] top_processes` | on | — (off: no T6, no fork, no walk, `top_processes` and `sampled_at_epoch_ms.top` absent) |
| `THREEPM_NTP_SERVER` | unchanged | — |

Changing the default of the existing knob is a behaviour change worth a
CHANGELOG line; the e2e harness that lowers it keeps working (it lowers
the fast tier, which is what the tests observe).

### 3.5 Tests (per CLAUDE.md checklist)

- `tiers.rs`: `assemble` with every subset of parts present (proptest over
  the 2⁴ combinations + random keys: a NIC present in fast but not identity
  still yields its counters with empty `mac`/`ips`).
- `fast.rs`: second tick yields cpu without sleeping (paused tokio time +
  a fake `CpuTicks` source); first tick warmup exactly once.
- `sampler.rs`: `start_paused` component test asserting tick counts per
  tier over a simulated 60 s (60 / 6 / 1 with a 10 s slow for the test),
  plus tier isolation (a T2 collector that blocks 5 s does not delay T1
  publishes).
- e2e: `ls` shows throughput segments within 3 s of daemon spawn;
  `hostmetrics --json` carries `sampled_at_epoch_ms` with
  `fast ≥ medium ≥ slow`; `THREEPM_HOSTMETRICS_MEDIUM_MS` below fast is
  clamped (log line asserted).
- Snapshots: `hostmetrics` render with and without the new ages.

### 3.6 As built

Shipped in one change set: parts + `assemble()`, the stateful
`FastCollector`, the multi-ticker sampler, the three knobs,
`sampled_at_epoch_ms`, and the doc updates (`hostagent.md`,
`metrics.md`, `protocol.md`, OV `3pm-docs-archived/architecture` §8.2, FEATURES, CHANGELOG).
T4 (`3pmDaemon`) followed as its own change set: `hostmetrics/threepm.rs`
(OS half), `hostagent/selfmetrics.rs` (the loop and the `rpc.sock` pull),
`Parts.threepm` landing the wire section as is. T5 (§3.7) and T6 (§3.8)
each came behind an option the daemon resolves once per image and hands
to every agent incarnation in the option's own env var.

### 3.7 Extended Monitoring (T5 + the slow tier's share)

`HostMetrics.extended` is a map of families (`HostExtended`), not typed
fields: the kernel names the values and varies them between versions.
`hostmetrics/extended/table.rs` has one row per kernel file — family,
path, shape (`parse.rs`), kind rule, skip list; `mod.rs` walks a table.
Adding a family is adding a row.

- **Two tables, two tiers.** `EXTENDED` on T5; `SLOW` (the three sysfs
  walks with a file per core per state or per zone: `cpuidle`, `cpufreq`,
  `thermal`) on T3. `assemble` merges the two into one map. With the
  mode on the map is present on every platform (macOS: no family); off,
  no T5 task exists, the slow tier opens nothing more, the map is absent.
- **Kinds.** A family is all counters (cumulative: since boot for the
  kernel's files, since the agent's eBPF load for `shortlived*`) or all
  gauges, for good. A file mixing both is split at the source:
  `vmstat` / `vmstat_gauge` (`nr_*` levels but seven `nr_*` counters,
  `workingset_nodes`), `snmp` / `snmp_gauge` (settings, `CurrEstab`),
  `pressure` / `pressure_gauge` (`avg*` vs `total`), `ksm` /
  `ksm_gauge` (`full_scans`, `pages_scanned`, `pages_skipped` count),
  `hwmon` / `hwmon_gauge` (`energy*_input` counts), `btrfs` /
  `btrfs_gauge` (`reclaim_*` count), `zfs_arc` / `zfs_arc_gauge`
  (every `*hit*`, `*misses*`, `evict_l2_*` and the named events), and
  `wireless`, `sctp` alike. `diskstats` is all counters (its one level,
  in-flight I/O, is `block_devices`').
- **Never twice.** Each source lists what a typed field already carries
  (`/proc/stat`'s all-cores line and scalars, the six sysinfo NIC
  counters, systemstat's nine `diskstats` columns, `meminfo` totals and
  `mem_detail`, `vmstat` paging, the twelve `tcp_counters`, `sockets`,
  `vfs_cache`, cpu 0's frequency, hwmon `temp*_input/_crit/_highest`, an
  amdgpu chip's `power1_average` (`gpus[].power_w`), a thermal zone's
  `temp` when an hwmon chip bears its type's name — or always, when no
  hwmon chip has a temperature and sysinfo reads the zones — and the
  three DRM gauges). Nor one family twice: vmstat's `nr_*` levels that
  `/proc/meminfo` prints (`nr_free_pages` = `MemFree`, the LRU lists,
  slab, `nr_dirty`, `nr_shmem`, THP, `nr_kernel_stack`, …) and every
  `nr_zone_*` go; `nr_file_pages`, `nr_isolated_*`, `nr_zspages`,
  `nr_free_cma` and the dirty thresholds stay.
- **Throwaway rows.** Loop devices and ramdisks (`diskstats`), veth
  pairs (`netdev`, `wireless`) are not sent: a series set per snap or
  container start that nobody reads.
  Fully typed files are not in the table (`file-nr`, `inode-nr`,
  `entropy_avail`, `uptime`, netfilter, `power_supply`).
- **Values.** Counters are whole numbers (the node state quantizes a
  fraction to 3 digits, which would break a rate); only `kB` → bytes is
  converted. No NaN or infinity, no name starting with `~`.
- **Availability, read by read.** A missing file: the family is absent,
  nothing said (no ZFS here). A refusal (`EACCES`/`EPERM`):
  `unavailable[family] = "permission"`, the family absent — or partial,
  for a sysfs walk where one attribute refused (a write-only attribute
  such as `emul_temp` does not count). Any other failure: `"error"` and a
  debug log. `EOPNOTSUPP` counts as missing (PSI compiled in, `psi=0`
  at boot). Never an euid check. A walker that panics (the slow share
  runs beside the NTP probe) costs only its families that tick, listed
  as `"error"`.
- **Rows.** `""` for a one-row family; else the kernel's key: core index
  (`cpu`, `cpufreq`), `<core>/<C-state>` (`cpuidle`), device, interface,
  protocol (`snmp`: `Ip`, `Tcp`…), `<resource>_<scope>` (`pressure`),
  sysfs entry (`thermal_zoneN`, `cooling_deviceN`, `cardN`), hwmon
  chip name (`nvme`, `coretemp`; `nvme_1` for a second `nvme`, by hwmon
  index — `hwmonN` follows probe order, the name does not),
  `<dev>/<port>` (InfiniBand), `<uuid>/<group>` (btrfs).
- **Cost (edgecr4ft.net, 5.4, 32 cores, 2026-09-29, debug build):** T5
  median 9.4 ms, 20 families / 941 series; the slow share 7.5 ms, 3
  families / 287 series (`cpuidle` 192). Families there: `cpu` 320,
  `netstat` 133, `vmstat` 87 + 9, `snmp6` 87, `snmp` 77 + 7,
  `diskstats` 72, `netdev` 60, `meminfo` 38, the rest under 15 each.
  On the link (same host, 2026-09-29, a node with the mode on beside one
  with it off, same 4 apps, 14 min): 777 against 529 B/s up, +248 B/s
  for host families and per-process `ext` together — a state patch
  carries only the leaves that moved, deflated; ~30 B/s more with the
  eBPF families (root agent).
- **Short-lived processes (eBPF, `ebpf.rs`).** What no tier sees: on
  edgecr4ft 84 % of the processes that exit live under a second.
  `ebpf/shortlived.bpf.c` hooks `sched:sched_process_exec` / `exit` and
  keeps `totals`, `by_comm` (name → count, total and max lifetime) and
  `starts` (pid → exec time, cleaned by each exit, `max_entries` fixed:
  full, it loses lifetimes, never node memory). Only a process whose exec
  was seen has a lifetime and a name: an exit without one — forked
  without exec (Postgres backends, prefork workers), started before the
  load, or its exec refused by a full `starts` — is `exits_without_exec`
  and appears in the totals only. A full map is counted, not silent:
  `starts_full` (execs `starts` refused), `by_comm_full` (exits
  `by_comm` refused). No CO-RE — tracepoint
  arguments only — so one committed object (`shortlived.bpf.o`, bpfel,
  8 352 bytes, rebuilt by `just bpf`, reproducible) serves every kernel
  and architecture, with or without BTF; embedded with
  `include_bytes_aligned!`, loaded by `aya` (a Linux-only dependency).
  Loaded on T5's first tick, once per agent: the state is shared by
  every collector T5 builds, so a rebuild after a panic neither reloads
  nor attaches twice. Before aya, `CapEff` (`/proc/self/status`): without
  `CAP_SYS_ADMIN`, or `CAP_BPF` + `CAP_PERFMON`, the answer is `not_root`
  and nothing is loaded — aya's loader parses the kernel's BTF
  (`/sys/kernel/btf/vmlinux`, 4.5 MB) before anything can fail; an
  unreadable `CapEff` tries (a deliberate exception, 2026-09-29, to "try,
  don't guess", plan §4; the uid is still never consulted). The check only rules out: the capabilities
  are necessary, not sufficient. Attaching also reads the tracepoint's id
  from tracefs (`/sys/kernel/tracing/events/…/id`, `0700` root on
  Ubuntu: uid 0 or `CAP_DAC_READ_SEARCH`), Ubuntu's
  `perf_event_paranoid = 3` may refuse `perf_event_open` without
  `CAP_SYS_ADMIN`, and before Linux 5.11 the maps and the program count
  against `RLIMIT_MEMLOCK`: on `EPERM` aya 0.14 raises it to unlimited
  and retries once (`with_raised_rlimit_retry`), which takes
  `CAP_SYS_RESOURCE` or an unlimited hard limit, so a capable but
  limited agent is reported `not_root`. The try says which. The load
  passes `btf(None)` (no CO-RE):
  one BTF parse instead of `Ebpf::load`'s two, the one left being
  `EbpfLoader::new`'s own, which aya 0.14 cannot skip. The agent's
  allocator is pinned against what that parse frees: `docs/hostagent.md`,
  "Memory". Refused:
  `unavailable.shortlived` = `not_root` (the `CapEff` check, `EPERM`,
  `EACCES` outside the verifier), `bpf_disabled` (`ENOSYS`) or
  `load_failed` (the verifier, a missing tracepoint: warn log), and no
  retry before the next agent start; the log line of a refusal quotes
  `unprivileged_bpf_disabled`, which classifies nothing (tracepoint
  programs need privilege whatever it says). Each T5 tick: `totals` →
  `shortlived` (counters, one row: `execs`, `exits`, `exits_under_1s`,
  `lifetime_ns`, `lifetime_under_1s_ns`, `exits_without_exec`,
  `by_comm_full`, `starts_full`, since the load); `by_comm` drained (keys listed, then each read and
  deleted), the window's 10 names with most exits kept, the rest — and a
  name that cannot be a row: empty, a control character, `~…`, `other`
  itself — summed under `other`: `shortlived_by_comm` (counters `exits`,
  `lifetime_ns`, running totals kept agent side) and
  `shortlived_by_comm_max` (gauge `max_ns`, the window's). A row appears
  only in the windows it made; its running total survives 30 windows
  without it, then goes (a later return is a counter reset): at most
  11 × 31 totals held. A name's first exit of the window inserts with
  `BPF_NOEXIST` (two first exits on two CPUs add up instead of
  overwriting); an exit racing the drain of its name is lost (no
  lookup-and-delete on a hash map before 5.14); `totals` loses none.
  Verified 2026-09-29: edgecr4ft (5.4, BTF) and a tablet (6.19, no BTF)
  as root, `true` × 200 → exactly 200 execs / exits / under 1 s and a
  `true` row of 200; as `ubuntu`/`envision`, `not_root`.
- **Tests.** Fixtures are `/proc` and `/sys` as captured read-only on
  edgecr4ft (5.4) and nirmata (6.8) under
  `crates/threepm-daemon/tests/fixtures/hostmetrics-extended/`, used as
  collection roots; `hostmetrics::extended::` runs on the Mac (eBPF
  with fake maps). The real load is `#[ignore]`d
  (`ebpf::tests::real_load_counts_fork_exec_or_says_not_root`): run the
  test binary by hand on Linux, `--ignored`, once with `sudo` and once
  without.

### 3.8 Top processes (T6)

`HostMetrics.top_processes`: which processes are eating the box, managed
by 3pm or not. One table, not three: the 10 largest by CPU, by memory
(the physical footprint on macOS, the RSS on Linux) and by network rate
(in plus out), merged — at most 30 rows,
sorted by pid so that the node state diffs it row by row. A dimension
with fewer than 10 non-zero figures is shorter, not padded.

- **Sources.** Linux: a walk of its own (`procwalk`: one
  `/proc/<pid>/stat` read per process — `comm`, `utime + stime`, `rss`,
  start time — and `status` once per process for its uid; CPU is a
  delta, so it cannot share T2's walk; until 2026-10 a `sysinfo` walk
  that read `statm` too and cost three times the syscalls) and the
  per-socket byte counters summed per pid: one `sock_diag` dump of both
  families' connected TCP sockets with their `tcp_info` (the
  `TCPDIAG_GETSOCK` request, one walk of the kernel's established table
  where `ss` made one per family), holders from the readable processes'
  fd tables (`top/sockets.rs`) — the figures `ss -tinpH` gave, by the
  same rules (`top/linux.rs`, `parse_ss`); `ss -tinpH` itself where the
  host refuses `sock_diag`. Until 2026-10 it was `ss` every tick: a fork,
  an exec and two walks every 5 s. macOS: `/bin/ps -A -o
  pid=,uid=,rss=,time=,ucomm=` (setuid root: unprivileged, no
  in-process API sees another user's process) and `/usr/bin/nettop -P -n
  -L 1 -x -J bytes_in,bytes_out` (`-n`: without it nettop waits 5 s on
  reverse DNS), plus the footprint from a `/usr/bin/top -l 0 -s <S>
  -stats pid,mem` kept running (below). Elsewhere: no reader, no table,
  no error. Each per-tick fork has a 2 s deadline.
- **The footprint (macOS).** What Activity Monitor and Stats call Memory
  (`ri_phys_footprint`), where the RSS misses the biggest consumers — a
  VM at 8.2 GB of footprint shows 803 MB of RSS, and WindowServer, at
  4 GB, is out of the RSS top ten. `proc_pid_rusage` answers only half the pids
  unprivileged; `top` sees them all, but a `top -l 1` costs 0.78 s of
  CPU per run. So one `top` runs for the tier's life, sampling every
  `S` = the tier period rounded up to whole seconds (at least 1: never
  more often than the tier reads), under `LC_ALL=C`;
  after its first sample it costs ~0.1 s of CPU per sample — measured
  2026-10-01 on the dev Mac: `top` 1.84 % of a core at 5 s (2.2 s over
  2 minutes), the agent 0.77 % and 33.7 MB against 0.74 % and 33.4 MB
  before. A thread reads its stdout block by block
  (`top/footprint.rs`: a block is complete at the row count its
  `Processes: N total` line announced, or at the next `Processes:`
  line; one the stream ends inside is dropped; units `B K M G T` in
  1024 steps, a trailing `+`/`-` ignored, a line that does not read
  skipped) and keeps the latest `pid → bytes` with its instant. A tick
  takes it when at most two intervals old. `top` is started at the
  tier's first read (the option off forks nothing), killed and reaped
  when the tier is dropped (a panic's rebuild leaves no orphan), and
  started again at most once per 30 s when it ended, with one log line.
  One alive but silent for three intervals (5 s at least: stopped,
  stuck) is killed and reaped with a log line, and restarted under the
  same 30 s limit. Its signal mask is cleared in the child
  (`sigmask::clear_in_child`: `Command` keeps the forking thread's) and
  `SIGPIPE` is back to its default, so a `top` left behind by a killed
  agent dies at its next write.
- **Figures.** `cpu_percent` is CPU time over the tick's window, 100 =
  one core, computed here on both platforms (never `ps %cpu`).
  `mem_bytes` is RSS on both platforms: shared pages count in every
  process holding them. `footprint_bytes` is macOS only, absent for a
  pid `top` did not list. When the reading has a footprint map, memory
  ranks on it and a pid without one has no memory rank — never ranked
  on its RSS, the two do not compare; without a map (Linux, or `top`
  down), memory ranks on the RSS.
  `net_*_bytes_per_s` are deltas of cumulative per-pid counters; absent,
  never zero, when the platform has no source, the source cannot see the
  process (Linux: TCP only, and other users' sockets only as root), or a
  counter went down over the window (a socket closed).
- **The first tick primes.** A collector's first reading publishes
  nothing; the table lands on the second tick, 5 s in. A process seen
  for the first time, or whose pid was reused, has no delta either: a
  process younger than one window has no row.
- **`None` keeps what is served.** The collector answers
  `Option<TopPart>`: `None` (the priming tick, two readings under 100 ms
  apart, a platform with no reader) lands nothing and the previous part
  stays, stamp included. So a collector rebuilt after a panic primes
  again without blanking the table.
- **A failed read blanks it.** When the process table cannot be read
  (macOS: `ps` failed or timed out; Linux: the walk found no process), an empty table replaces the one served —
  a stale one would pass for a sample — on any tick, a rebuilt
  collector's first included. The baseline stays, so the next reading is
  measured over both periods. A failed network source costs only the
  network columns of two windows; a wedged `nettop` or `ss` stretches
  its tick to the 2 s deadline, which the ticker absorbs (missed ticks
  are skipped, not caught up).
- **Names.** The executable's name, never a path or the arguments,
  cleaned and capped like the process-tree names. Linux: the kernel's
  `comm`. macOS: `ucomm`, the kernel's own, cut at 16 bytes; a name
  within four bytes of that cap (the padding hides a cut space, a cut
  character drops up to three bytes) is completed from the file name of
  the executable (`proc_pidpath`, unprivileged, no fork), kept only when
  it extends `ucomm`; when the kernel has no path (a deleted
  executable) the cut name stays. `ps`'s `comm` is never read: it is
  what the process wrote as its `argv[0]`, arguments included. `user` is the account name, or the uid as a number
  when it has no entry.
- **Tests.** Parsers on fixtures of real output
  (`crates/threepm-daemon/tests/fixtures/hostmetrics-top/`), on every
  platform; the core without an OS (`hostmetrics::top::`); the tier in
  `hostagent::sampler::` with fake collectors.