# Metrics Reference

Every metric 3pm collects, where it comes from, and how retrieval works.
Design rationale lives in OV `3pm-docs-archived/architecture` (§8); this is the field-level
reference. The research trail behind the host-metrics design (OTel
`hostmetricsreceiver` survey, Datadog gap analysis) is
`doc/3pm-multi-host-metrics-retrieval.md` in the archive
`~/3pm-rd-doc-2026-09-09.tgz` — it is no longer in any tree.

Three commands expose three scopes:

| Command | Scope | Wire type | Collected |
|---|---|---|---|
| `3pm ls` / `3pm show` | Per managed instance — its whole process tree | `ProcessRecord` | Continuously, on the metrics tick |
| `3pm ls --tree` / `3pm show` | Every process of the tree, one line each | `AppTree` | Same tick, served on request (`process_trees`) |
| `3pm internals` | The 3pm daemon itself | `DaemonInternals` | On request (self-sample rides the tick) |
| `3pm hostmetrics` | The whole host, and 3pm's own footprint on it (`3pmDaemon`) | `HostMetrics` | By the host agent process in cost tiers — cheap gauges every 1s, walks/forks every 10s, the top processes every 5s, NTP every 5min, 3pm's own processes + the daemon's counters (pulled over `rpc.sock`) every 10s — served from its cache (`collected_at_epoch_ms`, `sampled_at_epoch_ms`) |

All three support `--json` (the raw wire struct, one document on stdout).

## Retrieval model

- **The metrics tick** (`threepm-daemon/src/metrics/`): a daemon task
  reads the process table — cpu and memory in one pass — through one
  reused table (Linux: `/proc/<pid>/stat` read directly, one file a
  process, `metrics/procs_linux.rs`; elsewhere a `sysinfo::System`),
  every `THREEPM_METRICS_INTERVAL_MS`
  (default 1000ms, min 50), off the runtime (`spawn_blocking`), and
  attributes every process to the instance it belongs to (**Process
  trees** below); the daemon's own pid rides the same pass. The whole
  table is swept on the first tick and every thirtieth (`metrics/table.rs`,
  `FULL_SWEEP_EVERY`); in between a tick lists the pids alive and
  refreshes only those born or gone since, the leaders, the members
  known and 3pm's own — each pid at most once a tick (a second refresh
  would reset its cpu delta), no stat file held open. The full sweep
  only resyncs a pid that died and was reborn between two listings
  (the table keeps the dead process's start and parent): a member
  reborn there joins its tree at once through its process group when
  the dead process was younger than its leader, otherwise at the next
  full sweep (≤ 30 s) — a reborn double-fork daemon, in its own
  session, always waits, since the environment's answer is remembered
  by pid and start. Every other membership fact (births, deaths,
  groups, environments, a member's name and parent) is live on every
  tick; a stranger's stale parent pid, since reused by a member, never
  pulls it in (a row older than its parent is not its child, and
  nothing is claimed below a row the tick did not read — a member
  reborn within the tick keeps the dead process's start until the next
  one reads it). CPU% is a
  delta between refreshes, so the first sample after a spawn — and a
  member's first tick — legitimately reads 0%; so does a process
  adopted on a partial tick until its last two refreshes are one tick
  apart (its first spans every tick it was left alone, and would read
  as that many ticks of work done in one). Samples are keyed by OS
  pid and gated against the *current* pid in the supervisor, so a
  dead/reused pid is never attributed to a stale row. The same tick
  lists the TCP LISTEN sockets (Linux: netlink `sock_diag`, `/proc/net/tcp*`
  if refused; elsewhere the `listeners` crate) and attributes each port
  to the member that holds it, walking a member's fd table only when it
  is new, every 10 ticks, or when a LISTEN socket appears that no member
  is known to hold — that last walk at most once every 5 ticks, so a
  stranger opening a listener every second costs one walk in five, and
  a member's own new listener shows within 5 s (at once on a quiet
  host). A daemon with nothing running does not
  sweep the table (its own pid is still sampled); `3pm internals` prices
  the last sweep (`last metrics tick`).
- **Reads that can hang, and a sweep that stalls.** Two reads take the
  target's memory-map lock or its exec lock in the kernel and block for
  as long as the target holds it (`D` state: a hung NFS mount, an
  `exec` or a page fault that does not finish; heavy memory pressure):
  Linux's `/proc/<pid>/environ` (the membership probe) and the `io` /
  `smaps_rollup` of Extended Monitoring; macOS's `KERN_PROCARGS2` goes
  the same way. The rest of the tick takes neither lock: the refresh
  reads cpu, memory, parent and name from `stat` and the uid once from
  `status` (Linux; macOS asks sysinfo for cpu, memory and the uid only —
  not its default `io`, `exe` and tasks), a member read this tick is
  named from that `stat` (as its `comm` file shows it), one that was
  not from `comm`, ports from the fd tables; `process_start_time`, called on the
  supervisor, reads `stat` alone (macOS: `proc_pidinfo`
  `PROC_PIDTBSDINFO`). One read is left unguarded: on macOS, sysinfo
  reads every newly seen pid's `KERN_PROCARGS2` on the sweep's thread,
  whatever the refresh asks — there the stall report below is the
  backstop. The policy (`metrics/risky.rs`):
  each risky read runs on a reader thread under a 500 ms deadline
  (`READ_DEADLINE`, `metrics/guarded.rs`); a tick spends at most 1 s
  on them (`TICK_BUDGET`), after which the reads left wait for a later
  tick, nothing held against them — a tick pays at most 1.5 s however
  many members are slow; a read that missed (or panicked) is not tried
  again on that process before a wait doubling from 1 to 60 ticks, and
  an answer clears it; while the reader abandoned on a process is still
  in its read, that process is not read at all (quarantine, daemon-wide:
  a table rebuilt after a sweep panic keeps it), until the read returns
  or the process goes. A read not made is no answer: the process is not
  adopted through its environment this tick and is asked again on the
  next (the environment memo backs off only on an environment actually
  read unreadable), a member keeps its last extended counters (then
  possibly older than one read cycle: the wire carries no age for
  them). With four readers stuck at once, daemon-wide (`MAX_STUCK`,
  `guarded::STUCK`), both
  reads are off until one returns (one `error` in the daemon log, one
  `info` when they resume): a host where everything hangs does not grow
  the daemon a thread per tick. Cpu, RSS, the trees and the listeners
  go on, and so do `max_memory_restart` and `max_cpu_percent`. One
  sweep runs at a time; one that runs past `max(5 × period, 5 s)`
  anyway is reported: an `error` in the daemon log, `3pm internals`
  shows `metrics sweep stalled for …` (`DaemonInternals.metrics_stalled_ms`,
  counted from when that sweep began) — no sample, so no resource
  guard, for that long — and the loop keeps waiting for that same sweep
  (a second one would queue behind the same read); its return is
  logged, the line goes, sampling resumes.
- **On-request snapshot** (`internals`): assembled when asked.
- **The host agent** (`hostmetrics`): a separate process
  (`3pm __host-agent`, supervised by the daemon) collects in three cost
  tiers on independent tickers — fast `THREEPM_HOSTMETRICS_INTERVAL_MS`
  (default 1s: cpu, memory, load, PSI, counters; stateful, the tick is
  the CPU-% window), medium `THREEPM_HOSTMETRICS_MEDIUM_MS` (10s:
  process/socket walks, sensors, GPU tools, `statvfs`, batteries), slow
  `THREEPM_HOSTMETRICS_SLOW_MS` (5min: NTP, identity) — and caches the
  assembled result; the daemon proxies `host_metrics` to that cache and
  reports `err{unavailable}` when the agent is down, never collecting
  inline (OV 3pm-docs-archived/architecture §8.2, `hostagent.md`, `hostagent-tiers.md`
  for the per-field tier).
- **Cumulative counters, not rates.** Where the OS provides monotonic
  counters (disk I/O, network bytes, TCP segments, page faults), 3pm
  reports them as-is. Rates (`r/s`, `%util`, `await`) require two samples
  over time — that derivation belongs to whatever polls the daemon
  periodically (a future fleet layer), not a one-shot CLI.
- **Absent, never faked.** A metric the platform can't provide is `None`
  or empty — never zero. Notable case: recent macOS zeroes `netstat`'s
  TCP counters for unprivileged callers; 3pm detects the all-zero read
  and reports the whole section absent.
- **Wire growth contract.** Every metrics struct is fully
  `#[serde(default)]`: fields can be added one at a time without breaking
  a version-skewed CLI/daemon pair, in either direction.
- **No elevation.** Everything below works as the unprivileged daemon
  user on stock systems.

Env knobs: `THREEPM_METRICS_INTERVAL_MS` (tick; floor 50 ms; a longer
tick widens the resource guards' settle margin to `max(10 s, 3 ×
period)`, see `docs/resources.md`), `THREEPM_PROCESS_TREE`
(`off`, `0` or `false`: sample the spawned pid alone, as before process
trees), `THREEPM_NTP_SERVER` (clock-offset probe server; empty string
disables the probe).

## Per-process metrics (`ProcessRecord` — `3pm ls`, `3pm show`)

| Field | Meaning | Source |
|---|---|---|
| `cpu_percent` | CPU% summed over the instance's process tree, 100 = one core (the spawned pid's own share is the first line of its tree) | the table sweep (Linux `stat`, computed as sysinfo computed it; sysinfo elsewhere), metrics tick |
| `mem_bytes` | RSS summed over the tree — an upper bound when members share pages | same |
| `listen_ports` | TCP LISTEN ports of every member of the tree (cluster workers roll up) | `listeners::get_all()` sweep on the tick; reported only while online, cleared on respawn |
| `pss_bytes` | Extended Monitoring (Linux): the members' PSS summed — shared pages split between their sharers, so not inflated the way the RSS sum is. Present only when every member has one (read or carried), never a partial sum: absent when any member's PSS is unreadable (another account's process under a user daemon) or not yet read (a newcomer over the tick's read budget), and on macOS (its `footprint_bytes` stays per member) and with `THREEPM_PROCESS_TREE=off`; refreshed every 10 s | `smaps_rollup`, see "Extended counters" below |
| `net` | The tree's network traffic over the last 5 s window and since the incarnation was first measured: `rx_bytes_per_s`, `tx_bytes_per_s`, `rx_bytes`, `tx_bytes`, `partial` (a lower bound: notifications lost). Each socket once per tree. Absent before the meter's second window, on a platform with no source, and with `THREEPM_PROCESS_TREE=off`. Every `TreeProc` has its own (every socket it holds, whole) | `metrics/net.rs`, see "Network traffic per app" below |
| `proc_count` | Live processes in the tree, the spawned pid included: `1` alone, `0` when not running (or from a daemon predating trees) | tree membership, metrics tick |
| `tree_rev` | Bumped when the listed membership changes (a pid enters or leaves the first 256 lines, `detached` flips, a member takes another name, the incarnation exits), never on cpu/mem; members beyond the cap do not move it; the next value of one counter per daemon — never back within one image, from 0 again on a daemon started cold; an in-place `3pm update` carries the counter but not the rows' revisions (an adopted row reports 0 until its next tick, and `boot_id` changes) | supervisor |
| `restarts` / `unstable_restarts` | Lifetime restarts / consecutive quick-crash count | policy state machine |
| `uptime_ms` | Current incarnation wall-clock uptime | daemon runtime state |
| `last_error` | Cause of the most recent spawn failure (e.g. `cwd '…' does not exist`) | spawner ENOENT diagnosis; cleared by a successful spawn |
| `status`, `pid`, `exit_code`, `instance`, `created_at_epoch_ms` | Lifecycle bookkeeping | supervisor |

## Process trees (`AppTree` — `3pm ls --tree`, `3pm show`)

An instance's figures are those of its **process tree**: the spawned pid
and every process the daemon attributes to it. `3pm ls` writes `saw +13`
when an app has other processes, `3pm ls --tree` (`-t`) draws them under
the row, `3pm show` lists them all, `--json` carries the tree under a
`tree` key (`Request::ProcessTrees`, `docs/protocol.md`). A daemon
predating trees answers `unknown_request`: the CLI renders without and
says `run \`3pm update\``. On a linked host the tree reaches the fleet
backend too — the shape when it moves, the figures every second — and
each member becomes a series there (`docs/backend-agent/link-protocol.md`
§6c).

### Membership

Each tick, for every running instance, in this order — a process
belongs to the first instance that claims it (`metrics/tree.rs`, pure
and table-tested; the OS lookups are injected):

1. **The cgroup** (Linux, `cgroup_enforce`): the leaf's `cgroup.procs`,
   nested cgroups included — exact, and it replaces the rules below.
2. **Descendants** by parent pid, from the spawned pid.
3. **The leader's process group** (`getpgid`): the child whose parent
   died and was reparented (to pid 1, or to the nearest subreaper —
   `systemd --user`, `tini`). Compared to the leader's *actual* group,
   so a leader that `setsid`s keeps its own, and only when that group
   is the leader's own (led by itself, by a member, or by a pid that is
   gone) — never a live stranger's.
4. **The environment** of a process no earlier rule placed: the
   spawner injects `THREEPM_ID`, `THREEPM_GEN`, `THREEPM_CTL_SOCK` and
   the daemon's lineage `THREEPM_BOOT`, inherited by every descendant,
   so the classic daemon (double fork + `setsid` before the first tick)
   is still tied to its app. The socket path must be this daemon's, so
   two daemons of one user (two homes) never adopt each other's
   children; and the lineage the incarnation's — drawn at every cold
   start, kept by an in-place `3pm update` — so a survivor of an
   earlier daemon of the same home, which carries the same numbers, is
   never adopted either. A malformed `THREEPM_BOOT` makes the process a
   stranger; an incarnation adopted from an image that predates
   lineages is recognised without one until its next spawn. A readable
   answer holds for the process's life (an `exec` keeps the
   environment); an unreadable or empty one is asked again 1, 2, 4… up
   to 60 ticks later (`metrics/probe.rs`).
5. **Memory**: a member stays one while it lives, by `(pid, start
   time)`, under the instance's `(id, gen)` — a new leader that reuses a
   dead one's pid inherits nothing, a restart starts from nothing.
   Looked up before the environment: it only ever confirms an earlier
   tick's claim.

Then every descendant, by parent pid, of a member rules 3–5 placed (the
shell a daemonized worker runs a job through, whatever its own
environment): membership is closed under parent pid. And no rule ever
claims a process that started before the instance's leader: a real
descendant cannot predate its ancestor, whatever a stale parent pid, a
leftover in a cgroup leaf or an inherited environment says. The same
holds link by link: a row older than its parent is not its child, so a
stranger whose parent died, and whose pid a member now reuses, is not
claimed through that stale parent pid (a subreaper member adopts only
its own orphaned descendants, all younger than it). And nothing is
claimed below a row the tick did not read (a stranger's, kept between
full sweeps): it may be a dead process's whose pid a member was just
reborn on, with the dead one's start — old enough for the dead one's
orphans to pass as the member's children. A stranger adopted from a
kept row (rules 3–4) is read the next tick, as a member, and brings its
children then, one level per tick.

Never a thread (Linux tasks are not processes), never a zombie, never a
row without a start time (macOS: another user's process — no parent, no
start, no uid), never 3pm's own daemon or host agent. A member outside the leader's process group is
**detached**, whichever rule placed it: `3pm stop`/`restart` signal the
group and will not reach it. `⚠` in `--tree`, `⚠ detached` in `show`,
`detached: true` in JSON — never on the spawned pid.

What the rules cannot see:

- **Another user's process.** Linux: cpu and RSS are readable, the
  environment is not — ppid and group still tie it. macOS: the row is
  ignored altogether (rule 3 would otherwise attach a `sudo` child with
  0 bytes and a 56-year uptime).
- **macOS platform binaries.** The OS withholds the environment of
  Apple's own binaries (`/bin/sh`, `/bin/bash`, `sleep`, `perl`…) even
  from their owner: a daemonized `bash` is not adopted by rule 4 —
  until it `exec`s a program whose environment reads fine, the
  environment being asked again —, a daemonized `node`, `python` or
  Homebrew binary is.
- **A nested daemon.** An app that runs `3pm start` against a home
  with no daemon spawns one as its own descendant, inside the app's
  process group (the launcher neither `setsid`s nor sets a group): that
  daemon and its apps join the tree and count toward the limits.
- **After an in-place `3pm update`** the memory (rule 5) is rebuilt on
  the first tick from rules 1–4; only a detached member whose
  environment cannot be read is lost — another user's, or on macOS an
  Apple platform binary's. After a cold start (a crash, `3pm update
  --cold`) what an earlier daemon left running is never a member: its
  lineage is not the new daemon's.

### Figures

| Field | Meaning |
|---|---|
| `AppTree.id`, `gen`, `rev` | The instance, its incarnation, and the record's `tree_rev` at the tick |
| `AppTree.cpu_percent`, `mem_bytes` | The totals of the same tick as the lines, omitted members included — a `list` answered from another tick may differ, so a reader sums nothing itself |
| `AppTree.procs[]` | `procs[0]` is the spawned pid; a parent precedes its children, siblings by pid; members whose parent is not a member follow, by pid; empty when the instance has at most one process or is not running |
| `AppTree.omitted` | Members cut by the caps (elided at 0); their figures are still in the totals |
| `TreeProc.pid`, `ppid`, `name` | The kernel's ppid (not necessarily a member); `name` is the one the OS gives the process now, read again every tick (a new name moves `tree_rev`): on Linux its `comm`, 15 bytes at most — the executable's file name after an `exec`, or what the program set since with `prctl(PR_SET_NAME)`, which is how Node's `process.title` shows —; on macOS the executable's file name (it follows an `exec`, never a `process.title`); elsewhere the name first seen, 64 bytes at most, controls and bidi marks blanked, private-use glyphs kept, never the arguments — text the app controls (`process.title`, `PR_SET_NAME`) |
| `TreeProc.cpu_percent`, `mem_bytes` | This member's share: percent of one core (0 on its first tick), RSS |
| `TreeProc.started_at_epoch_ms` | Second resolution; uptime in the table, identity against pid reuse |
| `TreeProc.detached`, `listen_ports` | Elided when false / empty; 4 ports at most, the record's `listen_ports` is the uncapped union |
| `TreeProc.ext` | Extended Monitoring's counters, below; elided when the mode is off, and shed from the reply's tail past its byte budget (`tree::budget_extras`) |

Caps: 256 lines per tree, 2 000 per reply — `min(256, 2000 / n)` per
instance, `n` the instances answered that list members, at least one
line each, so a reply never drops an instance whole and an app alone or
stopped never shrinks another's share — 4 ports per line, 64-byte names. Memory is the **sum of the
members' RSS**, an upper bound when they share pages (Electron helpers
share the framework). With Extended Monitoring on, `pss_bytes` (Linux)
is the honest sum; the cgroup's `memory.current` is not read.

### Extended counters (`TreeProc.ext`)

`[hostmetrics] extended` (on by default, `THREEPM_HOSTMETRICS_EXTENDED`)
has the daemon read, for the **members of its trees only** — never the
rest of the host —, the kernel's own counters (`metrics/ext.rs`). A
member is read on the first tick that finds it, then once every ten
ticks (10 s at the default interval); in between its last value is
carried, present and unchanged. A tick reads at most 256 processes
(`EXT_READS_PER_TICK`), first sights included: a member over the budget
keeps its carried value (none if never read) and is read first on the
next tick — the never-read, then the longest unread. A member that leaves, and a pid reused (another
start time), start over. Every field is absent where the platform or
the permissions do not give it, never zero: no uid is checked, each read
that fails (`EACCES`, `hidepid`, a process gone) leaves its fields out and
the next read tries again — a user daemon reads another account's member
(`[users]`, a `setuid` child) without `io`, fds or PSS. A read slower
than its 500 ms deadline, or over the tick's 1 s budget for such reads,
leaves the member its last values, asked again later with a backoff
(**Retrieval model**, `metrics/risky.rs`).

| Field | Linux (`/proc/<pid>/…`) | macOS (`libc`) |
|---|---|---|
| `minflt` / `majflt` | `stat` | — / `ri_pageins` (pages read in) |
| `ctx_voluntary` / `ctx_involuntary` | `status` | — (the kernel's one combined count is not split) |
| `read_bytes` / `write_bytes` | `io` (storage) | `ri_diskio_bytesread` / `…written` |
| `rchar`, `wchar`, `syscr`, `syscw` | `io` | — |
| `threads` | `stat` | `PROC_PIDTASKINFO` |
| `fds` / `fd_limit` | fd count / `limits` soft `RLIMIT_NOFILE` (absent when unlimited) | `PROC_PIDLISTFDS` / — |
| `pss_bytes` | `smaps_rollup` | — |
| `footprint_bytes` | — | `ri_phys_footprint` |
| `swap_bytes` | `status` `VmSwap` | — |

Cost, measured on `edgecr4ft.net` (release, non-root): 161 µs per
sleeping child, 471 µs per real process (a larger address space makes
`smaps_rollup` longer). The budget bounds a tick's reads at 256, about
120 ms at 471 µs each, on the tick's blocking thread. `THREEPM_PROCESS_TREE=off`
reads nothing.

### Rendering

`3pm ls --tree` compacts (`threepm_protocol::tree::compact`, pure, in
the protocol crate so the fleet backend serves the same picture on
`GET …/nodes/{node}/trees`, `docs/backend-agent/link-protocol.md` §6c): a
chain of single children is one line
(`npm › bash › node › Electron`: figures summed, pid and uptime of the
last link, ports in union), siblings with the same label are one line
(`Electron Helper ×4`: figures summed, no pid, uptime of the oldest),
a detached link cuts its chain and is never grouped, so `⚠` is never
hidden; `--all` unfolds everything. Tree rows are dimmed, take the
table's columns (`id`, `↺`, `status`, `watching` empty) and widen
`name` up to 32 columns of label plus the guides; `… N more` closes a
capped tree. With `--watch` the period defaults to 1 s (the daemon's
tick) and a tree reply that overruns the frame's budget is left out of
that frame, with the host line and the orphan notice (the connection is
reopened for the next one). `3pm show` says `cpu 62.2% (14 processes)`,
`mem 3.0gb (14 processes, RSS sum)` and lists every process,
uncompacted, under ` Processes`. With Extended Monitoring, the `mem` row
reads `3.0gb RSS · 2.2gb PSS (14 processes, sums)` when the record has
`pss_bytes`, and the table gains `pss` (Linux) or `footprint` (macOS),
`threads` and `fds` — each column drawn only when a member has it, `-`
for a member without; the other `ext` counters are `--json` only
(`render::tree::EXT_COLS`, `render::show::rows::mem_text`).

### Switch and cost

`THREEPM_PROCESS_TREE=off` on the daemon restores the pre-tree sampler:
the spawned pid alone is refreshed, `proc_count = 1`, empty trees, the
limits judge it alone, ports attributed by process group. Otherwise one
refresh of the table per tick under `spawn_blocking` — of the whole table
every thirtieth tick, about 12 ms for a thousand processes on the
development Mac, listener scan included; in between, of the pids born
or gone, the leaders, the members and 3pm's own only —; `metrics_tick_ms` (`3pm internals`, `last
metrics tick`) prices it, and the daemon logs `metrics tick slower than
its period` when a tick first overruns, once per streak (the ticker then
delays, cpu deltas stay right). Bench: `bench_members_and_build_on_a_thousand_rows`
(`--release`, `#[ignore]`).

## Application metrics (`Request::AppMetrics` — `3pm show`)

What the app's own processes report on the control channel
(`docs/protocol.md`, "Control channel"), pm2's custom metrics without
the Node-only IPC: any process of the app connects to `ctl.sock`, says
`hello` with its `source` (`cv-engine`, `renderer`…) and sends its full
set of metrics every second, plus events as they happen. Nothing is
injected into the child; the app embeds an SDK (`packages/`) or writes
the NDJSON itself (`examples/metrics.py`). The SDKs — `@3pm/api` (Node,
Electron), `threepm-io` (Python), `threepm_io.hpp` (C++17) — and their
common surface: `docs/app-metrics.md`.

| Kind | Semantics | Stored as |
|---|---|---|
| `gauge` | last value | one series |
| `counter` | cumulative since process start (rates at read time) | one series |
| `histogram` | base-2 exponential buckets (`exphist`), cumulative `count`/`sum`/zero/buckets, quantiles at read time | 2 + one per non-empty bucket (+ zero) |

The daemon keeps the **last value** of every series `(source, name,
labels)` on the instance's runtime — no time series, nothing on disk —
and answers `app_metrics` with **the whole state**, always
(`def`/`val`/`hist` by daemon-assigned `sid`). It cuts no diff: a
reader that wants one — the fleet agent, for the link — holds what it
last sent and compares (`threepm_link::node_state`). The request is `id`
alone, and the answer says nothing about diffing: protocol v2 took out
the five fields that pretended otherwise. What the daemon does keep
is a **revision**, so a reader knows whether asking is worth it: only a
batch that defines or changes something moves `app_metrics_rev` — an
idle registry flushed every second leaves it alone —, and so does a
control connection attaching with a `source` or closing (the sources
are part of what a reader holds). Cost of a whole answer at the series
cap: 0.28 ms and 120 KB for 950 series (release, M-series), 5.5 ms a
second for the twenty instances a node manages at most. `ProcessRecord`
carries only `app_series` (live series, buckets included),
`app_metrics_rev`, `app_metrics_rejected` and `app_events_dropped`; the
series never ride the record (`3pm ls` stays small, the 1 MiB codec cap
stays far).

Bounds, per instance incarnation (constants in `app_metrics/limits.rs`):

| Bound | Value |
|---|---|
| live series, buckets included | 1 000 |
| distinct names | 200 |
| label sets per name; the excess folds into `overflow="true"` (counters and histograms summed, gauges last) — an SDK that folds its own excess sends `overflow="true"` itself, the one reserved label it may use | 50 |
| creations per hour (anti-churn): series / names | 1 000 / 200 |
| series not refreshed for | 120 s → dropped by a sweep every 10 s, whether or not the sampler runs (a source that disconnects drops its series at once) |
| entries per `metrics` line / non-empty buckets per histogram (an SDK lowers its scale before reaching it) | 256 / 128 |
| events per second: per instance (burst) / per fingerprint (burst) | 10 (50) / 1 (10) |
| event attributes / exception type / message / stacktrace | 16 keys of ≤ 256 chars, ≤ 4 KiB / 128 / 1 KiB / 8 KiB, truncated |

Refused entries count in `app_metrics_rejected`, suppressed events in
`app_events_dropped` (both on the record, shown by `3pm show`); a
suppressed event surfaces as `repeat` on the next one of its fingerprint
that gets through. Every incarnation starts empty: the sid space, the
revision and the counters restart with the process.

Events go out on `pub.sock` as `Event::App` (OpenTelemetry log-record
shape: `event_name`, `severity_number`, flat `attrs`, `exc.{type,
message,stacktrace}`, `fingerprint`, `repeat`); the daemon stamps the
time and replaces the instance's `cwd` and `$HOME` in the trace by `.`
and `~`. `3pm events -f app` follows them.

## Daemon self-monitoring (`DaemonInternals` — `3pm internals`)

| Field | Source |
|---|---|
| `mem_bytes`, `cpu_percent` | The daemon's own pid rides the metrics tick (real sampled values) |
| `metrics_tick_ms` | Duration of the last metrics tick — the process-table sweep behind the trees; `0` before the first tick |
| `metrics_stalled_ms` | How long the sweep in flight has been running, counted from when it began, once past `max(5 × period, 5 s)` (**Retrieval model**); absent otherwise. `3pm internals`: `metrics sweep  stalled for …`, red |
| `uptime_ms`, `pid`, `version`, `home` | Supervisor state |
| `open_fds` | Count of `/dev/fd` entries (macOS + Linux) |
| `nofile_soft/hard` | `getrlimit(RLIMIT_NOFILE)`; `RLIM_INFINITY` normalized to `u64::MAX` = "unlimited" |
| `load_avg` | `sysinfo::System::load_average()` |
| per-status process counts, restart totals | Supervisor process table |
| `bus_subscribers` | Live `pub.sock` subscribers (e.g. `3pm logs` followers) |
| `metrics_interval_ms`, `daemon_log_bytes`, `dump_count`, `dump_saved_at_epoch_ms` | Config/env + file metadata |

Reading it: daemon RSS trending up across days = daemon leak (`3pm update`
restarts it cleanly); `open_fds` climbing toward the soft limit = fd leak;
stale dump age = `resurrect` would restore old state.

## Host metrics (`HostMetrics` — `3pm hostmetrics`)

Grouped by section. **Platform**: ✔ = both, L = Linux-only, M = macOS-only.

### Identity & uptime — ✔

`hostname`, `os_name`, `os_version`, `kernel_version`, `arch`,
`boot_time_epoch_s`, `uptime_s` — sysinfo statics.

### CPU

| Field | Pf | Source |
|---|---|---|
| `cpu_brand`, `physical_cores`, `logical_cpus`, `cpu_frequency_mhz` | ✔ | sysinfo (frequency hidden in the human view when implausible — Apple Silicon reports junk) |
| `global_cpu_percent`, `per_core_percent` | ✔ | sysinfo, two refreshes over the warmup window |
| `per_core_freq_mhz` | ✔ | sysinfo, every fast tick (1 s), one per core in `per_core_percent`'s order (Linux: each core's `scaling_cur_freq`); `cpu_frequency_mhz` is its first |
| `cpu_states` (user/nice/system/interrupt/idle) | ✔ | Linux: own two-sample `/proc/stat` delta; macOS: `systemstat` (`host_processor_info`) |
| `cpu_states.iowait/softirq/steal/guest` | L | `/proc/stat` delta — `steal` is the hypervisor noisy-neighbor signal (red > 5%) |
| `kernel` (context switches, forks since boot, procs running/blocked, interrupts) | L | `/proc/stat` (`ctxt`, `processes`, `procs_*`, `intr`) |
| `load_avg` (1/5/15m) | ✔ | sysinfo |
| `rate_window_ms` (top level) | ✔ | Agent: length of the window every `*_per_s` rate below was derived over (≈ the fast tier period); absent on the first sample |
| `sampled_at_epoch_ms.{fast,medium,slow,extended,top}` (top level) | ✔ | Agent: when each sampling tier last landed (Unix ms); a tier not run yet is absent, the whole object absent before tiered sampling |
| `pressure` (PSI: cpu/mem/io, some+full avg10/60/300) | L | `/proc/pressure/*` (kernel ≥ 4.20; absent when CONFIG_PSI is off). The modern stall-based saturation signal — prefer it over load average |

### Memory

| Field | Pf | Source |
|---|---|---|
| `mem_total/used/available/free`, `swap_total/used` | ✔ | sysinfo |
| `mem_detail` (cached, buffers, slab, shmem, dirty, writeback, committed_as, commit_limit, anon_pages, swap_cached) | L | `/proc/meminfo` |
| `paging` (pgfault, pgmajfault, pswpin/out, pgpgin/out, **oom_kill**, allocstall, compact_stall) | L | `/proc/vmstat` — `oom_kill` is OOM detection without a logs pipeline (red when > 0) |
| `mem_compressor` (compressions, decompressions, compressor size, pageins/outs, swapins/outs) | M | `vm_stat` subprocess — the macOS memory-pressure signal |
| `cgroup` (mem limit/free/rss, cpu quota in cores, pids current/max) | L | `sysinfo::cgroup_limits()` + `/sys/fs/cgroup/{cpu.max,pids.*}` — the container's real allowance (host totals lie inside Docker/k8s) |

### Processes — ✔

`processes_count` and `process_states` (runnable/sleeping/zombie/… counts)
from a walk of `/proc/<pid>/stat` on Linux (the sysinfo process table
elsewhere; the state names are sysinfo's, lowercased). Userland threads
(Linux tasks) are excluded; kernel threads are counted — `ps ax`
semantics. With the top-processes table on, Linux takes the count from
that tier's own walk (every 5 s) while it is younger than the medium
period: a medium tick then reports figures up to one top period older
than its own walk would have been.

### Network

| Field | Pf | Source |
|---|---|---|
| `networks[]` (per interface: `ips` as addr/prefix, mac, rx/tx bytes + packets + errors, cumulative) | ✔ | Linux: counters from `/proc/net/dev` every fast tick, mac + ips from sysinfo `Networks` + `ip_networks()` (link-local filtered) every 10 s and when an interface appears; elsewhere sysinfo for all. Human view hides interfaces with no traffic and no address ("active only"); `--json` keeps all |
| `networks[].rx_bytes_per_s/tx_bytes_per_s/errors_per_s` | ✔ | Agent-derived over `rate_window_ms` (absent on the first sample / after a counter reset); feeds the `ls` host line |
| `tcp_states` (connection counts by state: established, listen, time_wait, …) | ✔ | Linux: `/proc/net/tcp{,6}` — every socket, other users' and ownerless TIME_WAIT included. macOS: `listeners::get_all()` — only sockets attributable to a visible process |
| `tcp_counters` (retrans/in/out segs, opens, failures, resets; + listen_drops, **listen_overflows**, syn_retrans, memory_pressures) | L (partial M) | Linux: `/proc/net/snmp` + `/proc/net/netstat` TcpExt. macOS: `netstat -s -p tcp` subprocess — absent when the OS zeroes the counters for unprivileged callers. Listen drops/overflows = the kernel silently refusing connections while the app looks healthy (red > 0) |
| `sockets` (in use, tcp orphan/tw/alloc/mem, udp) | L | `/proc/net/sockstat` |
| `conntrack` (count/max) | L | `/proc/sys/net/netfilter/nf_conntrack_*` (absent when the module isn't loaded; red > 90% full) |

### Storage

| Field | Pf | Source |
|---|---|---|
| `disks[]` (mount, fs, kind, used/total, cumulative read/written bytes) | ✔ | sysinfo `Disks` |
| `disks[].read_bytes_per_s/write_bytes_per_s` | ✔ | Agent-derived over `rate_window_ms`; the `ls` host line sums them per distinct device |
| `disks[].inodes_total/free` | ✔ | `statvfs` per mount — inode exhaustion looks like a "70% free" disk that can't create files |
| `block_devices[]` (read/write IOPS, merges, sectors, in-flight, io-time, queue-time) | L | `systemstat` `/proc/diskstats` decode (macOS would need hand-rolled IOKit — deliberately not done) |
| `raid[]` (array, member map, degraded flag) | L | `/proc/mdstat` — degraded = disk-failure early warning (red) |

### GPU

One `gpus[]` row per device; per-vendor blanks stay absent. Sources by
platform: `nvidia-smi` CSV (any box with the NVIDIA driver — chosen over
NVML FFI for the same no-unsafe rationale as the IOKit refusal, 2s
subprocess timeout), `/sys/class/drm` sysfs (L), `ioreg` IOAccelerator (M).
An NVIDIA card on a box without `nvidia-smi` still appears as an
identity-only row from sysfs (vendor/pci id/driver, gauges absent).

| Field | Pf | Source |
|---|---|---|
| `vendor`, `model`, `pci_id`, `driver`, `driver_version` | ✔ | nvidia-smi / sysfs `vendor`+`device`+`uevent` / ioreg `model`+`vendor-id`+`IOSourceVersion` |
| `utilization_percent` (instantaneous gauge) | ✔ | nvidia-smi `utilization.gpu` / amdgpu `gpu_busy_percent` / ioreg `Device Utilization %` |
| `vram_used/total_bytes` (dedicated VRAM) | L | nvidia-smi `memory.*` / amdgpu `mem_info_vram_*` |
| `unified_mem_used/alloc_bytes` | M | ioreg `In use/Alloc system memory` — driver-held **shared** memory, deliberately not called VRAM |
| `core_count` | M | ioreg `gpu-core-count` |
| `temperature_c`, `power_w`, `power_limit_w`, `fan_percent` | L | nvidia-smi / amdgpu `hwmon` (µW→W, m°C→°C) |
| `recovery_count` | M | ioreg `recoveryCount` — driver-recovered GPU hangs (red when > 0: the GPU is faulting) |

### Power

One `batteries[]` row per pack, plus the host-wide `on_ac`. Linux only:
`/sys/class/power_supply`, world-readable attribute files — no root, no
FFI, no daemon. macOS would need IOKit and stays absent rather than faked.
"No battery" has two shapes — the directory absent and the directory
present but empty — and neither is an error.

The kind of a supply comes from its `type` (`BAT0`, `BAT1`, `ACAD`,
`axp20x-battery`, `macsmc-battery` all exist), never from its name; a
`scope=Device` supply is a mouse or a headset and is excluded, while an
**absent** `scope` means System. **Every reading is optional and an
absent one stays absent**: an amp-hour gauge has no `energy_now` at all
and most packs have no `temp`, while a full pack legitimately draws
`0 W` — `null` and `0` are different answers and the renderers show
`N/A`, never a zero.

| Field | Pf | Source |
|---|---|---|
| `on_ac` | L | Any non-battery supply's `online` (Mains, USB, USB_PD, Wireless, UPS). Absent when the host exposes no such supply at all — not the same as running on battery |
| `batteries[].name`, `state`, `present` | L | The supply's directory name, `status` verbatim (`Charging`/`Discharging`/`Full`/`Not charging`/`Unknown` — the direction of the flow comes from here, never from the sign of a reading), `present` (`0` = an empty bay: the bay is reported, its stale numbers are not) |
| `batteries[].capacity_percent` | L | `capacity`, else derived from `energy_now`/`energy_full` |
| `batteries[].capacity_level` | L | `capacity_level` (`Critical`…`Full`) — the coarse last resort, carried only when no numeric capacity exists |
| `batteries[].energy_wh`, `full_wh`, `design_wh` | L | `energy_*` (µWh) directly, or `charge_*` (µAh) × volts on an amp-hour gauge; absent when such a gauge reports no usable voltage either |
| `batteries[].power_w` | L | `power_now` (µW), else `current_now` (µA) × volts. **Unsigned**: gauges disagree on the sign, so the magnitude is carried |
| `batteries[].voltage_v` | L | `voltage_now`, else `voltage_min_design`; a `0` reading is a driver exposing the file without the measurement, not a battery at zero volts |
| `batteries[].cycle_count`, `technology`, `model` | L | `cycle_count`, `technology`, `model_name` |
| `batteries[].temperature_c` | L | `temp`, tenths of °C — usually absent, and then absent; legitimately negative on a cold pack |

Health is `full_wh / design_wh` and is left to the consumer, like every
other derivative here. It can legitimately exceed 100% on a fresh pack:
nothing clamps it.

### System health

| Field | Pf | Source |
|---|---|---|
| `file_handles` (allocated/max, host-wide) | L | `/proc/sys/fs/file-nr` |
| `vfs_cache` (dentries, inodes) | L | `/proc/sys/fs/{dentry-state,inode-nr}` |
| `entropy_avail` | L | `/proc/sys/kernel/random/entropy_avail` |
| `sensors[]` (label, temp/max/critical) | ✔ (thin off Linux/Apple-Silicon) | sysinfo `Components` (hwmon / Apple PMU) |
| `ntp_offset_ms` | ✔ | `rsntp` SNTP probe, 500ms timeout, best-effort — clock drift breaks TLS/auth/log ordering and nothing else detects it (red past ±100ms) |

### 3pm itself (`3pmDaemon`) — ✔

What 3pm costs the box, measured by the agent (not self-reported), on
the medium period; the whole object is absent from a daemon predating
it and until the tier has landed once (`hostagent.md` → *3pm's own
metrics*).

| Field | Pf | Source |
|---|---|---|
| `3pmDaemon.at_epoch_ms` | ✔ | Agent: when this part landed (Unix ms) |
| `3pmDaemon.processes[]` — `role` (`daemon`, `host-agent`), `pid`, `cpu_percent`, `mem_bytes`, `uptime_ms` | ✔ | Agent: `sysinfo` refresh of the two pids (its parent and itself); cpu % is a delta over the tier's window (0 on the first sample); a pid the kernel no longer knows has no row |
| `3pmDaemon.processes[].open_fds` | ✔ agent row; daemon row: L from the OS, ✔ via the daemon | `/dev/fd` for the agent itself, `/proc/<pid>/fd` for the daemon on Linux; elsewhere the daemon row takes `Internals.open_fds` (the daemon's own count) |
| `3pmDaemon.processes[].threads` | L | `Threads:` of `/proc/<pid>/status`; absent elsewhere |
| `3pmDaemon.runtime` — `version`, `boot_id`, `uptime_ms`, `apps_total`, `online`/`launching`/`stopping`/`stopped`/`waiting_restart`/`errored`, `restarts_total`, `unstable_restarts_total`, `timers_armed`, `bus_subscribers`, `log_sinks`, `log_rotations_total`, `log_bytes_written_total`, `daemon_log_bytes`, `nofile_soft`/`nofile_hard`, `handoffs`, `host_agent_restarts` | ✔ | Agent: `Request::Internals` over `rpc.sock` (one connection per tick, 2 s bound), reduced to this health subset of `DaemonInternals` — no `home`, no pid, no rotation policy. Absent for the tick when the daemon did not answer (down, wedged, handing off, skewed) |

### Extended Monitoring (`extended`) — L

The kernel's own counters, family by family (`HostExtended`: `families`
→ `kind` + rows → fields, and `unavailable` → reason); absent with the
mode off, present and empty on macOS. The human view is `3pm hostmetrics
--extended` (`hostagent.md` → *Extended Monitoring*); the default view
never shows them.

### Top processes (`top_processes`) — L, macOS

The heaviest processes of the whole host, one table of at most 30 rows
sorted by pid: the ten largest by CPU, by memory (the footprint on
macOS, the RSS on Linux) and by network rate (in + out), merged. Absent with `[hostmetrics] top_processes` off,
on the first tick (it primes), and on a platform with no reader;
`sampled_at_epoch_ms.top` is when the table last landed, which is how a
reader tells a stale one.

| Field | Source |
|---|---|
| `pid`, `name` | Linux: the kernel's process name. macOS: `ps` `ucomm` (16 bytes), completed from the executable's file name (`proc_pidpath`) when it may be cut. Never a path or arguments; cleaned and cut like a tree name |
| `user` | the account of the process's uid; the number when it has no name; absent when the platform does not say |
| `cpu_percent` | cumulative CPU time differenced over the real window between two readings, 100 = one core. A pid seen for the first time (or reused) has no row that window |
| `mem_bytes` | RSS, on both platforms. Not the PSS / footprint a managed process has in its tree |
| `footprint_bytes` | macOS only: `phys_footprint`, what Activity Monitor calls Memory, from one `top -l 0 -s <period> -stats pid,mem` kept running by the host agent (~1.8 % of a core at 5 s after a 0.78 s first sample, measured 2026-10-01; a `top -l 1` per tick would cost 15 %). A sample is used while at most two periods old. Absent on Linux, for a pid `top` did not list, and while `top` is down. When a table has it, the memory top was ranked on it — a row without one has no memory rank, never its RSS; without it, on the RSS |
| `net_rx_bytes_per_s`, `net_tx_bytes_per_s` | cumulative per-pid counters differenced over the window. macOS: `nettop -P -n`. Linux: `sock_diag` (both families in one dump, `tcp_info`) and the fd tables, as `ss -tinpH` reported them (`ss` itself where `sock_diag` is refused), TCP only, `bytes_received` and `bytes_sent` (`bytes_acked` before kernel 4.19) summed per holder — a shared socket counts whole for each; other users' sockets only under root. Absent, never zero, when there is no source, the process is not visible to it, or a counter went backwards |

A failed process read publishes an empty table. No series is stored for
these rows by the backend (pids churn); they are live state.

## Known gaps (deliberate)

- **Rates/derivatives** (`await`, `%util`, per-second anything) — needs
  periodic polling; belongs to the consumer (see Retrieval model).
- **Exact memory of a tree** — the RSS sum double-counts shared pages.
  Extended Monitoring reads PSS (`/proc/<pid>/smaps_rollup`, summed into
  `pss_bytes`) and `phys_footprint` (macOS, per member only); with the
  mode off, and on macOS for the app total, the RSS sum is all there is.
  The cgroup's `memory.current` is not read.
- **Detached members and `3pm stop`** — a member that left the process
  group is counted and marked, not signalled: the stop leaves it alive.
- **macOS**: disk IOPS (IOKit) and page-fault counters (raw mach
  `host_statistics64`) — no maintained crates; hand-rolled unsafe not
  worth it.
- **Different-league sources**: per-IRQ rates, NUMA, buddyinfo, schedstat,
  SMART, systemd unit states — need root, heavy deps, or a daemon of
  their own. (Wi-Fi link quality, softirqs, `vmstat`, `/proc/net/*` and
  the other kernel counter files are Extended Monitoring families, Linux.) (Batteries were listed here until the Power section: sysfs
  `power_supply` needs none of the three.)
- **GPU processes / clocks / ECC detail** — the `gpus[]` section covers
  inventory + core gauges; per-process GPU memory, clock speeds and ECC
  counters would need NVML/vendor FFI, declined for the same no-unsafe
  reason as IOKit.

## Network traffic per app

`ProcessRecord.net` and `TreeProc.net`, measured by the daemon on its
metrics loop, one window every 5 ticks (`metrics::net::WINDOW_TICKS`; 5 s
at the default interval). The first window primes; figures are carried
between windows; the meter starts with the first app and is dropped by
the first tick with none (no `nettop`, no notification socket, no
thread on an idle daemon), and a handoff rebuilds it, so the counters
restart then.

**The app's figure is its traffic with the outside of the app.** A
connection whose two ends are processes of the same app (Electron ⇄
its `node`, `npm` ⇄ its server, on loopback or the host's own address)
is counted for each process (`TreeProc.net`, the `net ↓/↑` column of
`3pm show`'s ` Processes` table) and not for the app
(`ProcessRecord.net`, `3pm ls`, the backend's `process.net.*`): each of
its bytes would come in twice, once per end. The members' figures can
therefore add up to more than the app's. Traffic with another app or
with a local database still counts. One opened and closed between two
windows on Linux is seen on its accepted side alone and counts once.

| Platform | Source | Counts | Misses |
|---|---|---|---|
| Linux | `sock_diag` dumps of our TCP sockets with `INET_DIAG_INFO` (both families in one, `TCPDIAG_GETSOCK`: one walk of the kernel's table, not one per family) (`tcpi_bytes_received`, `tcpi_bytes_acked`), filtered in the kernel on our local ports (`INET_DIAG_REQ_BYTECODE`); the members' fd tables (`socket:[inode]`); the `SKNLGRP_INET{,6}_TCP_DESTROY` notifications (final counters by cookie) behind a BPF filter on the same ports — all unprivileged | each socket's delta between dumps; a socket closed since the last dump: its tail, by cookie; an accepted connection, alive or closed, through its listener (whole the first time) | an outbound connection opened and closed within a walk's period (10 s, 30 s when the members hold 2 000 sockets or more); a listener shared by several instances (counted as `unattributed`); UDP |
| macOS | one `/usr/bin/nettop -n -d -s 1 -L 0 -x -J interface,bytes_in,bytes_out` (per socket: each process row followed by its sockets) kept running on a pty (on a pipe it buffers ten seconds of output), under a sentinel `sh` that kills it when the daemon's pipe closes (`nettop` outlives its terminal), its stdin on that pipe (on `/dev/null` it spins a core and a half), signal mask cleared, `LC_ALL=C` | per-process deltas, closed flows included, loopback (`lo0`) too; the `lo0` sockets that moved bytes, to pair the two ends of a connection inside an app; the first block (cumulative) skipped. Measured 2026-10-02: 0.32 s of `nettop` CPU per 10 s (0.07 with `-P`), 12 KB a second to read | UDP pairs; a dead or silent (10 s) `nettop` is replaced at most every 30 s, figures absent meanwhile |
| other | — | — | everything (absent) |

**What it costs is the apps', not the host's** (Linux). The dump returns
only sockets bound to our ports (the members' listeners and the ports of
the sockets they hold, at most 1 024 — beyond, an unfiltered dump keeping
ours, and the notifications on the listeners alone); another process's
connection closing is dropped by the kernel before it wakes the reader.
With no port of ours there is no dump and no subscription. A member's fd
table is read when the member is new, every member's every 2 windows
while they hold fewer than 2 000 socket inodes (30 s / 6 windows above),
and when a filtered dump shows a socket on our ports nobody is known to
hold (at most every 2 windows). A walk that found a TCP socket no dump
has shown is followed by one whole dump, to learn its port; inodes no
whole dump shows (unix, UDP) are remembered so they never ask again.
Measured 2026-10-02 (edgecr4ft, release): 40 000 host sockets churning
at thousands of connections a second from processes 3pm does not manage
cost the notification thread nothing; an app holding 10 000 connections
(20 000 fds) costs the daemon ~1.2 % of a core and ~18 MB over the same
daemon without it measuring (2.5 % and 30 MB in all).

The accounting is socket by socket (`metrics/net/sockets.rs`): a socket
alive at the dump counts `now − before`, or `now` when the previous dump
did not have it; a counter that went backwards is another socket; a
socket the previous dump had and this one does not is kept one window
for a late notification. Shared sockets: whole for each member holding
one, once per instance. An accepted connection credited by its listener
goes to the instance, and to a member only when one pid alone holds the
listener. The two ends of a connection are paired by address (the
peer's local address is this one's remote): when an instance owns
both, neither counts for it; the tail of such a socket, once closed,
stays out of it too. A notification the reader's buffer (50 000) or the kernel's
queue (`ENOBUFS`) could not hold is counted in
`DaemonInternals.net_meter.events_lost` and flags the window `partial`.
`3pm internals` shows the source (`netlink`, `netlink (no destroy
notifications: …)`, `nettop`, `unavailable: …`), the unattributed bytes
and the lost notifications.