# 3pm — full documentation

> 3pm is a fast, robust process manager written in Rust, with the pm2 workflow (same commands, ecosystem.config.js works): one binary that is both CLI and daemon, zero-downtime self-update, process trees, host metrics. Application metrics, events and remote actions come from SDKs for Node.js, Python, C++ and Rust. `3pm link` joins a host to 3pm-backend, a fleet backend (buckets of hosts, live dashboard, JSON HTTP API, SSE feed).

Generated 2026-10-13T15:51:31Z from https://3pm.edgecr4ft.net/llms.txt. Each document starts with a `<document>` line naming its URL.

<document url="https://3pm.edgecr4ft.net/docs/concepts.md" title="Concepts">

# 3pm concepts

The model behind every command, SDK call and API route, in one page. Each
section ends with the document that goes deeper.

## The process manager

- **One binary.** `3pm` is both the CLI and the daemon. The first command
  spawns the daemon (detached, its own session); a lock guarantees exactly
  one daemon per home.
- **Home.** `~/.3pm`, or `$THREEPM_HOME`. It holds the sockets, `3pm.log`
  (the daemon's own log), `logs/<app>-out.log` / `-error.log`, `pids/`,
  `dump.json` (`3pm save`), the optional `config.toml`, the installed
  `modules/`, the files actions leave in `actions/`, and the fleet link
  state in `agent/`.
- **Sockets.** The CLI and the daemon talk NDJSON over Unix sockets:
  `rpc.sock` (request/response), `pub.sock` (event bus: lifecycle, log
  lines), `ctl.sock` (apps talking back: readiness, graceful stop,
  application metrics, actions), `host.sock` (the host-metrics agent). Wire
  catalogue: [protocol.md](protocol.md).
- **App and instance.** An app is a name owning N instances (`-i N`). Each
  instance is a process row with its own `id`, pid, status and restart
  counter, and gets `THREEPM_INSTANCE` = 0…N-1. Commands target a name, an
  id, a glob (`'api-*'`), `all`, or every app of an ecosystem file.
- **Status.** `launching` → `online`; `stopping` → `stopped`;
  `waiting restart` (backoff); `errored` (crash loop: 16 exits under
  `min_uptime`, re-armed by `3pm restart`).
- **Supervision.** Restart policy `always` (default) / `on_failure` /
  `never`; instant restart or exponential backoff; stop = `kill_signal`
  (SIGINT) to the process group, then SIGKILL after `kill_timeout`;
  memory/CPU guards on the whole process tree; cron restarts; watch mode.
- **Ecosystem file.** The declarative form of `3pm start`: TOML, JSON,
  YAML, or pm2's `ecosystem.config.js`. `validate` checks it, `apply`
  reconciles the daemon with it (create / restart changed / leave the
  rest), profiles with `--env`. [ECOSYSTEM.md](ECOSYSTEM.md).
- **Readiness and graceful stop.** An app connects to `$THREEPM_CTL_SOCK`,
  says `hello`, then `ready` (`--wait-ready`) or receives `shutdown`
  (`--shutdown-with-message`). Any language, a few lines.
- **Logs.** Captured per instance, rotated by the daemon (20 MB × 5, 14
  days by default), streamed live by `3pm logs` from the event bus.
  [logs.md](logs.md).
- **Persistence.** `3pm save` writes `dump.json`, `3pm resurrect` restores
  it, `3pm startup` installs a boot service (systemd, launchd…) that does.
- **Zero-downtime update.** `3pm update` hands the daemon off to the new
  binary in place: same pids, same pipes, apps keep running. `install.sh`
  run again does it.
- **Containers.** `3pm runtime` is a PID 1 entrypoint: foreground, output
  streamed, exits when nothing is left to supervise. [RUNTIME.md](RUNTIME.md).
- **Hosted users.** A root daemon can host other accounts: one socket set
  per account, each seeing only its apps. [logs.md](logs.md) (`[users]`).
- **Resources.** Soft guards (`max_memory_restart`, `max_cpu_percent`),
  hard cgroup v2 limits (`cgroup_enforce`, Linux), `deny_gpu`.
  [resources.md](resources.md).
- **Modules.** Optional features shipped next to the binary and run as
  apps: `serve` (static file server, [SERVE.md](SERVE.md)) and `agent`
  (the fleet link). [modules.md](modules.md).

## Metrics

| Layer | Where it comes from | Read with |
|---|---|---|
| Process | CPU and memory of each instance's whole process tree, every second | `3pm ls`, `3pm ls --tree`, `3pm show` |
| Host | A separate host-agent process, cached, in three tiers (1 s / 10 s / 5 min): CPU, PSI, memory, disks, RAID, network, TCP, sensors, GPUs, NTP | `3pm hostmetrics` |
| Daemon | 3pm's own memory, fds, supervision counters | `3pm internals` |
| Application | What the app reports through an SDK: gauges, counters, meters, histograms, events, errors | `3pm show <app>`, `3pm events -f app` |

References: [metrics.md](metrics.md), [hostmetrics.md](hostmetrics.md),
[app-metrics.md](app-metrics.md).

## SDKs

Three SDKs with the same surface, talking to `$THREEPM_CTL_SOCK`; outside
3pm every call is a no-op, and none of them blocks or throws.

| SDK | Language | Install | Reference |
|---|---|---|---|
| `@3pm/api` | Node.js ≥ 18, Electron | `npm i https://3pm.edgecr4ft.net/sdk/node/3pm-api-<version>.tgz` | [sdk/node.md](sdk/node.md) |
| `threepm-io` | Python ≥ 3.9 | `pip install --index-url https://3pm.edgecr4ft.net/simple/ threepm-io` | [sdk/python.md](sdk/python.md) |
| `threepm_io.hpp` | C++17, header-only | CMake `FetchContent` from `https://3pm.edgecr4ft.net/sdk/cpp/` | [sdk/cpp.md](sdk/cpp.md) |

Versions and sha256 of every archive: `https://3pm.edgecr4ft.net/sdk/sdk.json`.

- **Metrics**: `gauge`, `counter` (cumulative), `meter`, `histogram`
  (base-2 exponential), `metric(name, fn)` (a gauge read at every flush);
  labels, at most 50 label sets per metric.
- **Events**: `event(name, {level, attrs})`, `notifyError(err)`.
- **Actions**: `io.action(name, handler)` exposes a function to
  `3pm trigger <app> <name>` and to the fleet backend; `3pm:profile` (CPU
  or heap profile, Node) is built in and driven by `3pm profile <app>`.
- **Node only**: the programmatic client `ThreePM` (pm2's API:
  `start`, `list`, `stop`, `restart`, `delete`, `launchBus`), and fleet
  calls `io.call` / `io.open` / `io.stream` to other nodes.

## The fleet: `3pm link` and `3pm-backend`

`3pm-backend` is the server side: one binary (axum + SQLite), optional
VictoriaMetrics (metrics) and VictoriaLogs (events) behind it, a dashboard
and a JSON API. A host running 3pm joins it with `3pm link`.

| Thing | Identifier | What it is |
|---|---|---|
| Account | token `3pmu_…` | Owns buckets. The API takes it as `Authorization: Bearer 3pmu_…`; the dashboard trades it for a session cookie. The first account registered adopts existing buckets. |
| Bucket | cluster id `3pmc_…` | A group of hosts and its certificate authority. The pasted cluster key is `3pmc_<id>` or `3pmc_<id>.<tls pin>`. |
| Enrollment token | `3pmk_<id>.<secret>` | Minted per bucket, pasted once: `3pm link 3pmc_… 3pmk_… wss://backend`. |
| Node | `3pmn_…` | One enrolled host, with a key and a certificate. `pending` until approved (unless the token pre-approves), revocable. Presence: `online`, `stale` (3 s without a frame), `offline` (30 s). |
| Session | `3pms_…` | One live WebSocket link (`/api/v1/link`, sub-protocol `3pm.link.v1`). |
| Node bearer | `3pma_…` | Minted for the node in every `welcome`: lets the node call the backend's HTTP routes itself, within the scopes its owner grants (`fleet:list`, `fleet:read`, `fleet:invoke`). |
| Capabilities | `metrics:read`, `proc:restart`, `proc:stop`, `proc:action` | What an agent offers the backend (`3pm link --cap`). |

- **Telemetry**: the agent sends the redacted process table and host
  gauges about once a second, a full host frame every 60 s, application
  metrics and process trees when they change. The backend keeps a live
  window in memory, persists to VictoriaMetrics/VictoriaLogs when
  configured.
- **Live feed**: `GET /api/buckets/{clusterId}/events` (SSE): `init`, then
  `node`, `telemetry`, `emit`, `call`, `artifact`.
- **Calls**: `nodeCall` (`restart`, `stop`, `ping`) and `nodeAction` (an
  `io.action`) place a command on a node; its state goes `sent` → `acked`
  → `replied` (or `unacked`, `refused`, `lost`); read with `getCall`.
  Files it leaves (profiles, dumps) are artifacts, kept 30 days.
- **Node to node**: `3pm fleet nodes|apps|inspect|top|call` and `io.call`
  use the node's own bearer.

References: the HTTP API [backend/api.md](backend/api.md) (every route,
its body, answer and auth), the backend's features
[backend/features.md](backend/features.md), the agent
[fleet/agent.md](fleet/agent.md), the wire protocol
[fleet/link-protocol.md](fleet/link-protocol.md).

## Distribution

- `curl -fsSL https://3pm.edgecr4ft.net/install.sh | sh`: macOS and Linux
  (glibc ≥ 2.28), x86_64 and arm64, into `~/.3pm/bin`; rerun to update in
  place.
- `https://3pm.edgecr4ft.net/latest/version.json`: the current release,
  its assets and sha256.

</document>

<document url="https://3pm.edgecr4ft.net/docs/readme.md" title="README">

# 3pm

A fast, robust process manager, written in Rust. `3pm` keeps your apps alive:
it daemonizes them, restarts them when they crash, aggregates their logs, and
tracks the CPU/memory of their whole process tree — with the pm2 workflow
you already know.

```console
$ 3pm start app.js
[3PM] Spawning 3pm daemon with home=/Users/you/.3pm
[3PM] App [app] launched
┌────┬──────┬───────┬────────┬───┬────────┬──────┬────────┐
│ id ┆ name ┆ pid   ┆ uptime ┆ ↺ ┆ status ┆ cpu  ┆ mem    │
╞════╪══════╪═══════╪════════╪═══╪════════╪══════╪════════╡
│ 0  ┆ app  ┆ 4021  ┆ 0s     ┆ 0 ┆ online ┆ 0.0% ┆ 12.3mb │
└────┴──────┴───────┴────────┴───┴────────┴──────┴────────┘
```

## Install

```console
curl -fsSL https://3pm.edgecr4ft.net/install.sh | sh        # any platform; rerun to update
curl -fsSL https://releases.3pm.dev/setup-apt.sh | sudo sh   # Debian/Ubuntu
curl -fsSL https://releases.3pm.dev/setup-yum.sh | sudo sh   # RHEL/Fedora
```

Or grab a tarball / `.deb` / `.rpm` from the
[releases page](https://github.com/Unitech/3pm/releases), or build from
source: `cargo install --path crates/threepm-cli`.

## Commands

| Command | What it does |
|---|---|
| `3pm start <script>` | Start any executable; interpreter inferred from the extension (`.js`→node, `.py`→python3, `.rb`→ruby, `.sh`→bash, `.pl`→perl, `.ts`→node); a quoted command line (`3pm start "npm run dev"`) runs under `bash -c` |
| `3pm start ecosystem.toml` | Start every app declared in a config file (`.json`, `.yaml`, pm2's `ecosystem.config.js` too; `-` reads it on stdin; no target = the current directory's `ecosystem.*`); `--env <profile>`, `--only a,b` |
| `3pm apply ecosystem.toml` | Reconcile with the file, idempotently: create / update / restart / leave unchanged; `--prune`, `--dry-run` |
| `3pm validate ecosystem.toml` | Check the file (fields, profiles, dependencies) without starting anything; exit 1 on errors |
| `3pm start app.js -i 4` | Cluster mode: 4 instances of one app (`-i 0` = one per CPU); each gets `THREEPM_INSTANCE`/`NODE_APP_INSTANCE` |
| `3pm scale <name> <n>` | Resize an app to n instances without restarting the running ones |
| `3pm stop\|restart\|delete <name\|id\|all>…` | Lifecycle control, one or more targets at a time (`3pm delete 0 1 api`); `stop` returns once the process is actually dead; `restart --update-env` re-reads the app's config file |
| `3pm reset <name\|id\|all>…` | Zero the restart counters (the process keeps running) |
| `3pm sendSignal <SIG> <name\|id\|all>` | Send a signal to the running process (`--group` for its process group) |
| `3pm ls` | The process table (`list`/`status` aliases) + pm2's `host metrics` summary line (cpu, load, ram, gpu, NIC/disk throughput) from the host agent's cache; `--watch [--interval MS]` redraws it in place, where the cursor is; `--tree` (`-t`) draws each app's process tree under its row (`--all` unfolds it), `cpu`/`mem` being the tree's totals |
| `3pm show <name\|id>` | Detailed view (`describe`/`info` aliases), every process of the app listed (with PSS/footprint, threads and fds under Extended Monitoring) |
| `3pm attach <name\|id>` | The terminal of an app started with `--tty` (`console` alias): its screen, your keys, `Ctrl-Z` detaches; piped, stdin goes to the app and its output comes out; `--peek [--json]` prints the screen, `--read-only` watches, `--raw` relays the app's bytes, `--instance N` |
| `3pm logs [name\|id…]` | Last 30 lines (`--lines N`, `--rotated` to reach into `.1`/`.2.gz`) then realtime stream; `--err` `--out` `--raw` `--nostream` `--timestamp[=FMT]` `--format` (logfmt) `--highlight TEXT` `--size` |
| `3pm flush [name\|id]` | Empty log files in place (all apps + `3pm.log` without a target) |
| `3pm rotate [name\|id]` | Rotate log files now (`<file>` → `<file>.1`, …) |
| `3pm reloadLogs` | Reopen every log file by path — the hook for an external logrotate in create/move mode (`kill -USR2 <daemon pid>` does the same) |
| `3pm logrotate [--print]` | Install (root) or print a logrotate(8) `copytruncate` drop-in, for hosts that centralize rotation |
| `3pm save` / `3pm resurrect` | Persist / restore the process list across daemon restarts |
| `3pm startup` / `3pm unstartup` | Install / remove a boot service so the saved apps come back after a reboot (prints a `sudo` command to run) |
| `3pm internals` | Daemon self-monitoring: memory/CPU, host load, fds vs limit, per-status counts, bus subscribers, log & dump gauges (`report` alias) |
| `3pm hostmetrics` | Host-wide OS metrics, sampled by a dedicated agent process (cheap gauges every second, walks/forks every 10s, NTP every 5min) and served from its cache: CPU (per-core, per-state incl. steal), PSI pressure, memory/swap + page cache + overcommit, OOM kills, cgroup limits, load, disks + inodes, RAID health, block-device I/O, network, TCP states & health (listen drops), sockets, conntrack, file handles, sensors, GPUs (utilization, VRAM/unified memory, temp, power), memory compressor (macOS), NTP clock offset, the top processes of the host by CPU, memory and network; `--extended`: the kernel's own counters, family by family (Extended Monitoring, Linux) |
| `3pm kill` | Stop the daemon and every managed process |
| `3pm update` | Upgrade the resident daemon in place (save → kill → respawn → resurrect) |
| `3pm completions <shell>` | bash/zsh/fish completions |
| `3pm serve [dir\|page.html] [port]` | Serve a directory (or one HTML page, opened in your browser) over HTTP as a managed app (pm2 `serve`): free port picked, local + LAN links and a QR code for your phone; `--spa`, `--ftp` listing, `--basic-auth-username/-password`, `--host`, `--homepage`, `--name` — a bundled module, see `docs/modules.md` |
| `3pm install <module\|dir>` / `3pm uninstall <module>` / `3pm module list` | Modules: optional features bundled with the install (`serve`, `agent`, `autoupdate`, `lan` — a virtual Ethernet between the root nodes of a bucket, with an optional network namespace holding only it for programs that bind to the default route's interface, which is its trust boundary: a member can claim another's node id until the daemon passes a verified caller); `install` copies one into `~/.3pm/modules`, starts its apps (shown under **Modules** in `ls`, shielded from `stop\|restart\|delete all`) and saves the list |

`3pm start` options: `--name`, `--interpreter` (`none`: run the script
itself), `--interpreter-args "--inspect"` (`--node-args`), `--cwd`, `--env KEY=VALUE`
(repeatable), `--namespace`, `--restart-policy always|on-failure|never`
(`--no-autorestart` = `never`), `--stop-exit-codes 0,3`, `--start-delay MS`,
`--depends-on APP` (repeatable; the app is parked until APP is online),
`--cron-restart "0 2 * * *"`, `--max-memory-restart 200M`,
`--kill-signal SIGTERM`, `--wait-ready`, `--shutdown-with-message`,
`--watch` / `--watch=PATH` (+ `--ignore-watch GLOB`, `--watch-delay MS`),
log destinations (`-o/--output PATH`, `-e/--error PATH`, `-l/--log PATH`,
`--merge-logs`, `--log-date-format FMT` / `--time`, `--disable-logs` — see
[Logs](#logs)), `--tty` (run on a pseudo-terminal, for interactive
programs: `3pm attach` connects to it), and `-- args…` passed to the app.

A bare word that names a managed app (or its id, or `all`) restarts that
app with its configuration, as pm2 does (`3pm stop api && 3pm start api`).
Otherwise the target is a file (relative to the cwd) or, when there is no such file
and it has no `/`, a command on your PATH: `3pm start npm -- run dev`. A
target with a blank that is not a file is a command line for `bash -c`,
named after its words (`3pm start "npm run dev"` → `npm-run-dev`). A
config file is recognized by its name (`.toml`, `.json`, `.yaml`/`.yml`,
`.config.js|cjs|mjs`); `-` reads the config on stdin, and no target at
all uses the current directory's `ecosystem.*`.

## Supervision behavior

- **Auto-restart**: crashed apps respawn at once (pm2 default); a crash
  loop reaches `errored` after 16 quick exits. `restart_delay` adds a
  fixed pause; `exp_backoff_restart_delay` (config or
  `--exp-backoff-restart-delay`) switches on exponential backoff
  (×1.5 per crash, capped at 15s, reset after 30s of stable uptime).
- **Restart policy**: `always` (default, pm2 behavior), `on_failure`
  (respawn only on a non-zero exit or signal death — a clean `exit 0`
  parks the app in `stopped`), or `never`. User `stop`/`restart` and the
  memory guard are never gated by the policy.
- **Crash-loop protection**: 16 consecutive exits below `min_uptime` (1s)
  mark the app `errored` and stop the loop; `3pm restart` re-arms it.
- **Stop sequence**: `SIGINT` (configurable per app) to the process group →
  1.6s (`kill_timeout`) → `SIGKILL`. When the incarnation's leader exits, the
  rest of its process group is swept — background children die with it.
- **Memory guard**: `--max-memory-restart 200M` gracefully cycles the app
  when the RSS of its whole process tree crosses the limit
  (`limit_scope = "leader"` judges the spawned pid alone).
- **Watch mode**: `--watch` restarts the app when files change under its
  cwd (or under `--watch=PATH`, repeatable). OS events (inotify / FSEvents
  / kqueue) with a trailing debounce (`--watch-delay`, floor 100ms);
  dotfiles and `node_modules` are always ignored, `--ignore-watch GLOB`
  adds to that list. Falls back to polling — and says so in `ls`/`show` —
  when the native watcher can't be set up. `3pm restart <app> --watch`
  toggles it, like pm2.
- **Orphan reconciliation**: if the daemon is killed uncleanly (`kill -9`),
  surviving app processes are identified by pid + start time on the next
  start/resurrect and reclaimed — never double-started. Until then
  `3pm ls` names them (`1 process from a previous daemon still running
  unmanaged: api (pid 4321)`) so a taken port is never a mystery.
- Every timing knob (`min_uptime`, `kill_timeout`, `restart_delay`,
  `exp_backoff_restart_delay`, `backoff_*`, `max_restarts`) is
  configurable per app.

## Graceful start & stop (control channel)

Apps can talk back to 3pm over a Unix socket — any language, four lines
of code ([examples/hello-ready.py](examples/hello-ready.py)):

```python
s = socket.socket(socket.AF_UNIX); s.connect(os.environ["THREEPM_CTL_SOCK"])
s.sendall((json.dumps({"msg": "hello", "id": int(os.environ["THREEPM_ID"]),
                       "gen": int(os.environ["THREEPM_GEN"])}) + "\n").encode())
s.sendall(b'{"msg":"ready"}\n')            # done booting — mark me online
```

- `--wait-ready`: the app stays `launching` until it sends `ready`
  (`listen_timeout`, default 3s, forces it online with a warning —
  pm2 semantics). `3pm start` returns when the app is actually ready.
- `--shutdown-with-message`: a graceful stop arrives as a
  `{"msg":"shutdown"}` line on the socket instead of a signal; clean up
  and exit (SIGKILL still follows after `kill_timeout` if you don't).

Node apps get pm2 parity for free later via a shim; everything is opt-in —
apps that never connect are supervised with plain signals.

## Boot persistence

Surviving a reboot needs the OS init system — nothing in userland restarts
itself after a power cycle. `3pm startup` prints the privileged command to
register a boot service (via the [`service-manager`](https://crates.io/crates/service-manager)
crate — launchd, systemd, OpenRC, rc.d):

```console
$ 3pm startup
[3PM] Init system: launchd
[3PM] To set up the startup script, copy/paste the following command:
sudo env PATH=$PATH THREEPM_HOME=/Users/you/.3pm /usr/local/bin/3pm startup launchd -u you --hp /Users/you
```

Run that once (it's the only step needing root); the service is system-level
but runs the daemon **as you** — it runs `3pm __daemon --boot`, which restores
whatever `3pm save` last recorded. The init system supervises the real daemon
and restarts it on crash. `3pm unstartup` prints the matching removal command.

## Containers

`3pm runtime` is the entrypoint: daemon in the foreground as PID 1, app
output on stdout/stderr, SIGTERM drains, and it exits `2` once nothing is
left to supervise so the orchestrator's restart policy takes over
(`--no-auto-exit` to stay up). Every `start` flag applies.

```dockerfile
CMD ["3pm", "runtime", "ecosystem.toml", "--env", "production"]
```

The apps run in a real daemon, so `docker exec c 3pm ls` / `logs` /
`events` work while it runs. See `examples/Dockerfile`.

## Ecosystem file

```toml
[defaults]                   # optional: every app starts from these
restart_policy = "on_failure"

[[apps]]
name = "api"
script = "./server.js"
args = ["--port", "3000"]
max_memory_restart = "300M"
max_cpu_percent = 80         # restart when a CPU sample exceeds this
limit_scope = "tree"         # what both limits judge: the whole process tree, or "leader"
cgroup_enforce = false       # Linux: make both limits hard (cgroup v2)
deny_gpu = false             # hide GPUs via CUDA/ROCm env vars
restart_policy = "on_failure"  # always (default) | on_failure | never
watch = true                 # or "src", or ["src", "config"]
ignore_watch = ["*.log"]     # added to the built-in dotfiles + node_modules
watch_delay = 500            # ms
stop_exit_codes = [0, 3]     # these exits park the app instead of restarting it
depends_on = ["worker"]      # start after every worker instance is online
start_delay = 1000           # ms before the first spawn
cron_restart = "0 2 * * *"   # restart nightly at 02:00 (5 or 6 fields)
log_date_format = "%Y-%m-%d %H:%M:%S"   # persisted prefix; none by default
log_rotate_size = "50M"      # per-app override of the daemon policy

[apps.env]
NODE_ENV = "development"
[apps.env_production]        # applied with --env production (pm2)
NODE_ENV = "production"
[apps.profiles.staging]      # any field, applied with --env staging
instances = 2

[[apps]]
name = "worker"
script = "./worker.py"
instances = 4
out_file = "/var/log/worker/{name}-{instance}.out"  # placeholders: {name} {instance} {namespace}
error_file = "NULL"                                # no file for stderr (the stream still flows)
merge_logs = false
```

Relative paths resolve against the config file's directory. Unknown fields
warn with a did-you-mean instead of failing (`3pm validate` makes them
errors). pm2's aliases (`out`, `error`, `err_file`, `log`, `combine_logs`,
`exec`/`cmd`/`command`, `exec_interpreter`, `node_args`, …) are accepted.
The same file can be written in JSON or YAML, or as pm2's
`ecosystem.config.js` (evaluated by your `node`, in your shell's
environment); see `docs/ECOSYSTEM.md`. `3pm start eco.toml` always
restarts (pm2); `3pm apply eco.toml` is the idempotent form — unchanged
apps keep running.

## Logs

Each instance writes `logs/<app>-out.log` and `logs/<app>-error.log`
(`<app>-<i>-…` for instances ≥ 1; `<app>.log` when merged). The file holds
the app's bytes verbatim; with `log_date_format` every persisted line is
prefixed `"<strftime>: "`. Custom destinations (`out_file`, `error_file`,
`log_file`) accept `{name}`, `{instance}`, `{namespace}`; a custom path
shared by several instances is infixed `-<instance>` automatically so two
instances never interleave in one file. `NULL` / `/dev/null` disables a
stream's file (it still streams), `disable_logs` disables everything.
An app on a terminal (`tty`) has one stream: one file (`<app>.log`),
stripped of escape sequences so it reads as text.

**Rotation is built in and hot**: the daemon counts bytes as it writes and
rotates the moment a file crosses the threshold — no restart needed, no
line lost or torn. Policy, most specific wins:

| Layer | Where |
|---|---|
| per app | `log_rotate_size` / `_files` / `_days` / `_compress` in the ecosystem file |
| environment of the daemon | `THREEPM_LOG_MAX_SIZE` (`"0"` = off), `THREEPM_LOG_MAX_FILES`, `THREEPM_LOG_MAX_DAYS`, `THREEPM_LOG_COMPRESS` |
| `~/.3pm/config.toml` | `[logs] rotate_size = "20M"`, `rotate_files = 5`, `rotate_days = 14`, `rotate_compress = false` |
| defaults | 20 MiB, 5 generations, 14 days, no gzip |

Generations are `<file>.1` (freshest) … `<file>.N`; with `compress`, `.2`
and older are gzipped (`.1` stays plain so it tails). An hourly sweep
prunes by count and age even for files that never rotate. `3pm internals`
shows the effective policy, open files and rotations; `3pm logs --size`
and `3pm show` show per-file sizes.

Reading: `3pm logs` tails then follows from the event bus (subscribe-first,
so a rotation or a chatty app never opens a gap). `--timestamp` stamps
streamed lines with their arrival time (the wire carries `ts`, epoch ms),
`--format` emits logfmt, `--json` emits the wire events — history and
stream in one schema.

## Architecture

```
3pm CLI ──rpc.sock (NDJSON)──▶ 3pm daemon ──▶ supervised processes
   ▲                              │
   └────pub.sock (event bus)◀─────┘   log lines + lifecycle events
```

One binary. The first `3pm` command spawns the daemon (detached, own
session); a `flock` guarantees exactly one daemon per home even under
concurrent cold starts. State lives under `~/.3pm` (`THREEPM_HOME`
overridable): sockets, `3pm.log`, `logs/<app>-{out,error}.log` (+ rotated
`.N`/`.N.gz`), `pids/`, `dump.json`, optional `config.toml`.

The workspace is split so the supervision logic stays pure and provable:

| Crate | Role |
|---|---|
| `threepm-core` | Sans-io domain: the lifecycle policy state machine, backoff, paths — no tokio, no I/O |
| `threepm-protocol` | Wire types, capped NDJSON codec, client (connect / auto-spawn / subscribe) |
| `threepm-daemon` | Runtime: supervisor actor, per-child tasks, pumps, RPC server, event bus, metrics |
| `threepm-cli` | The `3pm` binary: commands + pure renderers |
| `threepm-testsupport` | E2E harness (isolated homes, leak-proof daemon cleanup) |

Design notes worth knowing:

- All lifecycle decisions run through a pure `step(state, input, now) →
  actions` function — deterministically unit- and property-tested (the
  actor never mutates policy state itself).
- Signaling and reaping are serialized in one task per child, so a signal
  can never hit a reaped (possibly reused) pid.
- Log pumps are byte-oriented with a 64 KiB cap: binary output, `\r`-only
  progress bars, or invalid UTF-8 can't wedge logging or the daemon.
- Realtime `3pm logs` streams from the event bus (subscribe-first, then
  tail), not from file watching.

## Divergences from pm2 (deliberate)

- **Group-kill, not treekill**: signals go to the process group; a child
  that `setsid()`s itself escapes (pm2's ps-walk has different blind
  spots) — it still counts in the app's figures and `3pm ls --tree`
  marks it `⚠`. Wrap-scripts should `exec` the real binary.
- **Crash-loop accounting**: consecutive-quick-exit counting (reset by one
  stable run) instead of pm2's wall-clock window — the window breaks as
  soon as `restart_delay` or a backoff paces the restarts.
- Sockets are `0600` and peer-uid-checked (pm2 uses group-writable `775`).
- **Watch mode is derived from state**: one watcher per app (not per
  instance), armed while the app runs; `stop` turns watch off as in
  pm2, so the list never says "enabled" with no watcher. `ignore_watch`
  adds to the built-in ignores instead of replacing them, `watch_delay`
  is milliseconds everywhere, and a failed native watcher is reported
  (`watch_error`) instead of logged and forgotten.

## Development

Recipes live in the [`justfile`](justfile) ([just](https://github.com/casey/just),
`brew install just`):

```console
just             # list recipes
just test        # fast tests: unit, property, component (paused-time), snapshot
just e2e         # real binary against real daemons in isolated /tmp homes
just ci          # everything CI gates: fmt, clippy, snapshots, tests, e2e
just cov         # coverage report (cargo-llvm-cov; spawned daemons included)
```

Without `just`, the equivalent raw commands:

```console
cargo nextest run --workspace          # the whole suite
cargo clippy --workspace --all-targets -- -D warnings
cargo insta review                     # snapshot changes
```

The e2e suite runs the real binary against real daemons in isolated
`/tmp` homes; every test cleans up its daemon even on panic, and CI sweeps
stragglers by `--test-marker`.

## Documentation

- [Extensibility & pm2 Feature Map](docs/extensibility.md) — the extension
  seams (commands, spec fields, lifecycle triggers, control messages) and
  how the remaining pm2 surface maps onto them.
- [Metrics Reference](docs/metrics.md) — every metric `ls`/`show`/
  `internals`/`hostmetrics` report, the process tree behind an app's
  figures, per-platform sources, retrieval model.
- [Application Metrics SDKs](docs/app-metrics.md) — `io` for Node/Electron
  (`@3pm/api`), Python (`threepm-io`) and C++ (`threepm_io.hpp`): custom
  metrics, histograms, events, `notify_error`.
- [Host Metrics Agent](docs/hostagent.md) — the separate process behind
  `3pm hostmetrics`: cache, lifecycle, failure semantics, knobs, files;
  [Sampling Tiers](docs/hostagent-tiers.md) — the collector cost audit
  behind its fast / medium / slow periods.
- [Runtime Cheatsheet](docs/RUNTIME.md) — `3pm runtime` for containers:
  invocations, exit codes, Dockerfile shape, rules of thumb.
- [Ecosystem File Cheatsheet](docs/ECOSYSTEM.md) — every field with its
  default and unit, layering rules, `validate` / `apply` workflow.
- [Serve Cheatsheet](docs/SERVE.md) — `3pm serve`: invocations, flags,
  responses, attached mode, pm2 ecosystem compatibility.
- [Modules](docs/modules.md) — `3pm install` / `uninstall` / `module
  list`, `3pm <module> …`, the `module.toml` contract, writing a module.
- [Fleet link](docs/backend-agent/README.md) — `3pm agent` and
  `3pm-backend`: the wire contract, the shared crate, the agent module,
  the backend API, status and caveats.
- [Logs Reference](docs/logs.md) — capture, destinations, line format,
  hot rotation and its policy layers, `logs`/`flush`/`rotate`/`reloadLogs`.
- [Download site](docs/SITE.md) — https://3pm.edgecr4ft.net: what is
  online, the three `just publish*` commands that upload it, the landing
  page, the generated documentation and `llms.txt`, the nginx vhost.
- [Resource Limits Reference](docs/resources.md) — `max_memory_restart`,
  `max_cpu_percent`, `limit_scope`, `cgroup_enforce` (Linux cgroup v2),
  `deny_gpu`: cheatsheet, semantics, permissions, soft vs hard.

Six contributor-facing documents left the tree on 2026-09-09 and live in
OpenViking under `viking://resources/3pm-docs-archived/`: `architecture`
(internals: the policy state machine, daemon lifecycle, race hardening —
the §numbers quoted across this codebase are its), `development` (task
runner, test layers, CI map), `coverage`, `releasing`, `nix-testing`, and
`backend-agent/threepm-link`.

## License

MIT

</document>

<document url="https://3pm.edgecr4ft.net/docs/environment.md" title="Environment variables">

# Environment variables

Every variable 3pm, its modules, its SDKs, its installer and the fleet
backend read or set. Grouped by who reads it. Defaults are the values used
when the variable is unset or empty.

## Set by you, read by the `3pm` CLI and daemon

| Variable | Default | Effect |
|---|---|---|
| `THREEPM_HOME` | `~/.3pm` | The home: sockets (`rpc.sock`, `pub.sock`, `ctl.sock`, `host.sock`), `3pm.pid`, `3pm.log`, `logs/`, `pids/`, `dump.json`, `config.toml`, `modules/`, `actions/`, `agent/`. One daemon per home. Like pm2's `PM2_HOME` (which 3pm does **not** read). |
| `THREEPM_USERS_DIR` | `/var/lib/3pm/users` | Where a root daemon puts the homes of the accounts it hosts (`[users] allow` in `config.toml`); a client finds its hosted home there. |
| `THREEPM_MODULES_DIR` | `<exe dir>/modules`, then `<exe dir>/../lib/3pm/modules` | Where the bundled modules (`serve`, `agent`) are looked up; first hit wins. |
| `THREEPM_SPAWN_TIMEOUT_MS` | `3000` | How long a client waits for an auto-spawned daemon to be ready. |
| `THREEPM_JSON` | unset | `1` = same as `--json`: machine output on stdout (modules too). |
| `THREEPM_LOG` | `info` | The daemon's own log level (`3pm.log`), a `tracing` filter: `warn`, `debug`, … |
| `THREEPM_LOG_MAX_SIZE` | `20M` | Log rotation threshold per file; `0` turns rotation off. Per app `log_max_size` > env > `config.toml` > default. |
| `THREEPM_LOG_MAX_FILES` | `5` | Rotated files kept. |
| `THREEPM_LOG_MAX_DAYS` | `14` | Rotated files older than this are removed. |
| `THREEPM_LOG_COMPRESS` | off | `1`/`true`/`yes`/`on`: gzip rotated files. |
| `THREEPM_METRICS_INTERVAL_MS` | `1000` | Per-process CPU/memory sampling period (`3pm ls`). |
| `THREEPM_PROCESS_TREE` | on | `off`: sample the spawned pid alone instead of its whole process tree. |
| `THREEPM_HOSTMETRICS_INTERVAL_MS` | `1000` | Host agent fast tier (CPU, memory, load, network and disk rates). |
| `THREEPM_HOSTMETRICS_MEDIUM_MS` | `10000` | Host agent medium tier (process table, TCP states, sensors, GPUs, mounts, RAID). |
| `THREEPM_HOSTMETRICS_SLOW_MS` | `300000` | Host agent slow tier (NTP offset, host identity). |
| `THREEPM_NTP_SERVER` | `pool.ntp.org` | Clock-offset probe server; empty disables the probe. |

Variables of the daemon are read when it starts: set them on the command
that spawns it (`THREEPM_LOG_MAX_SIZE=50M 3pm ls`), then `3pm update` or
`3pm kill` to apply them to a running one.

## Injected by the daemon into every app it starts

| Variable | Value | Use |
|---|---|---|
| `THREEPM_INSTANCE` | `0` … `N-1` | The instance number of this process within its app (`-i N`). |
| `NODE_APP_INSTANCE` | same | pm2 compatibility. |
| `THREEPM_ID` | integer | The process id (`id` column of `3pm ls`). |
| `THREEPM_GEN` | integer | The incarnation, bumped at every restart; the control channel refuses a stale one. |
| `THREEPM_BOOT` | 16 hex digits | The daemon lineage, kept across `3pm update` handoffs; tells a child of this daemon from a survivor of a previous one. |
| `THREEPM_CTL_SOCK` | path | The control socket: `hello` (with id + gen), `ready` (`--wait-ready`), `shutdown` (`--shutdown-with-message`), and the application metrics, events and actions of the SDKs. Its absence makes every SDK call a no-op. |

These four — `THREEPM_CTL_SOCK`, `THREEPM_ID`, `THREEPM_GEN`, `THREEPM_BOOT` —
are inherited by every descendant, which is how 3pm attributes a process
tree to its app.

## Read by module commands (`3pm serve`, `3pm link`, your own)

| Variable | Set by | Meaning |
|---|---|---|
| `THREEPM_EXE` | the CLI | Path of the `3pm` binary. |
| `THREEPM_MODULE_DIR` | the CLI | The module's directory. |
| `THREEPM_HOME`, `THREEPM_JSON` | the CLI | As above. |

### `serve` module (pm2 `serve` names accepted too)

| Variable | Default | pm2 name |
|---|---|---|
| `THREEPM_SERVE_PATH` | `.` | `PM2_SERVE_PATH` |
| `THREEPM_SERVE_PORT` | `8080` | `PM2_SERVE_PORT` |
| `THREEPM_SERVE_HOST` | `0.0.0.0` | `PM2_SERVE_HOST` |
| `THREEPM_SERVE_SPA` / `THREEPM_SERVE_FTP` | `false` | `PM2_SERVE_SPA` / `PM2_SERVE_FTP` |
| `THREEPM_SERVE_HOMEPAGE` | `/index.html` | `PM2_SERVE_HOMEPAGE` |
| `THREEPM_SERVE_BASIC_AUTH_USERNAME` / `_PASSWORD` | none (both or neither) | `PM2_SERVE_BASIC_AUTH_*` |

### `agent` module (`3pm link`, the fleet link)

| Variable | Default | Meaning |
|---|---|---|
| `THREEPM_LINK_CLUSTER` | — | Cluster key `3pmc_…` when not given as an argument (provisioning scripts). |
| `THREEPM_LINK_TOKEN` | — | Enrollment token `3pmk_…`. |
| `THREEPM_LINK_SERVER` | `wss://fleet.edgecr4ft.net` | Backend address; wins over the argument. |
| `THREEPM_LINK_INSECURE` | off | `1`: accept a non-loopback `ws://` or any TLS certificate. |
| `THREEPM_AGENT_STATE_DIR` | `<home>/agent` | `node.key`, `node.crt`, `ca.pub`, `link.toml`, `session.json`, `data.sock`. |
| `THREEPM_AGENT_INTERVAL_MS` | `1000` (floor 200) | Telemetry period. |
| `THREEPM_AGENT_FULL_EVERY_S` | `60` | Period of the full host frame; the backend's `welcome` overrides it. |
| `THREEPM_AGENT_NAME` | `agent` | Name of the supervised agent app. |
| `THREEPM_AGENT_LOG` | `info` | Agent log level (set it on the `agent` app). |

## Read by the SDKs (`@3pm/api`, `threepm-io`, `threepm_io.hpp`)

| Variable | Meaning |
|---|---|
| `THREEPM_CTL_SOCK`, `THREEPM_ID`, `THREEPM_GEN` | Injected by the daemon (above): where and as whom to connect. |
| `THREEPM_DATA_SOCK` | Fleet calls (`io.call` / `io.open` / `io.stream`, Node): the linked agent's data socket; default `<home>/agent/data.sock`. Set it on both the agent app and your app to move the socket. |
| `THREEPM_IO_DEBUG` | `1`: trace to stderr every call the SDK refuses or drops. |
| `THREEPM_IO_MAIN` | Written by the Node SDK in its own environment (`<id>.<gen>.<pid>`) so child processes do not also claim the `main` source. Do not set it. |
| `THREEPM_HOME` | Node client (`ThreePM`): the daemon to talk to. |
| `THREEPM_BIN` | Node client: the `3pm` binary used to auto-spawn a daemon (default `3pm` on `PATH`). |

## Read by `install.sh`

| Variable | Default | Effect |
|---|---|---|
| `THREEPM_VERSION` | latest | Version to install (`sh -s -- 0.3.0` does the same). |
| `THREEPM_INSTALL` | `~/.3pm` | Install directory (`bin/3pm`, `modules/`). |
| `THREEPM_CDN_URL` | `https://3pm.edgecr4ft.net` | Download base. |
| `THREEPM_NO_MODIFY_PATH` | unset | Set: do not edit shell rc files. |
| `THREEPM_HOME` | `~/.3pm` | Where to look for a running daemon to hand off to the new binary. |

## Read by `3pm-backend` (the fleet backend)

Every flag of `3pm-backend` has an environment variable: `--listen` is
`THREEPM_BACKEND_LISTEN`, `--metrics-url` is `THREEPM_BACKEND_METRICS_URL`,
and so on. The full list with defaults is in [backend/cli.md](backend/cli.md).

| Variable | Default | Meaning |
|---|---|---|
| `THREEPM_BACKEND_LISTEN` | `127.0.0.1:9616` | Bind address. |
| `THREEPM_BACKEND_ADVERTISE` | hostname + port | What agents connect to (goes into the pasted `3pm link` line), e.g. `wss://fleet.example.com`. |
| `THREEPM_BACKEND_DB` | `3pm-backend.sqlite` | SQLite database. |
| `THREEPM_BACKEND_DATA_DIR` | `<db>.d/` | Artifacts left by actions (profiles, dumps) and blobs. |
| `THREEPM_BACKEND_DEV` | off | Clear-text HTTP/WS, localhost advertise allowed, open registration. |
| `THREEPM_BACKEND_REGISTRATION` | `closed` | `closed`: first account only; `open`: anyone. |
| `THREEPM_BACKEND_COOKIE_SECURE` | off | `Secure` session cookie (behind HTTPS). |
| `THREEPM_BACKEND_TRUST_PROXY` | none | Peers whose `X-Forwarded-For` is believed. |
| `THREEPM_BACKEND_METRICS_URL` | none | VictoriaMetrics base URL; unset = telemetry not persisted. |
| `THREEPM_BACKEND_EVENTS_URL` | none | VictoriaLogs base URL for events. |

## Internal (tests, handoff) — do not set

`THREEPM_READY_FD`, `THREEPM_INHERITED_FDS`, `THREEPM_FLEET_SECRET`,
`THREEPM_HANDOFF_*`, `THREEPM_TEST_*`, `THREEPM_E2E_*`: plumbing between a
daemon and its successor, the agent gate, or the test harness.

</document>

<document url="https://3pm.edgecr4ft.net/docs/cli.md" title="CLI reference">

# 3pm CLI reference

Generated from `3pm --help` and every subcommand's `--help` (3pm 0.64.0).
Every command accepts `--json` (machine output on stdout, errors on stderr) unless its help says otherwise.
Targets (`<name|id|pattern|all|ecosystem file>`) are described in features.md → Target patterns.

## Index

- [`3pm start`](#3pm-start): Start an app (script, binary, "command line", or ecosystem file: .toml, .json, .yaml, .config.js; `-` = stdin)
- [`3pm bench`](#3pm-bench): Start a demo fleet (4 node, 2 python, 2 shell apps) for testing; rerun to restart it, --delete to remove it
- [`3pm validate`](#3pm-validate): Check ecosystem files (syntax, fields, profiles, dependencies) without starting anything; exit 1 on any error
- [`3pm apply`](#3pm-apply): Reconcile the daemon with ecosystem files: create missing apps, restart changed ones, leave unchanged ones running (idempotent)
- [`3pm scale`](#3pm-scale): Resize an app to N instances (never restarts running ones)
- [`3pm stop`](#3pm-stop): Stop one or more processes by name, id, pattern, 'all', or every app of an ecosystem file
- [`3pm restart`](#3pm-restart): Restart one or more processes by name, id, pattern, 'all', or every app of an ecosystem file
- [`3pm rename`](#3pm-rename): Rename an app in place, without restarting it
- [`3pm delete`](#3pm-delete): Stop and remove one or more processes by name, id, pattern, 'all', or every app of an ecosystem file
- [`3pm reset`](#3pm-reset): Zero the restart counters of one or more processes (name, id, pattern, 'all', or an ecosystem file)
- [`3pm sendSignal`](#3pm-sendsignal): Send a signal to a running process (name, id, pattern or 'all'), e.g. `3pm sendSignal SIGUSR2 api`
- [`3pm ls`](#3pm-ls): List managed processes
- [`3pm show`](#3pm-show): Show detailed information about a process
- [`3pm attach`](#3pm-attach): Attach to the terminal of an app started with --tty: its screen, your keys; Ctrl-Z detaches. Piped, stdin goes to the app and its output comes out; --peek prints the screen
- [`3pm cd`](#3pm-cd): Go to an app's working directory: a shell there on a terminal (exit to return), the path otherwise (`cd "$(3pm cd api)"`)
- [`3pm trigger`](#3pm-trigger): Run an action the app declared (`io.action`) on every instance of a name, or on one; without an action, list what each source declares
- [`3pm call`](#3pm-call): Call an action with bytes attached, on this host or on a node of the bucket (`--node`): one instance, one answer, `--file` in and `--out` out
- [`3pm profile`](#3pm-profile): Profile an app's CPU or heap (its SDK's `3pm:profile` action): the file is kept under ~/.3pm/actions/ and the hot frames are printed
- [`3pm internals`](#3pm-internals): Show 3pm daemon internal metrics (memory, load, fds, supervision, its own processes)
- [`3pm hostmetrics`](#3pm-hostmetrics): Show host-wide OS metrics (CPU incl. steal, PSI pressure, memory, cgroup limits, disks, RAID, network, TCP health, sensors, NTP offset)
- [`3pm logs`](#3pm-logs): Show process logs (last N lines, then stream)
- [`3pm events`](#3pm-events): Stream the daemon event bus (lifecycle, logs, rotations, shutdown); filtered server-side, one line per event
- [`3pm runtime`](#3pm-runtime): Container entrypoint (PID 1): run the daemon in the foreground, stream app output, exit when there is nothing left to supervise
- [`3pm flush`](#3pm-flush): Empty log files in place (all apps + the daemon's, or one target)
- [`3pm reloadLogs`](#3pm-reloadlogs): Reopen every log file by path (after an external logrotate/mv); SIGUSR2 to the daemon does the same
- [`3pm logrotate`](#3pm-logrotate): Install a logrotate(8) config for the 3pm logs (root; `--print` to just show it). Built-in rotation is usually enough — this is for hosts that centralize rotation in /etc/logrotate.d
- [`3pm rotate`](#3pm-rotate): Rotate log files now, regardless of size
- [`3pm save`](#3pm-save): Save the current process list for `resurrect`
- [`3pm resurrect`](#3pm-resurrect): Restore the last saved process list
- [`3pm kill`](#3pm-kill): Stop the daemon and all managed processes
- [`3pm update`](#3pm-update): Update the daemon to this binary without restarting the apps (in-place handoff; falls back to save → kill → respawn → resurrect)
- [`3pm startup`](#3pm-startup): Configure 3pm to start on boot (prints the command to run as root)
- [`3pm unstartup`](#3pm-unstartup): Remove the 3pm boot service (prints the command to run as root)
- [`3pm install`](#3pm-install): Install a module (a bundled name or a directory holding a module.toml) and start its apps; without argument, list modules
- [`3pm uninstall`](#3pm-uninstall): Uninstall a module: delete its apps and remove its files
- [`3pm link`](#3pm-link): Link this 3pm to a fleet backend: the line pasted from its bucket page; alone, show the link and exit 0 only if it is established; `link stop` / `link start` pause and resume it, `link delete` forgets it (`3pm link --help` lists the agent's flags)
  - [`3pm link stop`](#3pm-link-stop): Stop the agent app and keep the enrollment (`link start` resumes)
  - [`3pm link start`](#3pm-link-start): Start the agent app again on an existing enrollment (registers it again if it was deleted)
  - [`3pm link status`](#3pm-link-status): Same as `3pm agent status`
  - [`3pm link delete`](#3pm-link-delete): Disconnect and forget the enrollment (key, link.toml) — same as `3pm unlink`
  - [`3pm link authorize`](#3pm-link-authorize): Let the caller whose key this is open `3pm fleet ssh` sessions here (`3pm fleet key` prints it on the caller's machine). Local: the fleet cannot add a key
  - [`3pm link deauthorize`](#3pm-link-deauthorize): Stop admitting a caller's key, named by its fingerprint or its comment
- [`3pm unlink`](#3pm-unlink): Unlink from the fleet backend and remove the agent app
- [`3pm fleet`](#3pm-fleet): The nodes of this bucket, from this host's own link: `fleet nodes` (alias `fleet ls`), `fleet apps [node]`, `fleet inspect <node> [app]`, `fleet call …`, `fleet top`, `fleet ssh <node>`, `fleet cp`; alone, print this list
- [`3pm module`](#3pm-module): Modules: optional features bundled with 3pm (`3pm module list`); alone, lists them and prints this help
  - [`3pm module list`](#3pm-module-list): List installed and bundled modules
- [`3pm completions`](#3pm-completions): Generate shell completions
- [`3pm agent`](#3pm-agent): Fleet link: enroll this host and stream its processes to a 3pm backend
  - [`3pm agent link`](#3pm-agent-link): Enroll this host and start the agent app (the pasted line); alone, show the link and exit 0 only if it is established; `link stop` / `link start` / `link status` / `link delete` act on an existing link
    - [`3pm agent link stop`](#3pm-agent-link-stop): Stop the agent app and keep the enrollment (`link start` resumes)
    - [`3pm agent link start`](#3pm-agent-link-start): Start the agent app again on an existing enrollment (registers it again if it was deleted)
    - [`3pm agent link status`](#3pm-agent-link-status): Same as `3pm agent status`
    - [`3pm agent link delete`](#3pm-agent-link-delete): Disconnect and forget the enrollment (key, link.toml) — same as `3pm unlink`
    - [`3pm agent link authorize`](#3pm-agent-link-authorize): Let the caller whose key this is open `3pm fleet ssh` sessions here (`3pm fleet key` prints it on the caller's machine). Local: the fleet cannot add a key
    - [`3pm agent link deauthorize`](#3pm-agent-link-deauthorize): Stop admitting a caller's key, named by its fingerprint or its comment
  - [`3pm agent unlink`](#3pm-agent-unlink): Stop and delete the agent app, forget the enrollment
  - [`3pm agent status`](#3pm-agent-status): What this host is linked to, and how the agent app is doing
  - [`3pm agent fleet`](#3pm-agent-fleet): Read this node's own bucket with this node's own identity: `fleet nodes`, `fleet apps [node]`, `fleet inspect <node> [app]`, `fleet top`, `fleet call`, `fleet ssh <node>`, `fleet cp`
    - [`3pm agent fleet nodes`](#3pm-agent-fleet-nodes): Every node of the bucket: host, os, state, how long since it was seen, how many apps it runs, whose account runs it and where it is reached
    - [`3pm agent fleet apps`](#3pm-agent-fleet-apps): Every process of the bucket, or of one node (a `3pmn_…` id, a hostname, or an unambiguous start of one)
    - [`3pm agent fleet inspect`](#3pm-agent-fleet-inspect): One node in full — and one of its apps, when named
    - [`3pm agent fleet top`](#3pm-agent-fleet-top): The bucket live, in two panels, until `q`: nodes on top, processes below, redrawn once a second from the event stream
    - [`3pm agent fleet call`](#3pm-agent-fleet-call): Place a call on another node of the bucket, with this node's own identity, and wait for it: `--file` sends bytes, `--out` keeps what came back
    - [`3pm agent fleet ssh`](#3pm-agent-fleet-ssh): An interactive login shell on another node of the bucket, like ssh (`~.` after Enter to leave) — or one command, after `--`
    - [`3pm agent fleet attach`](#3pm-agent-fleet-attach): The terminal of a `tty = true` app on another node of the bucket, like `3pm attach` (`~.` after Enter to detach: the app runs on)
    - [`3pm agent fleet cp`](#3pm-agent-fleet-cp): Copy files to or from another node of the bucket, like scp: one side is `<node>:<path>`
    - [`3pm agent fleet known-nodes`](#3pm-agent-fleet-known-nodes): The shell keys of the nodes this host has met, pinned at the first session (`~/.3pm/known_nodes`, like ssh's `known_hosts`)
      - [`3pm agent fleet known-nodes ls`](#3pm-agent-fleet-known-nodes-ls): The pinned nodes: id, fingerprint, name, date
      - [`3pm agent fleet known-nodes rm`](#3pm-agent-fleet-known-nodes-rm): Forget a node's key (its `3pmn_…` id, or its name as pinned), like `ssh-keygen -R`: the next session asks again
    - [`3pm agent fleet key`](#3pm-agent-fleet-key): This host's caller key (`cli.key`, or THREEPM_CLI_KEY): its fingerprint and the line a node needs to admit it; `key new`, `key push`, `key revoke`
      - [`3pm agent fleet key new`](#3pm-agent-fleet-key-new): A new caller key in a file of its own, offline (a rescue key: keep it off this host; use it with THREEPM_CLI_KEY=<file>)
      - [`3pm agent fleet key push`](#3pm-agent-fleet-key-push): Admit a key on nodes, through a `fleet ssh` session on each (this host's key must already be admitted there)
      - [`3pm agent fleet key revoke`](#3pm-agent-fleet-key-revoke): Stop admitting a key on nodes, the same way
- [`3pm serve`](#3pm-serve): Static HTTP file server (pm2 serve): --spa, --ftp, basic auth

## 3pm

```text
3pm — a fast, robust process manager

Usage: 3pm [OPTIONS] <COMMAND>

Commands:
  start        Start an app (script, binary, "command line", or ecosystem file: .toml, .json, .yaml, .config.js; `-` = stdin)
  bench        Start a demo fleet (4 node, 2 python, 2 shell apps) for testing; rerun to restart it, --delete to remove it
  validate     Check ecosystem files (syntax, fields, profiles, dependencies) without starting anything; exit 1 on any error
  apply        Reconcile the daemon with ecosystem files: create missing apps, restart changed ones, leave unchanged ones running (idempotent)
  scale        Resize an app to N instances (never restarts running ones)
  stop         Stop one or more processes by name, id, pattern, 'all', or every app of an ecosystem file
  restart      Restart one or more processes by name, id, pattern, 'all', or every app of an ecosystem file
  rename       Rename an app in place, without restarting it
  delete       Stop and remove one or more processes by name, id, pattern, 'all', or every app of an ecosystem file
  reset        Zero the restart counters of one or more processes (name, id, pattern, 'all', or an ecosystem file)
  sendSignal   Send a signal to a running process (name, id, pattern or 'all'), e.g. `3pm sendSignal SIGUSR2 api`
  ls           List managed processes
  show         Show detailed information about a process
  attach       Attach to the terminal of an app started with --tty: its screen, your keys; Ctrl-Z detaches. Piped, stdin goes to the app and its output comes out; --peek prints the screen
  cd           Go to an app's working directory: a shell there on a terminal (exit to return), the path otherwise (`cd "$(3pm cd api)"`)
  trigger      Run an action the app declared (`io.action`) on every instance of a name, or on one; without an action, list what each source declares
  call         Call an action with bytes attached, on this host or on a node of the bucket (`--node`): one instance, one answer, `--file` in and `--out` out
  profile      Profile an app's CPU or heap (its SDK's `3pm:profile` action): the file is kept under ~/.3pm/actions/ and the hot frames are printed
  internals    Show 3pm daemon internal metrics (memory, load, fds, supervision, its own processes)
  hostmetrics  Show host-wide OS metrics (CPU incl. steal, PSI pressure, memory, cgroup limits, disks, RAID, network, TCP health, sensors, NTP offset)
  logs         Show process logs (last N lines, then stream)
  events       Stream the daemon event bus (lifecycle, logs, rotations, shutdown); filtered server-side, one line per event
  runtime      Container entrypoint (PID 1): run the daemon in the foreground, stream app output, exit when there is nothing left to supervise
  flush        Empty log files in place (all apps + the daemon's, or one target)
  reloadLogs   Reopen every log file by path (after an external logrotate/mv); SIGUSR2 to the daemon does the same
  logrotate    Install a logrotate(8) config for the 3pm logs (root; `--print` to just show it). Built-in rotation is usually enough — this is for hosts that centralize rotation in /etc/logrotate.d
  rotate       Rotate log files now, regardless of size
  save         Save the current process list for `resurrect`
  resurrect    Restore the last saved process list
  kill         Stop the daemon and all managed processes
  update       Update the daemon to this binary without restarting the apps (in-place handoff; falls back to save → kill → respawn → resurrect)
  startup      Configure 3pm to start on boot (prints the command to run as root)
  unstartup    Remove the 3pm boot service (prints the command to run as root)
  install      Install a module (a bundled name or a directory holding a module.toml) and start its apps; without argument, list modules
  uninstall    Uninstall a module: delete its apps and remove its files
  link         Link this 3pm to a fleet backend: the line pasted from its bucket page; alone, show the link and exit 0 only if it is established; `link stop` / `link start` pause and resume it, `link delete` forgets it (`3pm link --help` lists the agent's flags)
  unlink       Unlink from the fleet backend and remove the agent app
  fleet        The nodes of this bucket, from this host's own link: `fleet nodes` (alias `fleet ls`), `fleet apps [node]`, `fleet inspect <node> [app]`, `fleet call …`, `fleet top`, `fleet ssh <node>`, `fleet cp`; alone, print this list
  module       Modules: optional features bundled with 3pm (`3pm module list`); alone, lists them and prints this help
  completions  Generate shell completions
  help         Print this message or the help of the given subcommand(s)

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm start

```text
Start an app (script, binary, "command line", or ecosystem file: .toml, .json, .yaml, .config.js; `-` = stdin)

Usage: 3pm start [OPTIONS] [TARGET] [-- <ARGS>...]

Arguments:
  [TARGET]   Script/binary to run, a quoted command line ("npm run dev", run by bash -c), an ecosystem file (.toml, .json, .yaml, .config.js), or `-` for a config on stdin. Default: the ecosystem file of the current directory
  [ARGS]...  Arguments passed to the app (everything after --)

Options:
      --attach                          After starting, stream the logs of the started apps (Ctrl-C to leave)
      --json                            Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --only <APPS>                     With an ecosystem file: start only these apps (comma-separated)
      --name <NAME>                     Process name (defaults to the script's file stem)
      --interpreter <INTERPRETER>       Interpreter to run the script with (default: inferred from the extension); `none` runs the script itself
      --interpreter-args <ARGS>         The interpreter's own flags, before the script (pm2's --node-args), split on blanks: --interpreter-args "--inspect --max-old-space-size=64"; "" clears them [alias: --node-args]
      --cwd <CWD>                       Working directory (default: current directory)
      --env <KEY=VALUE|PROFILE>         Environment variables, KEY=VALUE (repeatable). With an ecosystem file: the profile to apply (`--env production` selects [apps.profiles.production] and [apps.env_production])
      --namespace <NAMESPACE>           Namespace grouping label
      --user <NAME>                     Unix account to run the app as (root daemon, user declared in `[users] allow`); "" runs it as the daemon again
      --persist <BOOL>                  Keep the app in dump.json (`3pm save`, `3pm update`); false makes it ephemeral: it survives restarts, never a reboot [possible values: true, false]
  -i, --instances <N>                   Number of instances to run (cluster mode); 0 = one per CPU
      --depends-on <APP>                App that must be online before this one starts (repeatable)
      --start-delay <MS>                Wait this long before the first spawn (ms); crash restarts use --restart-delay / --exp-backoff-restart-delay instead
      --restart-delay <MS>              Fixed pause before every crash restart (ms; default 0: restart at once, like pm2)
      --exp-backoff-restart-delay <MS>  Exponential backoff between crash restarts, starting at this delay (ms) and growing ×1.5 up to 15s; off by default (pm2)
      --stop-exit-codes <CODES>         Exit codes that stop the app instead of restarting it (e.g. 0,3)
      --cron-restart <CRON>             Restart on a cron schedule, e.g. "0 2 * * *" (5 fields, or 6 with seconds first)
      --no-autorestart                  Do not restart the app on exit (same as --restart-policy never)
      --restart-policy <POLICY>         When an unrequested exit respawns the app: always (default), on-failure (non-zero exit or signal only), never [possible values: always, on-failure, never]
      --wait-ready                      Hold `launching` until the app signals ready on the control socket
      --shutdown-with-message           Graceful stop via a control-socket message instead of a signal
      --max-memory-restart <SIZE>       Restart when memory exceeds this amount (e.g. 200M, 1G)
      --max-cpu-percent <PERCENT>       Restart when CPU usage exceeds this percent of one core (may be above 100 on multi-core hosts)
      --cgroup-enforce                  Linux only: confine the app in a cgroup v2 whose cpu.max / memory.max are hard limits derived from --max-cpu-percent / --max-memory-restart
      --deny-gpu                        Hide GPUs from the app (CUDA / ROCm visibility env vars)
      --ulimit-rtprio <PRIO>            Linux only: the app's RLIMIT_RTPRIO (0-99), so its threads may run SCHED_FIFO / SCHED_RR up to that priority
      --ulimit-memlock <SIZE>           The app's RLIMIT_MEMLOCK: how much it may lock in RAM (e.g. 64M, 4G, unlimited)
      --ulimit-nice <N>                 Linux only: the app's RLIMIT_NICE (0-40), so its threads may lower their nice down to 20 minus this (28 for -8)
      --ulimit-rttime <US>              Linux only: the app's RLIMIT_RTTIME, in microseconds: the CPU a real-time thread may take without blocking before the kernel kills it
      --nice <N>                        The app's nice value, -20 (most CPU) to 19
      --ioprio <CLASS>                  Linux only: the app's I/O class: idle, best-effort[:0-7] or realtime[:0-7]
      --oom-score-adj <N>               Linux only: the app's oom_score_adj, -1000 (never killed when memory runs out) to 1000 (killed first)
      --cpu-affinity <CPUS>             Linux only: the CPUs the app may run on (e.g. 0-3,6)
      --kill-signal <KILL_SIGNAL>       Signal sent on stop before SIGKILL escalation (default: SIGINT)
      --watch[=<PATH>]                  Restart on file changes: `--watch` for the working directory, or `--watch=PATH` (repeatable) for specific dirs/files
      --ignore-watch <GLOB>             Glob to exclude from watching, added to the built-in ones (dotfiles, node_modules); repeatable
      --watch-delay <MS>                Debounce before a watch-triggered restart, in milliseconds
  -o, --output <PATH>                   stdout log file (default: ~/.3pm/logs/<name>-out.log); NULL to disable the file. May contain {name}, {instance}, {namespace}
  -e, --error <PATH>                    stderr log file (default: ~/.3pm/logs/<name>-error.log); NULL to disable the file
  -l, --log <PATH>                      One file for both streams (implies --merge-logs)
      --merge-logs                      Write stdout and stderr to a single file
      --log-date-format <FMT>           Prefix each persisted log line with a strftime timestamp
      --time                            Shorthand for --log-date-format '%Y-%m-%dT%H:%M:%S' (pm2 --time)
      --disable-logs                    No log files, no log stream, no crash tail
      --tty                             Run on a terminal (pty) instead of pipes, for interactive programs; one output stream, logged without escape sequences
  -h, --help                            Print help
  -V, --version                         Print version
```

## 3pm bench

```text
Start a demo fleet (4 node, 2 python, 2 shell apps) for testing; rerun to restart it, --delete to remove it

Usage: 3pm bench [OPTIONS]

Options:
      --dir <DIR>  Where the demo scripts are written (default: <home>/bench)
      --json       Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --delete     Delete the bench apps instead of starting them
  -h, --help       Print help
  -V, --version    Print version
```

## 3pm validate

```text
Check ecosystem files (syntax, fields, profiles, dependencies) without starting anything; exit 1 on any error

Usage: 3pm validate [OPTIONS] [FILES]...

Arguments:
  [FILES]...  Ecosystem file(s) (.toml, .json, .yaml, .config.js), `-` for stdin. Default: the ecosystem file of the current directory

Options:
      --env <PROFILE>  Profile to resolve (as `start --env <PROFILE>` would)
      --json           Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --only <APPS>    Only these apps (comma-separated)
      --check-managed  Ask the daemon whether external `depends_on` names are managed
  -h, --help           Print help
  -V, --version        Print version
```

## 3pm apply

```text
Reconcile the daemon with ecosystem files: create missing apps, restart changed ones, leave unchanged ones running (idempotent)

Usage: 3pm apply [OPTIONS] [FILES]...

Arguments:
  [FILES]...  Ecosystem file(s) (.toml, .json, .yaml, .config.js). Default: the ecosystem file of the current directory

Options:
      --env <PROFILE>  Profile to apply (as `start --env <PROFILE>` would)
      --json           Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --only <APPS>    Only these apps (comma-separated)
      --prune          Delete managed apps that came from these files but are gone from them
      --prune-all      Delete every managed app absent from these files (CLI-started too)
      --dry-run        Print the plan and exit without changing anything
  -h, --help           Print help
  -V, --version        Print version
```

## 3pm scale

```text
Resize an app to N instances (never restarts running ones)

Usage: 3pm scale [OPTIONS] <NAME> <INSTANCES>

Arguments:
  <NAME>       App name (all its instances share it)
  <INSTANCES>  Desired instance count (1 or more)

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm stop

```text
Stop one or more processes by name, id, pattern, 'all', or every app of an ecosystem file

Usage: 3pm stop [OPTIONS] <TARGET>...

Arguments:
  <TARGET>...  Processes to act on: names, ids, 'victoria*' patterns, 'all', or ecosystem files

Options:
      --attach   After stopping, stream the logs of the affected apps (Ctrl-C to leave)
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm restart

```text
Restart one or more processes by name, id, pattern, 'all', or every app of an ecosystem file

Every `start` flag is accepted and reconfigures the apps on the way (pm2's attribute update): what you type is folded into each app's configuration, which then restarts on it — `3pm restart api --name web -i 4` renames the app in place, keeping its ids and counters, and runs it with four instances. `--watch` alone still toggles watch mode; `--watch=PATH` sets the watched paths.

Usage: 3pm restart [OPTIONS] <TARGET>...

Arguments:
  <TARGET>...
          Processes to act on: names, ids, 'victoria*' patterns, 'all', or ecosystem files

Options:
      --attach
          After restarting, stream the logs of the affected apps (Ctrl-C to leave)

      --json
          Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)

      --update-env
          Re-read the app's ecosystem file (same profile) and restart with the fresh spec
          
          The new values come from the file, so no configuration flag may be typed alongside it.

      --name <NAME>
          Process name (defaults to the script's file stem)

      --interpreter <INTERPRETER>
          Interpreter to run the script with (default: inferred from the extension); `none` runs the script itself

      --interpreter-args <ARGS>
          The interpreter's own flags, before the script (pm2's --node-args), split on blanks: --interpreter-args "--inspect --max-old-space-size=64"; "" clears them
          
          [alias: --node-args]

      --cwd <CWD>
          Working directory (default: current directory)

      --env <KEY=VALUE|PROFILE>
          Environment variables, KEY=VALUE (repeatable). With an ecosystem file: the profile to apply (`--env production` selects [apps.profiles.production] and [apps.env_production])

      --namespace <NAMESPACE>
          Namespace grouping label

      --user <NAME>
          Unix account to run the app as (root daemon, user declared in `[users] allow`); "" runs it as the daemon again

      --persist <BOOL>
          Keep the app in dump.json (`3pm save`, `3pm update`); false makes it ephemeral: it survives restarts, never a reboot
          
          [possible values: true, false]

  -i, --instances <N>
          Number of instances to run (cluster mode); 0 = one per CPU

      --depends-on <APP>
          App that must be online before this one starts (repeatable)

      --start-delay <MS>
          Wait this long before the first spawn (ms); crash restarts use --restart-delay / --exp-backoff-restart-delay instead

      --restart-delay <MS>
          Fixed pause before every crash restart (ms; default 0: restart at once, like pm2)

      --exp-backoff-restart-delay <MS>
          Exponential backoff between crash restarts, starting at this delay (ms) and growing ×1.5 up to 15s; off by default (pm2)

      --stop-exit-codes <CODES>
          Exit codes that stop the app instead of restarting it (e.g. 0,3)

      --cron-restart <CRON>
          Restart on a cron schedule, e.g. "0 2 * * *" (5 fields, or 6 with seconds first)

      --no-autorestart
          Do not restart the app on exit (same as --restart-policy never)

      --restart-policy <POLICY>
          When an unrequested exit respawns the app: always (default), on-failure (non-zero exit or signal only), never
          
          [possible values: always, on-failure, never]

      --wait-ready
          Hold `launching` until the app signals ready on the control socket

      --shutdown-with-message
          Graceful stop via a control-socket message instead of a signal

      --max-memory-restart <SIZE>
          Restart when memory exceeds this amount (e.g. 200M, 1G)

      --max-cpu-percent <PERCENT>
          Restart when CPU usage exceeds this percent of one core (may be above 100 on multi-core hosts)

      --cgroup-enforce
          Linux only: confine the app in a cgroup v2 whose cpu.max / memory.max are hard limits derived from --max-cpu-percent / --max-memory-restart

      --deny-gpu
          Hide GPUs from the app (CUDA / ROCm visibility env vars)

      --ulimit-rtprio <PRIO>
          Linux only: the app's RLIMIT_RTPRIO (0-99), so its threads may run SCHED_FIFO / SCHED_RR up to that priority

      --ulimit-memlock <SIZE>
          The app's RLIMIT_MEMLOCK: how much it may lock in RAM (e.g. 64M, 4G, unlimited)

      --ulimit-nice <N>
          Linux only: the app's RLIMIT_NICE (0-40), so its threads may lower their nice down to 20 minus this (28 for -8)

      --ulimit-rttime <US>
          Linux only: the app's RLIMIT_RTTIME, in microseconds: the CPU a real-time thread may take without blocking before the kernel kills it

      --nice <N>
          The app's nice value, -20 (most CPU) to 19

      --ioprio <CLASS>
          Linux only: the app's I/O class: idle, best-effort[:0-7] or realtime[:0-7]

      --oom-score-adj <N>
          Linux only: the app's oom_score_adj, -1000 (never killed when memory runs out) to 1000 (killed first)

      --cpu-affinity <CPUS>
          Linux only: the CPUs the app may run on (e.g. 0-3,6)

      --kill-signal <KILL_SIGNAL>
          Signal sent on stop before SIGKILL escalation (default: SIGINT)

      --watch[=<PATH>]
          Restart on file changes: `--watch` for the working directory, or `--watch=PATH` (repeatable) for specific dirs/files

      --ignore-watch <GLOB>
          Glob to exclude from watching, added to the built-in ones (dotfiles, node_modules); repeatable

      --watch-delay <MS>
          Debounce before a watch-triggered restart, in milliseconds

  -o, --output <PATH>
          stdout log file (default: ~/.3pm/logs/<name>-out.log); NULL to disable the file. May contain {name}, {instance}, {namespace}

  -e, --error <PATH>
          stderr log file (default: ~/.3pm/logs/<name>-error.log); NULL to disable the file

  -l, --log <PATH>
          One file for both streams (implies --merge-logs)

      --merge-logs
          Write stdout and stderr to a single file

      --log-date-format <FMT>
          Prefix each persisted log line with a strftime timestamp

      --time
          Shorthand for --log-date-format '%Y-%m-%dT%H:%M:%S' (pm2 --time)

      --disable-logs
          No log files, no log stream, no crash tail

      --tty
          Run on a terminal (pty) instead of pipes, for interactive programs; one output stream, logged without escape sequences

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version
```

## 3pm rename

```text
Rename an app in place, without restarting it

Same ids, pids and restart counters; the pid files and the log files named after the app (live and rotated) move to the new name, and the app writes on into them. The running process keeps the `name` variable it was started with until its next restart. `3pm restart <app> --name <new>` renames and restarts.

Usage: 3pm rename [OPTIONS] <APP> <NEW_NAME>

Arguments:
  <APP>
          The app to rename: its name, or the id of one of its instances

  <NEW_NAME>
          Its new name

Options:
      --json
          Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version
```

## 3pm delete

```text
Stop and remove one or more processes by name, id, pattern, 'all', or every app of an ecosystem file

Usage: 3pm delete [OPTIONS] <TARGET>...

Arguments:
  <TARGET>...  Processes to act on: names, ids, 'victoria*' patterns, 'all', or ecosystem files

Options:
      --flush    Also delete their log files (live and rotated)
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm reset

```text
Zero the restart counters of one or more processes (name, id, pattern, 'all', or an ecosystem file)

Usage: 3pm reset [OPTIONS] <TARGET>...

Arguments:
  <TARGET>...  Processes to act on: names, ids, 'victoria*' patterns, 'all', or ecosystem files

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm sendSignal

```text
Send a signal to a running process (name, id, pattern or 'all'), e.g. `3pm sendSignal SIGUSR2 api`

Usage: 3pm sendSignal [OPTIONS] <SIGNAL> <TARGET>

Arguments:
  <SIGNAL>  POSIX signal name (SIGUSR1, SIGHUP, …)
  <TARGET>  

Options:
      --group    Signal the whole process group instead of the process alone
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm ls

```text
List managed processes

Usage: 3pm ls [OPTIONS]

Options:
      --json           Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --watch          Redraw the table in place every --interval ms until Ctrl-C (pm2's `ls --watch`); with --json, one document per tick
      --interval <MS>  Refresh period of --watch in milliseconds (default 300, or 1000 with --tree: the daemon samples trees once a second; min 100)
  -t, --tree           Draw each app's process tree under its row: chains of single children folded (`npm › node`), twins grouped (`worker ×4`), `⚠` on a process that left the group (`3pm stop` will not reach it); with --json, a `tree` object per app
      --all            With --tree: one row per process, nothing folded or grouped
  -h, --help           Print help
  -V, --version        Print version
```

## 3pm show

```text
Show detailed information about a process

Usage: 3pm show [OPTIONS] <TARGET>

Arguments:
  <TARGET>  

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm attach

```text
Attach to the terminal of an app started with --tty: its screen, your keys; Ctrl-Z detaches. Piped, stdin goes to the app and its output comes out; --peek prints the screen

Usage: 3pm attach [OPTIONS] <TARGET>

Arguments:
  <TARGET>  App name, or id

Options:
      --instance <N>  Which instance of the app (default 0)
      --json          Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --read-only     Watch only: neither your keys nor your terminal's size reach the app
      --raw           The app's bytes as it writes them instead of the screen the daemon renders: everything the app draws, at your terminal's risk
      --peek          Print the screen as text and exit (--json: a JSON document)
      --paste         Send stdin as one bracketed paste when the app turned pasting on (a prompt then takes several lines as one message)
  -h, --help          Print help
  -V, --version       Print version
```

## 3pm cd

```text
Go to an app's working directory: a shell there on a terminal (exit to return), the path otherwise (`cd "$(3pm cd api)"`)

Usage: 3pm cd [OPTIONS] <TARGET>

Arguments:
  <TARGET>  Process name or id

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm trigger

```text
Run an action the app declared (`io.action`) on every instance of a name, or on one; without an action, list what each source declares

Usage: 3pm trigger [OPTIONS] <TARGET> [ACTION] [PARAMS]

Arguments:
  <TARGET>  Process name or id (`all` is refused)
  [ACTION]  Action name, as declared; omitted: list the actions of each source
  [PARAMS]  Parameters, as JSON (default null)

Options:
      --instance <N>        One instance of the app (default: every instance that declares the action, at most 16)
      --json                Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --src <SOURCE>        The source (process of the app) to call, when several declare it; without an action, list that source's actions only
      --timeout <DURATION>  How long to wait (`30s`, `2m`, `1500ms`); the declaration's own timeout bounds it (default 30s, ceiling 10m)
      --file <PATH>         A file to send with the call: the action must declare an `input` that accepts its size and extension. Nothing is copied — the handler is given this very path — and the call then reaches one instance (`--instance`, 0 by default) instead of every one
      --out <DIR>           Directory to put the file the action produced into (it stays under the home's `actions/` too, until retention sweeps it)
  -h, --help                Print help
  -V, --version             Print version
```

## 3pm call

```text
Call an action with bytes attached, on this host or on a node of the bucket (`--node`): one instance, one answer, `--file` in and `--out` out

Usage: 3pm call [OPTIONS] <APP> <ACTION> [PARAMS]

Arguments:
  <APP>     App name (never `all`: a call names one app)
  <ACTION>  Action name, as declared
  [PARAMS]  Parameters, as JSON (default null)

Options:
      --json                Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --node <NODE>         A node of the bucket (`3pmn_…` id, or a hostname unique in it); omitted, the call runs on this host. A call that names a node leaves through the fleet agent and carries no file (`3pm fleet call` does that)
      --instance <N>        One instance of the app (default 0)
      --src <SOURCE>        The source (process of the app) to call, when several declare it
      --timeout <DURATION>  How long to wait (`30s`, `2m`, `1500ms`); the declaration's own timeout bounds it (default 30s, ceiling 10m)
      --file <PATH>         A file to send with the call (an absolute or relative path of this host; the action must declare an `input`)
      --out <DIR>           Directory to put the file the action produced into
  -h, --help                Print help
  -V, --version             Print version
```

## 3pm profile

```text
Profile an app's CPU or heap (its SDK's `3pm:profile` action): the file is kept under ~/.3pm/actions/ and the hot frames are printed

Usage: 3pm profile [OPTIONS] <TARGET>

Arguments:
  <TARGET>  Process name or id

Options:
      --instance <N>         One instance of the app (default: every instance)
      --json                 Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --src <SOURCE>         The source (process of the app) to profile; required when several declare `3pm:profile` (an Electron app: the main and each renderer)
      --heap                 A heap sampling profile instead of a CPU one
      --duration <DURATION>  How long to sample (`10s`, `1500ms`; 1s to 60s) [default: 10s]
      --sampling <N>         Sampling interval: microseconds for CPU (default 1000), bytes for heap (default 32768)
      --top <N>              Hot frames to print [default: 15]
  -h, --help                 Print help
  -V, --version              Print version
```

## 3pm internals

```text
Show 3pm daemon internal metrics (memory, load, fds, supervision, its own processes)

Usage: 3pm internals [OPTIONS]

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm hostmetrics

```text
Show host-wide OS metrics (CPU incl. steal, PSI pressure, memory, cgroup limits, disks, RAID, network, TCP health, sensors, NTP offset)

Usage: 3pm hostmetrics [OPTIONS]

Options:
      --extended  Extended Monitoring instead of the overview: every kernel counter family (`[hostmetrics] extended`), one line per field, then the sources that could not be read and why
      --json      Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help      Print help
  -V, --version   Print version
```

## 3pm logs

```text
Show process logs (last N lines, then stream)

Usage: 3pm logs [OPTIONS] [TARGET]...

Arguments:
  [TARGET]...  Process names, ids or 'victoria*' patterns (default: all)

Options:
      --json               Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --lines <LINES>      Number of history lines to print per log file [default: 30]
      --err                Only error output
      --out                Only standard output
      --raw                No prefixes or colors
      --nostream           Print the tail and exit without streaming (`--no-stream` too)
      --timestamp[=<FMT>]  Prefix streamed lines with their arrival time (strftime; default "%Y-%m-%d %H:%M:%S"). History lines have no time and get padding
      --format             logfmt output: ts= app= id= instance= type= message=
      --highlight <TEXT>   Highlight this text in streamed and history lines
      --rotated            Continue the history into rotated generations (.1, .2, .gz) when the live file has fewer than --lines
      --size               Print log file sizes and rotated generations, then exit
  -h, --help               Print help
  -V, --version            Print version
```

## 3pm events

```text
Stream the daemon event bus (lifecycle, logs, rotations, shutdown); filtered server-side, one line per event

Usage: 3pm events [OPTIONS]

Options:
  -f, --filter <GLOB>    Event glob to subscribe to (repeatable): `process`, `process:exit`, `log`, `log:err`, `log*`, `log_rotated`, `app`, `lagged`, `daemon_shutdown` (`log_out`/`log_err`/`log_*` still accepted for one release). Default: all
      --json             Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -p, --process <NAME>   Only events about this app, exact or a 'web-*' pattern (daemon-level events still pass)
      --raw              No time column, prefixes or colors: `<event> <detail>` only
      --timestamp <FMT>  Time column format (strftime; default "%H:%M:%S"). Log events use their daemon receive time, other events the arrival time [default: %H:%M:%S]
  -h, --help             Print help
  -V, --version          Print version
```

## 3pm runtime

```text
Container entrypoint (PID 1): run the daemon in the foreground, stream app output, exit when there is nothing left to supervise

Usage: 3pm runtime [OPTIONS] [TARGET] [-- <ARGS>...]

Arguments:
  [TARGET]   Script/binary to run, a quoted command line ("npm run dev", run by bash -c), an ecosystem file (.toml, .json, .yaml, .config.js), or `-` for a config on stdin. Default: the ecosystem file of the current directory
  [ARGS]...  Arguments passed to the app (everything after --)

Options:
      --attach                          After starting, stream the logs of the started apps (Ctrl-C to leave)
      --json                            Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --only <APPS>                     With an ecosystem file: start only these apps (comma-separated)
      --name <NAME>                     Process name (defaults to the script's file stem)
      --interpreter <INTERPRETER>       Interpreter to run the script with (default: inferred from the extension); `none` runs the script itself
      --interpreter-args <ARGS>         The interpreter's own flags, before the script (pm2's --node-args), split on blanks: --interpreter-args "--inspect --max-old-space-size=64"; "" clears them [alias: --node-args]
      --cwd <CWD>                       Working directory (default: current directory)
      --env <KEY=VALUE|PROFILE>         Environment variables, KEY=VALUE (repeatable). With an ecosystem file: the profile to apply (`--env production` selects [apps.profiles.production] and [apps.env_production])
      --namespace <NAMESPACE>           Namespace grouping label
      --user <NAME>                     Unix account to run the app as (root daemon, user declared in `[users] allow`); "" runs it as the daemon again
      --persist <BOOL>                  Keep the app in dump.json (`3pm save`, `3pm update`); false makes it ephemeral: it survives restarts, never a reboot [possible values: true, false]
  -i, --instances <N>                   Number of instances to run (cluster mode); 0 = one per CPU
      --depends-on <APP>                App that must be online before this one starts (repeatable)
      --start-delay <MS>                Wait this long before the first spawn (ms); crash restarts use --restart-delay / --exp-backoff-restart-delay instead
      --restart-delay <MS>              Fixed pause before every crash restart (ms; default 0: restart at once, like pm2)
      --exp-backoff-restart-delay <MS>  Exponential backoff between crash restarts, starting at this delay (ms) and growing ×1.5 up to 15s; off by default (pm2)
      --stop-exit-codes <CODES>         Exit codes that stop the app instead of restarting it (e.g. 0,3)
      --cron-restart <CRON>             Restart on a cron schedule, e.g. "0 2 * * *" (5 fields, or 6 with seconds first)
      --no-autorestart                  Do not restart the app on exit (same as --restart-policy never)
      --restart-policy <POLICY>         When an unrequested exit respawns the app: always (default), on-failure (non-zero exit or signal only), never [possible values: always, on-failure, never]
      --wait-ready                      Hold `launching` until the app signals ready on the control socket
      --shutdown-with-message           Graceful stop via a control-socket message instead of a signal
      --max-memory-restart <SIZE>       Restart when memory exceeds this amount (e.g. 200M, 1G)
      --max-cpu-percent <PERCENT>       Restart when CPU usage exceeds this percent of one core (may be above 100 on multi-core hosts)
      --cgroup-enforce                  Linux only: confine the app in a cgroup v2 whose cpu.max / memory.max are hard limits derived from --max-cpu-percent / --max-memory-restart
      --deny-gpu                        Hide GPUs from the app (CUDA / ROCm visibility env vars)
      --ulimit-rtprio <PRIO>            Linux only: the app's RLIMIT_RTPRIO (0-99), so its threads may run SCHED_FIFO / SCHED_RR up to that priority
      --ulimit-memlock <SIZE>           The app's RLIMIT_MEMLOCK: how much it may lock in RAM (e.g. 64M, 4G, unlimited)
      --ulimit-nice <N>                 Linux only: the app's RLIMIT_NICE (0-40), so its threads may lower their nice down to 20 minus this (28 for -8)
      --ulimit-rttime <US>              Linux only: the app's RLIMIT_RTTIME, in microseconds: the CPU a real-time thread may take without blocking before the kernel kills it
      --nice <N>                        The app's nice value, -20 (most CPU) to 19
      --ioprio <CLASS>                  Linux only: the app's I/O class: idle, best-effort[:0-7] or realtime[:0-7]
      --oom-score-adj <N>               Linux only: the app's oom_score_adj, -1000 (never killed when memory runs out) to 1000 (killed first)
      --cpu-affinity <CPUS>             Linux only: the CPUs the app may run on (e.g. 0-3,6)
      --kill-signal <KILL_SIGNAL>       Signal sent on stop before SIGKILL escalation (default: SIGINT)
      --watch[=<PATH>]                  Restart on file changes: `--watch` for the working directory, or `--watch=PATH` (repeatable) for specific dirs/files
      --ignore-watch <GLOB>             Glob to exclude from watching, added to the built-in ones (dotfiles, node_modules); repeatable
      --watch-delay <MS>                Debounce before a watch-triggered restart, in milliseconds
  -o, --output <PATH>                   stdout log file (default: ~/.3pm/logs/<name>-out.log); NULL to disable the file. May contain {name}, {instance}, {namespace}
  -e, --error <PATH>                    stderr log file (default: ~/.3pm/logs/<name>-error.log); NULL to disable the file
  -l, --log <PATH>                      One file for both streams (implies --merge-logs)
      --merge-logs                      Write stdout and stderr to a single file
      --log-date-format <FMT>           Prefix each persisted log line with a strftime timestamp
      --time                            Shorthand for --log-date-format '%Y-%m-%dT%H:%M:%S' (pm2 --time)
      --disable-logs                    No log files, no log stream, no crash tail
      --tty                             Run on a terminal (pty) instead of pipes, for interactive programs; one output stream, logged without escape sequences
      --no-auto-exit                    Keep running after every app is stopped or errored (default: exit 2)
      --delay <SECS>                    Wait this many seconds before starting the apps [default: 0]
      --log-files                       Also write per-app log files (default: output goes to the container's stdout/stderr only, unless -o/-e/-l are given)
      --raw                             Bare lines, no `id|name |` prefix (pm2-runtime's default)
      --format                          logfmt output: ts= app= id= instance= type= message=
      --timestamp[=<FMT>]               Prefix streamed lines with their arrival time (strftime; default "%Y-%m-%d %H:%M:%S")
  -h, --help                            Print help
  -V, --version                         Print version
```

## 3pm flush

```text
Empty log files in place (all apps + the daemon's, or one target)

Usage: 3pm flush [OPTIONS] [TARGET]

Arguments:
  [TARGET]  Process name, id, namespace or pattern (default: everything)

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm reloadLogs

```text
Reopen every log file by path (after an external logrotate/mv); SIGUSR2 to the daemon does the same

Usage: 3pm reloadLogs [OPTIONS]

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm logrotate

```text
Install a logrotate(8) config for the 3pm logs (root; `--print` to just show it). Built-in rotation is usually enough — this is for hosts that centralize rotation in /etc/logrotate.d

Usage: 3pm logrotate [OPTIONS]

Options:
      --json         Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --print        Print the logrotate(8) config instead of installing it
      --user <USER>  User owning the 3pm home (default: $SUDO_USER, then the caller)
  -h, --help         Print help
  -V, --version      Print version
```

## 3pm rotate

```text
Rotate log files now, regardless of size

Usage: 3pm rotate [OPTIONS] [TARGET]

Arguments:
  [TARGET]  Process name, id, namespace or pattern (default: all)

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm save

```text
Save the current process list for `resurrect`

Usage: 3pm save [OPTIONS]

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm resurrect

```text
Restore the last saved process list

Usage: 3pm resurrect [OPTIONS]

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm kill

```text
Stop the daemon and all managed processes

Usage: 3pm kill [OPTIONS]

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm update

```text
Update the daemon to this binary without restarting the apps (in-place handoff; falls back to save → kill → respawn → resurrect)

The apps keep running; 3pm's own processes are restarted every time: the host agent, and the module apps (the fleet agent, `3pm serve` sites) unless `--skip-modules`.

Usage: 3pm update [OPTIONS]

Options:
      --cold
          Cold restart (save → kill → respawn → resurrect): every app restarts

      --json
          Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)

      --no-fallback
          When the in-place handoff is refused, fail (exit 3) instead of falling back to a cold restart; the apps are left untouched

      --skip-modules
          Leave the modules alone: no reinstall of the installed copies whose bundled version changed, no restart of module apps (the fleet link stays up, `3pm serve` sites keep serving)

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version
```

## 3pm startup

```text
Configure 3pm to start on boot (prints the command to run as root)

Usage: 3pm startup [OPTIONS] [PLATFORM]

Arguments:
  [PLATFORM]  Init system (launchd, systemd, openrc, rcd); auto-detected if omitted

Options:
      --json                         Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --level <LEVEL>                Service level: `system` (root installs, daemon runs as --user) or `user` (installed in your session without root: a LaunchAgent in the GUI session on macOS, `systemctl --user` on Linux) [default: system] [possible values: system, user]
  -u, --user <USER>                  User the service runs as (default: $SUDO_USER or the current user)
      --hp <HOME_PATH>               Home directory used to locate the 3pm home (default: $HOME) [alias: --home]
      --service-name <SERVICE_NAME>  Service name / label (default: io.3pm.daemon)
  -h, --help                         Print help
  -V, --version                      Print version
```

## 3pm unstartup

```text
Remove the 3pm boot service (prints the command to run as root)

Usage: 3pm unstartup [OPTIONS] [PLATFORM]

Arguments:
  [PLATFORM]  Init system (launchd, systemd, openrc, rcd); auto-detected if omitted

Options:
      --json                         Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
      --service-name <SERVICE_NAME>  Service name / label (default: io.3pm.daemon)
      --level <LEVEL>                Service level the service was installed at (system, user) [default: system] [possible values: system, user]
  -h, --help                         Print help
  -V, --version                      Print version
```

## 3pm install

```text
Install a module (a bundled name or a directory holding a module.toml) and start its apps; without argument, list modules

Usage: 3pm install [OPTIONS] [SOURCE]

Arguments:
  [SOURCE]  Module name (bundled, installed, else downloaded from the site), `<name>@<version>` (from the site), or a directory holding a module.toml (omit to list what is installed and bundled)

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm uninstall

```text
Uninstall a module: delete its apps and remove its files

Usage: 3pm uninstall [OPTIONS] <NAME>

Arguments:
  <NAME>  Module name

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm link

```text
Enroll this host and start the agent app (the pasted line); alone, show the link and exit 0 only if it is established; `link stop` / `link start` / `link status` / `link delete` act on an existing link

Usage: 3pm agent link [OPTIONS] [CLUSTER_KEY] [SECRET_KEY] [SERVER]
       3pm agent link <COMMAND>

Commands:
  stop         Stop the agent app and keep the enrollment (`link start` resumes)
  start        Start the agent app again on an existing enrollment (registers it again if it was deleted)
  status       Same as `3pm agent status`
  delete       Disconnect and forget the enrollment (key, link.toml) — same as `3pm unlink`
  authorize    Let the caller whose key this is open `3pm fleet ssh` sessions here (`3pm fleet key` prints it on the caller's machine). Local: the fleet cannot add a key
  deauthorize  Stop admitting a caller's key, named by its fingerprint or its comment
  help         Print this message or the help of the given subcommand(s)

Arguments:
  [CLUSTER_KEY]  Cluster key `3pmc_<id>` (or THREEPM_LINK_CLUSTER)
  [SECRET_KEY]   Enrollment token `3pmk_<id>.<secret>` (or THREEPM_LINK_TOKEN)
  [SERVER]       Backend address: wss://host[:port] or ws://host[:port]; default wss://fleet.edgecr4ft.net (or THREEPM_LINK_SERVER, which wins)

Options:
      --insecure              Accept a clear-text ws:// server that is not loopback, or any TLS certificate (or THREEPM_LINK_INSECURE=1)
      --name <NAME>           Name of the supervised agent app [default: agent]
      --cap <CAP>             Capability to offer the backend (repeatable; default: all of metrics:read, proc:restart, proc:stop, proc:action, node:shell, node:forward)
      --no-env-keys           Do not ship the environment variable *names* of the processes
      --force                 Enroll again even if this host is already linked
      --authorize <SHA256:…>  Admit this caller's key to `3pm fleet ssh` sessions here (repeatable; also THREEPM_LINK_AUTHORIZE, comma-separated)
      --json                  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help                  Print help
```

## 3pm link stop

```text
Stop the agent app and keep the enrollment (`link start` resumes)

Usage: 3pm agent link stop [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm link start

```text
Start the agent app again on an existing enrollment (registers it again if it was deleted)

Usage: 3pm agent link start [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm link status

```text
Same as `3pm agent status`

Usage: 3pm agent link status [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm link delete

```text
Disconnect and forget the enrollment (key, link.toml) — same as `3pm unlink`

Usage: 3pm agent link delete [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm link authorize

```text
Let the caller whose key this is open `3pm fleet ssh` sessions here (`3pm fleet key` prints it on the caller's machine). Local: the fleet cannot add a key

Usage: 3pm agent link authorize [OPTIONS] <SHA256:…>

Arguments:
  <SHA256:…>  The caller's key, `SHA256:…`

Options:
      --comment <COMMENT>  A name for it, shown by `3pm link status`
      --json               Machine output on stdout (also THREEPM_JSON=1)
  -h, --help               Print help
```

## 3pm link deauthorize

```text
Stop admitting a caller's key, named by its fingerprint or its comment

Usage: 3pm agent link deauthorize [OPTIONS] <SHA256:…|COMMENT>

Arguments:
  <SHA256:…|COMMENT>  `SHA256:…`, or the key's comment

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm unlink

```text
Stop and delete the agent app, forget the enrollment

Usage: 3pm agent unlink [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm fleet

```text
`3pm fleet --help` needs a linked host, and answered here:
[3PM][ERROR] `3pm fleet` needs a link to a fleet backend: run `3pm link <cluster-key> <token>` (nothing at ~/.3pm/agent/link.toml)

The same subcommands, with their help, are under `3pm agent fleet`.
```

## 3pm module

```text
Modules: optional features bundled with 3pm (`3pm module list`); alone, lists them and prints this help

Usage: 3pm module [OPTIONS] [COMMAND]

Commands:
  list  List installed and bundled modules
  help  Print this message or the help of the given subcommand(s)

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm module list

```text
List installed and bundled modules

Usage: 3pm module list [OPTIONS]

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm completions

```text
Generate shell completions

Usage: 3pm completions [OPTIONS] <SHELL>

Arguments:
  <SHELL>  [possible values: bash, elvish, fish, powershell, zsh]

Options:
      --json     Machine-readable JSON on stdout (NDJSON for `logs`); chrome and errors go to stderr. Ignored by CLI-local commands (startup, completions)
  -h, --help     Print help
  -V, --version  Print version
```

## 3pm agent

```text
Link this 3pm to a fleet backend and stream its processes there

Usage: 3pm agent <COMMAND>

Commands:
  link    Enroll this host and start the agent app (the pasted line); alone, show the link and exit 0 only if it is established; `link stop` / `link start` / `link status` / `link delete` act on an existing link
  unlink  Stop and delete the agent app, forget the enrollment
  status  What this host is linked to, and how the agent app is doing
  fleet   Read this node's own bucket with this node's own identity: `fleet nodes`, `fleet apps [node]`, `fleet inspect <node> [app]`, `fleet top`, `fleet call`, `fleet ssh <node>`, `fleet cp`
  help    Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

Examples:
  3pm agent link 3pmc_… 3pmk_…     # to the hosted backend, wss://fleet.edgecr4ft.net
  3pm agent link 3pmc_… 3pmk_… wss://fleet.example.com
  3pm agent link           # the link, exit 0 only when established
  3pm agent link stop      # pause the link, keep the enrollment
  3pm agent link start     # resume it
  3pm agent link delete    # disconnect, forget the enrollment (= unlink)
  3pm agent status
  3pm agent unlink
  3pm agent fleet nodes    # every node of the bucket (needs fleet:list)
  3pm agent fleet apps edge-7
  3pm agent fleet inspect edge-7 vision
  3pm agent fleet top      # the bucket live, until `q`
  3pm agent fleet call edge-7 vision infer '{"n":1}' --file in.jpg --out .
  3pm agent fleet cp -r ./site edge-7:/srv
```

## 3pm agent link

```text
Enroll this host and start the agent app (the pasted line); alone, show the link and exit 0 only if it is established; `link stop` / `link start` / `link status` / `link delete` act on an existing link

Usage: 3pm agent link [OPTIONS] [CLUSTER_KEY] [SECRET_KEY] [SERVER]
       3pm agent link <COMMAND>

Commands:
  stop         Stop the agent app and keep the enrollment (`link start` resumes)
  start        Start the agent app again on an existing enrollment (registers it again if it was deleted)
  status       Same as `3pm agent status`
  delete       Disconnect and forget the enrollment (key, link.toml) — same as `3pm unlink`
  authorize    Let the caller whose key this is open `3pm fleet ssh` sessions here (`3pm fleet key` prints it on the caller's machine). Local: the fleet cannot add a key
  deauthorize  Stop admitting a caller's key, named by its fingerprint or its comment
  help         Print this message or the help of the given subcommand(s)

Arguments:
  [CLUSTER_KEY]  Cluster key `3pmc_<id>` (or THREEPM_LINK_CLUSTER)
  [SECRET_KEY]   Enrollment token `3pmk_<id>.<secret>` (or THREEPM_LINK_TOKEN)
  [SERVER]       Backend address: wss://host[:port] or ws://host[:port]; default wss://fleet.edgecr4ft.net (or THREEPM_LINK_SERVER, which wins)

Options:
      --insecure              Accept a clear-text ws:// server that is not loopback, or any TLS certificate (or THREEPM_LINK_INSECURE=1)
      --name <NAME>           Name of the supervised agent app [default: agent]
      --cap <CAP>             Capability to offer the backend (repeatable; default: all of metrics:read, proc:restart, proc:stop, proc:action, node:shell, node:forward)
      --no-env-keys           Do not ship the environment variable *names* of the processes
      --force                 Enroll again even if this host is already linked
      --authorize <SHA256:…>  Admit this caller's key to `3pm fleet ssh` sessions here (repeatable; also THREEPM_LINK_AUTHORIZE, comma-separated)
      --json                  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help                  Print help
```

## 3pm agent link stop

```text
Stop the agent app and keep the enrollment (`link start` resumes)

Usage: 3pm agent link stop [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm agent link start

```text
Start the agent app again on an existing enrollment (registers it again if it was deleted)

Usage: 3pm agent link start [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm agent link status

```text
Same as `3pm agent status`

Usage: 3pm agent link status [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm agent link delete

```text
Disconnect and forget the enrollment (key, link.toml) — same as `3pm unlink`

Usage: 3pm agent link delete [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm agent link authorize

```text
Let the caller whose key this is open `3pm fleet ssh` sessions here (`3pm fleet key` prints it on the caller's machine). Local: the fleet cannot add a key

Usage: 3pm agent link authorize [OPTIONS] <SHA256:…>

Arguments:
  <SHA256:…>  The caller's key, `SHA256:…`

Options:
      --comment <COMMENT>  A name for it, shown by `3pm link status`
      --json               Machine output on stdout (also THREEPM_JSON=1)
  -h, --help               Print help
```

## 3pm agent link deauthorize

```text
Stop admitting a caller's key, named by its fingerprint or its comment

Usage: 3pm agent link deauthorize [OPTIONS] <SHA256:…|COMMENT>

Arguments:
  <SHA256:…|COMMENT>  `SHA256:…`, or the key's comment

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm agent unlink

```text
Stop and delete the agent app, forget the enrollment

Usage: 3pm agent unlink [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm agent status

```text
What this host is linked to, and how the agent app is doing

Usage: 3pm agent status [OPTIONS]

Options:
      --json  Machine output on stdout (also THREEPM_JSON=1)
  -h, --help  Print help
```

## 3pm agent fleet

```text
Read this node's own bucket with this node's own identity: `fleet nodes`, `fleet apps [node]`, `fleet inspect <node> [app]`, `fleet top`, `fleet call`, `fleet ssh <node>`, `fleet cp`

Usage: 3pm agent fleet <COMMAND>

Commands:
  nodes        Every node of the bucket: host, os, state, how long since it was seen, how many apps it runs, whose account runs it and where it is reached
  apps         Every process of the bucket, or of one node (a `3pmn_…` id, a hostname, or an unambiguous start of one)
  inspect      One node in full — and one of its apps, when named
  top          The bucket live, in two panels, until `q`: nodes on top, processes below, redrawn once a second from the event stream
  call         Place a call on another node of the bucket, with this node's own identity, and wait for it: `--file` sends bytes, `--out` keeps what came back
  ssh          An interactive login shell on another node of the bucket, like ssh (`~.` after Enter to leave) — or one command, after `--`
  attach       The terminal of a `tty = true` app on another node of the bucket, like `3pm attach` (`~.` after Enter to detach: the app runs on)
  cp           Copy files to or from another node of the bucket, like scp: one side is `<node>:<path>`
  known-nodes  The shell keys of the nodes this host has met, pinned at the first session (`~/.3pm/known_nodes`, like ssh's `known_hosts`)
  key          This host's caller key (`cli.key`, or THREEPM_CLI_KEY): its fingerprint and the line a node needs to admit it; `key new`, `key push`, `key revoke`
  help         Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help
```

## 3pm agent fleet nodes

```text
Every node of the bucket: host, os, state, how long since it was seen, how many apps it runs, whose account runs it and where it is reached

Usage: 3pm agent fleet nodes [OPTIONS]

Options:
      --json            Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>  Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help            Print help
```

## 3pm agent fleet apps

```text
Every process of the bucket, or of one node (a `3pmn_…` id, a hostname, or an unambiguous start of one)

Usage: 3pm agent fleet apps [OPTIONS] [NODE]

Arguments:
  [NODE]  

Options:
      --json            Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>  Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help            Print help
```

## 3pm agent fleet inspect

```text
One node in full — and one of its apps, when named

Usage: 3pm agent fleet inspect [OPTIONS] <NODE> [APP]

Arguments:
  <NODE>  
  [APP]   

Options:
      --json            Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>  Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help            Print help
```

## 3pm agent fleet top

```text
The bucket live, in two panels, until `q`: nodes on top, processes below, redrawn once a second from the event stream

Usage: 3pm agent fleet top [OPTIONS]

Options:
      --json            Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>  Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help            Print help
```

## 3pm agent fleet call

```text
Place a call on another node of the bucket, with this node's own identity, and wait for it: `--file` sends bytes, `--out` keeps what came back

Usage: 3pm agent fleet call [OPTIONS] <NODE> <APP> <ACTION> [PARAMS]

Arguments:
  <NODE>    A node of the bucket: a `3pmn_…` id, a hostname, or an unambiguous start of one
  <APP>     App name on that node (never `all`: a call names one app)
  <ACTION>  Action name, as the app declared it
  [PARAMS]  Parameters, as JSON (default null)

Options:
      --instance <N>        One instance of the app (default 0; required with `--file`)
      --src <SOURCE>        The source (process of the app) to call, when several declare it
      --timeout <DURATION>  How long to wait (`30s`, `2m`, `1500ms`); the declaration's own timeout bounds it (default 30s, ceiling 10m)
      --file <PATH>         A file of this host to send with the call (the action must declare an `input`)
      --out <DIR>           Directory to write the file the action produced into
      --stream              Not from here: a stream is opened by an app with `io.open`
      --json                Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>      Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help                Print help
```

## 3pm agent fleet ssh

```text
An interactive login shell on another node of the bucket, like ssh (`~.` after Enter to leave) — or one command, after `--`

Usage: 3pm agent fleet ssh [OPTIONS] <NODE> [-- <CMD>...]

Arguments:
  <NODE>    A node of the bucket: a `3pmn_…` id, a hostname, or an unambiguous start of one; `<user>@<node>` is `-l <user> <node>`
  [CMD]...  The command to run instead of a shell, through the node's login shell (`sh -c`); the process exits with its code. One word is the whole line, as ssh has it (`-- 'ls | wc -l'`); several are each quoted, so they reach the program as given (`-- cat 'my file'`)

Options:
  -l, --user <USER>               Run as this unix account of the node rather than as its agent's own; only an agent that runs as root becomes someone else
  -t                              Run the command on a pty, even when stdin is not a terminal
  -T                              Run the command without a pty, even from a terminal: stdin, stdout and stderr as pipes, binary-safe
  -L <[BIND:]PORT:HOST:HOSTPORT>  Forward a local port to `host:hostport` as the node reaches it, every connection its own; repeatable. The bind address is 127.0.0.1 unless given (`*` for every interface)
  -R <[BIND:]PORT:HOST:HOSTPORT>  Forward a port of the node to `host:hostport` as this host reaches it, every connection its own; repeatable. The node binds its loopback unless given, and anything else only when its link.toml says `[shell] gateway_ports = true`; port 0: one the node picks, printed
  -N                              No shell: only the `-L` and `-R` forwards, until Ctrl-C
  -W <HOST:PORT>                  Connect stdin and stdout to `host:port` as the node reaches it, like ssh's `-W`: for ssh's `ProxyCommand`. Exits 0 when both ways are closed, 255 when the node or the backend refused or cut it
  -v                              Say on stderr which way the session's legs go: through this host's agent (LAN direct when they can) or the backend's relay, and why
  -C, --compress                  Compress the command's bytes, both ways, when it runs without a pty (`-T`, or stdin not a terminal) and the node's agent can (zstd, inside the encrypted channel; `-v` says whether it is). Never a shell, a pty or a forward
      --accept-new                Pin the key of a node met for the first time without asking (for scripts); a key that changed is still refused
      --no-e2e                    In clear, for a node whose agent is too old to encrypt: the backend can read and change the session. An up-to-date node refuses it
      --record                    Record the session (its screen and its command lines) when the bucket records shells (`record_shells`); without it, never — the handshake binds it, so the backend cannot record it anyway
      --json                      Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>            Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help                      Print help
```

## 3pm agent fleet attach

```text
The terminal of a `tty = true` app on another node of the bucket, like `3pm attach` (`~.` after Enter to detach: the app runs on)

Usage: 3pm agent fleet attach [OPTIONS] <NODE> <APP>

Arguments:
  <NODE>  A node of the bucket: a `3pmn_…` id, a hostname, or an unambiguous start of one
  <APP>   A `tty = true` app of that node, by name

Options:
      --instance <N>    One instance of the app (default 0)
      --raw             The app's bytes as it wrote them, not the node's render of its screen
      --peek            Print the screen as text and leave
      --read-only       Watch only: no keystroke and no resize reaches the app
      --accept-new      Pin the key of a node met for the first time without asking (for scripts); a key that changed is still refused
      --no-e2e          In clear, for a node whose agent is too old to encrypt: the backend can read and change the session. An up-to-date node refuses it
      --record          Record the session (its screen and its command lines) when the bucket records shells (`record_shells`); without it, never — the handshake binds it, so the backend cannot record it anyway
      --json            Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>  Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help            Print help
```

## 3pm agent fleet cp

```text
Copy files to or from another node of the bucket, like scp: one side is `<node>:<path>`

Usage: 3pm agent fleet cp [OPTIONS] <SRC… DST> <SRC… DST>...

Arguments:
  <SRC… DST> <SRC… DST>...  The sources, then the destination. Exactly one side is `<node>:<path>` (a node as `fleet ssh` takes it; `<node>:` alone is its home); a local path with a `:` before any `/` is written `./a:b`. Several sources need a destination directory

Options:
  -r                    Copy directories, whole (a file is copied as a file)
  -p                    Keep each file's mode and modification time
  -l, --user <USER>     Run as this unix account of the node rather than as its agent's own; only an agent that runs as root becomes someone else
      --no-compress     Send the bytes as they are. By default they are compressed inside the encrypted channel when the node's agent can (zstd; a file that does not compress goes as it is)
      --accept-new      Pin the key of a node met for the first time without asking (for scripts); a key that changed is still refused
      --no-e2e          In clear, for a node whose agent is too old to encrypt: the backend can read and change the session. An up-to-date node refuses it
      --record          Record the session (its screen and its command lines) when the bucket records shells (`record_shells`); without it, never — the handshake binds it, so the backend cannot record it anyway
      --json            Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>  Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help            Print help
```

## 3pm agent fleet known-nodes

```text
The shell keys of the nodes this host has met, pinned at the first session (`~/.3pm/known_nodes`, like ssh's `known_hosts`)

Usage: 3pm agent fleet known-nodes <COMMAND>

Commands:
  ls    The pinned nodes: id, fingerprint, name, date
  rm    Forget a node's key (its `3pmn_…` id, or its name as pinned), like `ssh-keygen -R`: the next session asks again
  help  Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help
```

## 3pm agent fleet known-nodes ls

```text
The pinned nodes: id, fingerprint, name, date

Usage: 3pm agent fleet known-nodes ls [OPTIONS]

Options:
      --json            Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>  Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help            Print help
```

## 3pm agent fleet known-nodes rm

```text
Forget a node's key (its `3pmn_…` id, or its name as pinned), like `ssh-keygen -R`: the next session asks again

Usage: 3pm agent fleet known-nodes rm [OPTIONS] <NODE>

Arguments:
  <NODE>  

Options:
      --json            Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>  Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help            Print help
```

## 3pm agent fleet key

```text
This host's caller key (`cli.key`, or THREEPM_CLI_KEY): its fingerprint and the line a node needs to admit it; `key new`, `key push`, `key revoke`

Usage: 3pm agent fleet key [OPTIONS] [COMMAND]

Commands:
  new     A new caller key in a file of its own, offline (a rescue key: keep it off this host; use it with THREEPM_CLI_KEY=<file>)
  push    Admit a key on nodes, through a `fleet ssh` session on each (this host's key must already be admitted there)
  revoke  Stop admitting a key on nodes, the same way
  help    Print this message or the help of the given subcommand(s)

Options:
      --json            Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>  Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help            Print help
```

## 3pm agent fleet key new

```text
A new caller key in a file of its own, offline (a rescue key: keep it off this host; use it with THREEPM_CLI_KEY=<file>)

Usage: 3pm agent fleet key new [OPTIONS] --out <PATH>

Options:
      --out <PATH>      Where to write it (0600); an existing file is never replaced
      --json            Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>  Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help            Print help
```

## 3pm agent fleet key push

```text
Admit a key on nodes, through a `fleet ssh` session on each (this host's key must already be admitted there)

Usage: 3pm agent fleet key push [OPTIONS] <NODE… SHA256:…>...

Arguments:
  <NODE… SHA256:…>...  The nodes (as `fleet ssh` takes them), then the key's `SHA256:…`

Options:
      --all                Every node of the bucket
      --comment <COMMENT>  A name for the key, shown by `3pm link status` (push only)
      --accept-new         Pin the key of a node met for the first time without asking
      --json               Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>     Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help               Print help
```

## 3pm agent fleet key revoke

```text
Stop admitting a key on nodes, the same way

Usage: 3pm agent fleet key revoke [OPTIONS] <NODE… SHA256:…>...

Arguments:
  <NODE… SHA256:…>...  The nodes (as `fleet ssh` takes them), then the key's `SHA256:…`

Options:
      --all                Every node of the bucket
      --comment <COMMENT>  A name for the key, shown by `3pm link status` (push only)
      --accept-new         Pin the key of a node met for the first time without asking
      --json               Machine output on stdout (also THREEPM_JSON=1)
      --ca-file <PATH>     Trust the certificates of this PEM file **as well as** the built-in roots — for a private CA, or a public root this build is too old to know
  -h, --help               Print help
```

## 3pm serve

```text
Serve a directory over HTTP as a 3pm-managed app (pm2 serve)

Usage: 3pm serve [OPTIONS] [PATH] [PORT]

Arguments:
  [PATH]  Directory to serve (default: current directory), or a single .html file: its directory is served with that page as homepage, opened in your browser
  [PORT]  Port to listen on (default: 8080, or the next free port above it; 0 lets the kernel pick)

Options:
      --port <PORT>                     Port to listen on (same as the positional)
      --host <HOST>                     Address to bind (default: 0.0.0.0)
      --spa                             Single-page app: serve the homepage for unknown paths
      --ftp                             Directory listing, like `python -m http.server`
      --homepage <FILE>                 File served at `/` (default: /index.html)
      --basic-auth-username <USER>      Basic auth user (with --basic-auth-password)
      --basic-auth-password <PASSWORD>  Basic auth password (with --basic-auth-username)
      --name <NAME>                     App name (default: static-page-server-<port>)
      --namespace <NAMESPACE>           App namespace
      --no-qr                           Do not print the QR code of the network link
      --attach                          Stay in the foreground and stop the server on Ctrl-C (the default when serving a single .html file)
      --detach                          Leave the server running in the background and return (the default for a directory)
      --json                            Machine output: {"affected":[…]} on stdout (also THREEPM_JSON=1)
  -h, --help                            Print help
  -V, --version                         Print version

Examples:
  3pm serve ./dist 8080 --spa
  3pm serve --port 9000 --ftp
  3pm serve . --basic-auth-username admin --basic-auth-password s3cret
```

</document>

<document url="https://3pm.edgecr4ft.net/docs/ECOSYSTEM.md" title="Ecosystem file">

# Ecosystem file cheatsheet

One file, many apps, one `3pm apply`. TOML is the native format;
`.json`, `.yaml`/`.yml` and pm2's `*.config.js|cjs|mjs` carry the same
fields (see [Other formats](#other-formats)). Every field below comes
from one table, `StartSpec` in `crates/threepm-core/src/spec.rs` — the
same names, the same units, whether it comes from the file,
`[defaults]`, a profile or a CLI flag; the
[field reference](#field-reference) below is generated from it. Unknown
keys are **errors** (with a did-you-mean), never ignored. Long-form
rationale: `FEATURES.md` → *Config workflow*.

## The whole file, annotated

Every value shown is the **default** unless the comment says otherwise;
a line you delete keeps behaving exactly as written here.

```toml
[defaults]                          # applied to every app first; may not
restart_policy = "always"           # carry name / script / profiles / env_*
log_rotate_size = "0"

[[apps]]
# ── identity ──────────────────────────────────────────────────────────
name = "api"                        # required, unique in the file
script = "./server.js"              # required; relative to the file's dir; aliases exec, cmd, command;
                                    # "npm run dev" (a blank, no such file) runs under bash -c
args = ["--port", "3000"]           # list, replaced (not merged) by profiles
interpreter = "node"                # default by extension: js/ts→node, py→python3, rb→ruby, sh→bash, pl→perl;
                                    # "none" = run the script itself; alias exec_interpreter
interpreter_args = []               # the interpreter's flags, before the script (pm2 node_args,
                                    # interpreterArgs); a string is split on blanks
cwd = "~/srv/api"                   # default: the file's directory
namespace = "backend"               # groups apps for `stop|restart|delete <ns>`
instances = 1                       # 0 = one per CPU (cluster-style)
user = "srt"                        # default unset: the daemon's own account. A ROOT daemon runs
                                    # the app as this Unix account (uid, gid, groups, clean env),
                                    # and only for a user it hosts ([users] allow); "" = the daemon's
persist = true                      # false = ephemeral: kept across restarts and `3pm update`,
                                    # never written to dump.json, so a reboot does not bring it back

# ── environment ───────────────────────────────────────────────────────
[apps.env]                          # merged by key across layers
NODE_ENV = "development"
PORT = 3000                         # bare numbers / booleans are stringified
DATA = "${HOME}/data"               # $VAR, ${VAR}, leading ~ expand; $$ = literal $
                                    # an unset variable is an error, not ""

[apps.env_production]               # pm2 sugar: env-only profile (--env production)
NODE_ENV = "production"

[apps.profiles.staging]             # full profile: ANY field (--env staging)
instances = 2
[apps.profiles.staging.env]
NODE_ENV = "staging"

[[apps]]
name = "worker"
script = "./worker.py"

# ── restart policy ────────────────────────────────────────────────────
restart_policy = "always"           # always | on_failure | never
autorestart = true                  # pm2 bool; false == "never" (contradiction = error)
max_restarts = 16                   # unstable restarts before `errored`
min_uptime = 1000                   # ms; an exit before this counts as unstable
restart_delay = 0                   # ms, fixed pause before every crash restart (0 = at once, pm2)
exp_backoff_restart_delay = 0       # ms; > 0 enables exponential backoff between crash
                                    # restarts starting at this delay (pm2 name; off by default)
backoff_factor = 1.5                #     ┐ backoff tuning (only with exp_backoff_restart_delay):
backoff_max = 15000                 # ms  │ growth per quick crash, cap, and the uptime after
backoff_reset_after = 30000         # ms  ┘ which the delay resets to the first one
stop_exit_codes = []                # exit codes that mean "stopped on purpose"
cron_restart = ""                   # "0 3 * * *" (5 or 6 fields); "" = off

# ── stop / readiness ──────────────────────────────────────────────────
kill_signal = "SIGINT"              # portable POSIX names (SIGTERM, SIGHUP, …)
kill_timeout = 1600                 # ms before SIGKILL escalation
shutdown_with_message = false       # send `shutdown` on ctl.sock before the signal
wait_ready = false                  # wait for `ready` on ctl.sock …
listen_timeout = 3000               # … at most this many ms

# ── start order ───────────────────────────────────────────────────────
depends_on = ["api"]                # default []: wait until these are online
depends_timeout = 60000             # ms; 0 = wait forever
start_order = 0                     # lower starts first among independents
start_delay = 0                     # ms before the first spawn

# ── resources ─────────────────────────────────────────────────────────
max_memory_restart = "200M"         # default unset; "512", "100K", "1G"
max_cpu_percent = 150.0             # default unset; % of one core, may exceed 100
limit_scope = "tree"                # what both judge: "tree" (the app's whole process tree) or "leader" (the spawned pid)
cgroup_enforce = false              # Linux only: hard cpu.max / memory.max
deny_gpu = false                    # hide CUDA/ROCm devices via env
ulimit_rtprio = 95                  # default unset; Linux only: RLIMIT_RTPRIO, so the app may run SCHED_FIFO up to 95
ulimit_memlock = "unlimited"        # default unset; RLIMIT_MEMLOCK: a size ("64M", "4G") or "unlimited"
ulimit_nice = 31                    # default unset; Linux only: RLIMIT_NICE, so the app's threads may go down to nice -11
ulimit_rttime = 200000              # default unset; Linux only: RLIMIT_RTTIME, µs a real-time thread may run without blocking
nice = -5                           # default unset (the daemon's); -20 (most CPU) to 19
ioprio = "idle"                     # default unset; Linux only: "idle", "best-effort[:0-7]", "realtime[:0-7]" (BFQ only)
oom_score_adj = -900                # default unset; Linux only: -1000 (never OOM-killed) to 1000 (first)
cpu_affinity = "0-3"                # default unset; Linux only: the CPUs it may run on

# ── watch ─────────────────────────────────────────────────────────────
watch = true                        # default off; true = cwd, or a path (a dir, or one file), or a list
ignore_watch = ["dist"]             # on top of the built-ins: ".*", "node_modules"
watch_delay = 0                     # ms debounce (a floor is always applied)
watch_poll = 1000                   # ms; default unset = native notifications

# ── logs ──────────────────────────────────────────────────────────────
out_file = "~/.3pm/logs/{name}-out.log"    # aliases: out, output, out_log; NULL = no file
error_file = "~/.3pm/logs/{name}-error.log"# aliases: error, err, err_file, err_log
log_file = ""                       # alias: log; one merged file for both streams
merge_logs = false                  # alias: combine_logs — instances share files
log_date_format = ""                # strftime persisted in front of each line
disable_logs = false                # no file, no stream, no crash tail
log_rotate_size = "0"               # "0" off, "10M", "1G" — per-app override of the daemon policy
log_rotate_files = 5                # generations kept (.1 .. .N) once rotation is on
log_rotate_days = 14                # 0 = keep by count only
log_rotate_compress = false         # gzip rotated generations
```

`{name}`, `{instance}`, `{namespace}` expand in log paths. A multi-instance
app whose custom path lacks `{instance}` gets `-<instance>` infixed.

## Field reference

Generated from the daemon's own table (`just gen`, source
`crates/threepm-core/src/spec.rs`): one line per field a file or a flag
may set, with the spellings it answers to, the wire name when it differs,
and the default the daemon uses when the field is absent.

<!-- spec:begin -->
| Field | Type | Default | What it sets |
|---|---|---|---|
| `name` | string | *unset* | Unique app name; the CLI defaults it to the script's file stem. |
| `script`, `exec`, `cmd`, `command` | string | *unset* | Absolute path to the script or binary (resolved client-side). |
| `args` | list | `[]` | Arguments passed to the app, after the script (`3pm start x -- a b`). |
| `interpreter`, `exec_interpreter` | string | *unset* | Absolute path to the interpreter (resolved client-side against the invoking shell's PATH — the daemon's env freezes at first spawn). `None` means direct exec of `script`; `INTERPRETER_NONE` clears it. |
| `interpreter_args`, `node_args`, `interpreterArgs` | list | `[]` | The interpreter's own flags, between it and the script (pm2's `node_args`: `node --inspect app.js`). Never without an interpreter. A string is split on blanks, like pm2. |
| `cwd` | string | *unset* | Absolute working directory. |
| `env` | map | `{}` | Values are strings on the wire; numbers and booleans in a config file (`PORT = 3000`, pm2 habit) are stringified on decode. Merged by key across layers, never replaced. |
| `namespace` | string | *unset* | Grouping label: `stop\|restart\|delete <namespace>` acts on all of them. |
| `instances` | u32 | `1` | Desired instance count (cluster mode). `0` = one per CPU, resolved by the daemon at instance-creation time. |
| `autorestart` → `restart_policy` | bool | `true` | pm2's bool: `false` is shorthand for `restart_policy = "never"`. |
| `restart_policy` | enum(always\|on_failure\|never) | `always` | Wins over `autorestart` when both are given and agree; a contradiction is a `SpecError::ConflictingRestartPolicy`. |
| `kill_signal` | string | `SIGINT` | Signal name sent on stop before escalation (pm2 default: SIGINT). |
| `wait_ready` | bool | `false` | Hold `launching` until the app signals readiness on the control socket (pm2's wait_ready); `listen_timeout_ms` bounds the wait. |
| `listen_timeout` → `listen_timeout_ms` | u64 (ms) | `3000` | Max wait for the ready signal before forcing online (pm2's GRACEFUL_LISTEN_TIMEOUT). |
| `shutdown_with_message` | bool | `false` | Deliver the graceful stop as a control-socket `shutdown` message instead of `kill_signal` (SIGKILL escalation unchanged). |
| `max_memory_restart` | size | *unset* | Restart when RSS exceeds this many bytes. Written as a human size string: "512", "100K", "200M", "1G". |
| `max_cpu_percent` | f64 (percent) | *unset* | Restart when a sampled CPU% exceeds this (soft ceiling, same observe-and-restart shape as `max_memory_restart`). Positive, finite; may exceed 100 on multi-core hosts. |
| `cgroup_enforce` | bool | `false` | Linux only: put each incarnation in its own cgroup v2 with `cpu.max` / `memory.max` derived from `max_cpu_percent` / `max_memory_restart` — a HARD kernel limit, not a restart trigger. |
| `limit_scope` | enum(tree\|leader) | `tree` | What `max_memory_restart` / `max_cpu_percent` judge: the whole process tree (the default) or the spawned pid alone. |
| `deny_gpu` | bool | `false` | Hide GPUs from the child (best effort: the CUDA / ROCm visibility env vars are set to "none"). |
| `ulimit_rtprio` | u8 | *unset* | Linux only: the app's `RLIMIT_RTPRIO` (soft = hard, 0–99), so its threads may ask for `SCHED_FIFO` / `SCHED_RR` up to that priority. Unset = inherited from the daemon (0 under a systemd unit). Above the daemon's own limit only a root daemon can grant it. |
| `ulimit_memlock` | size | *unset* | The app's `RLIMIT_MEMLOCK` (soft = hard): how much it may lock in RAM (`mlock`, real-time audio buffers). A size ("64M", "4G") or "unlimited". Unset = inherited from the daemon. |
| `ulimit_nice` | u8 | *unset* | Linux only: the app's `RLIMIT_NICE` (soft = hard, 0–40), so its threads may lower their nice value down to `20 - v` (Chromium's audio threads ask for -8: 28). Unset = inherited from the daemon (0 under a systemd unit: nothing below the nice it starts with). |
| `ulimit_rttime` → `ulimit_rttime_us` | u64 (us) | *unset* | Linux only: the app's `RLIMIT_RTTIME` (soft = hard), in microseconds: the CPU time a real-time thread may take without blocking before the kernel kills it (SIGKILL, no SIGXCPU warning: soft = hard) — the guard against a `SCHED_FIFO` loop that freezes the machine. Unset = inherited (unlimited under a systemd unit). |
| `nice` | i8 | *unset* | The app's nice value (-20–19, lower = more CPU), set before `exec` and inherited by its children. Unset = the daemon's own. Below the daemon's takes privilege or `ulimit_nice`. |
| `ioprio` | ioprio | *unset* | Linux only: the app's I/O scheduling class: "idle", "best-effort[:0-7]" or "realtime[:0-7]" (0 first). Honoured by the BFQ and mq-deadline disk schedulers, not by `none` (the NVMe default). Unset = the daemon's own. |
| `oom_score_adj` | i16 | *unset* | Linux only: the app's `oom_score_adj` (-1000–1000): lower is killed later when memory runs out, -1000 never. Unset = the daemon's own. Below the daemon's takes privilege. |
| `cpu_affinity` | cpulist | *unset* | Linux only: the CPUs the app may run on, as the kernel lists them ("0-3,6"). Each must be one this daemon may use. Unset = all of the daemon's. |
| `max_restarts` | u32 | `16` | Unstable restarts before the app is parked in `errored`. |
| `min_uptime` → `min_uptime_ms` | u64 (ms) | `1000` | An exit before this counts as unstable. |
| `kill_timeout` → `kill_timeout_ms` | u64 (ms) | `1600` | Grace before SIGKILL escalation. |
| `restart_delay` → `restart_delay_ms` | u64 (ms) | `0` | Fixed delay before every crash respawn (pm2 `restart_delay`); `0` = respawn at once (pm2 default). |
| `exp_backoff_restart_delay`, `backoff_initial` → `backoff_initial_ms` | u64 (ms) | `0` | pm2 `exp_backoff_restart_delay`: `0` = exponential backoff OFF (the default — crash respawns use `restart_delay_ms` only); otherwise the first backoff delay, grown by `backoff_factor` per quick crash up to `backoff_max_ms`. Wire name kept for compatibility; see `StartSpec::exp_backoff_enabled`. |
| `backoff_factor` | f64 | `1.5` | Growth of the backoff delay per quick crash. |
| `backoff_max` → `backoff_max_ms` | u64 (ms) | `15000` | Cap on the backoff delay. |
| `backoff_reset_after` → `backoff_reset_after_ms` | u64 (ms) | `30000` | Uptime after which the backoff delay resets to the first one. |
| `watch` | bool\|path\|list | `false` | Absolute directories/files to watch; empty = watch off. Relative entries are resolved against `cwd` at the CLI edge (pm2 rule). Written as `true` (the cwd), a path, or a list of paths. |
| `ignore_watch` | list | `[]` | Glob patterns ADDED to `watch::DEFAULT_IGNORE_WATCH`. |
| `watch_delay` → `watch_delay_ms` | u64 (ms) | `0` | Trailing debounce before a watch-triggered restart (floored at `watch::WATCH_MIN_DEBOUNCE_MS`). |
| `watch_poll` → `watch_poll_ms` | u64 (ms) | *unset* | When set, poll at this interval instead of using OS events (NFS, container bind mounts). `None` = native backend, polling only as a fallback when the native one cannot be set up. |
| `log_date_format` | string | *unset* | strftime format prefixed (plus `": "`) to every persisted log line. `None` = raw bytes, exactly the child's output (pm2's `log_date_format`). Validated by `validate_date_format`. |
| `merge_logs`, `combine_logs` | bool | `false` | stdout and stderr share one file (pm2's `merge_logs`). |
| `out_file`, `out`, `output`, `out_log` | string | *unset* | Custom log destinations. `None` = derived from `Paths`. Absolute (the CLI edge resolves relative paths against `cwd`); may contain `{name}`, `{instance}`, `{namespace}` placeholders — see `paths::Paths::log_targets`. `"NULL"` / `/dev/null` disables that stream's file (the bus still carries it). |
| `error_file`, `error`, `err`, `err_file`, `err_log` | string | *unset* | stderr destination, same rules as `out_file`. |
| `log_file`, `log` | string | *unset* | pm2's `log_file`: one file for both streams; implies `merge_logs`. |
| `disable_logs` | bool | `false` | pm2's `disable_logs`: no file, no bus, no crash tail. The pumps still drain the pipes so the child never blocks. |
| `tty` | bool | `false` | Run the app on a pseudo-terminal instead of pipes: `isatty` is true, stdin is the terminal, and `3pm attach` connects to it. One output stream (no stderr): the log is the merged file, stripped of escape sequences; `error_file` is refused. `TERM` and `COLORTERM` are set unless `env` names them. |
| `log_rotate_size` → `log_rotate` | size | *unset* | Rotate once the live file reaches this size; "0" = never. Unset = the daemon-wide policy. |
| `log_rotate_files` → `log_rotate` | u32 | *unset* | Rotated generations kept (`.1`..`.N`); `0` = keep none. |
| `log_rotate_days` → `log_rotate` | u32 (days) | *unset* | Delete generations older than this; `0` = no age pruning. |
| `log_rotate_compress` → `log_rotate` | bool | *unset* | gzip generations `.2` and older (`.1` stays plain). |
| `stop_exit_codes` | list | `[]` | Exit codes that park the app in `stopped` instead of respawning, whatever `restart_policy` says (pm2's `stop_exit_codes`). |
| `start_delay` → `start_delay_ms` | u64 (ms) | `0` | Delay before the first spawn of a start/restart series, ms. Crash respawns use `restart_delay_ms`/backoff, never this. |
| `depends_on` | list | `[]` | Apps that must be online before this one spawns (see `start_waves` for bulk starts; the daemon gates single spawns). |
| `start_order` | i32 | `0` | Tie-break among apps of one wave (lower starts first). |
| `depends_timeout` → `depends_timeout_ms` | u64 (ms) | `60000` | Max wait for `depends_on` before spawning anyway; `0` = forever. |
| `cron_restart` | cron | *unset* | Restart on a schedule (pm2's `cron_restart`), stored in the 6-field spelling of `cron` (`"0 0 2 * * *"`). `None` = off; a 5-field expression is accepted and `""` switches it off. |
| `user` | string | *unset* | Unix account the app runs as (uid, gid and groups set at spawn, clean environment). `None` = the daemon's own account; `""` hands the app back to it. Only a root daemon honors it, and only for a user declared in `[users] allow`; a hosted caller's apps are forced to its own name. |
| `persist` | bool | `true` | Whether `dump.json` keeps this app (`3pm save`, the implicit save of `3pm update`). `false` = ephemeral: it survives restarts and handoffs, never a reboot. A spec predating the field is `true`. |
<!-- spec:end -->

## Other formats

The format comes from the file name. JSON, YAML and JavaScript take the
same fields and, like pm2, three shapes: `{ apps: [...] }`, a bare list of
apps, or one app alone (an object with a `script` and no `apps`).
`[defaults]` becomes a `defaults` object. pm2's `deploy` section is
ignored.

```yaml
# ecosystem.yaml
apps:
  - name: api
    script: ./server.js
    node_args: --max-old-space-size=256
    env:
      PORT: 3000
  - name: dev
    script: npm run dev        # a command line: bash -c "npm run dev"
```

```js
// ecosystem.config.js — pm2's file, as is
module.exports = {
  apps: [{
    name: 'api',
    script: './server.js',
    instances: 2,
    env: { PORT: process.env.PORT || 3000 },   // your shell's environment
  }],
};
```

A JavaScript config is **code the CLI runs**, as under pm2: `node` from
your PATH evaluates it in your cwd and environment, stdin closed, 10 s and
16 MiB of output at most; `module.exports`, `export default` and `export
const apps` work, and functions vanish (the export crosses as JSON).
`restart --update-env` evaluates it again.

`3pm start -` reads the config on stdin: JSON when it opens with `{` (or
`[` then `{`/`]`), TOML otherwise; relative paths resolve against the cwd.
With no target, `start`, `validate` and `apply` use the current
directory's `ecosystem.toml`, `.json`, `.yaml`, `.yml`, `.config.js`,
`.config.cjs`, `.config.mjs` — the first found.

## Layering

```
[defaults]  →  [[apps]]  →  [apps.profiles.<name>]  →  [apps.env_<name>]
```

| Kind | Rule |
|---|---|
| scalars | later layer wins |
| `env` | merged by key |
| lists (`args`, `watch`, `ignore_watch`, `depends_on`, `stop_exit_codes`) | **replaced** |
| `--env <name>` not defined in the file | error (pm2 would silently ship dev settings) |

## Workflow

```console
$ 3pm validate eco.toml                      # no daemon needed; exit 1 on any error
$ 3pm validate eco.toml --env staging --json # findings as JSON
$ 3pm validate eco.toml --check-managed      # ask the daemon about external depends_on

$ 3pm apply eco.toml                         # idempotent reconcile
$ 3pm apply eco.toml --env production        # with a profile (recorded on the app)
$ 3pm apply eco.toml --only api,worker       # subset, decided on declared names
$ 3pm apply eco.toml --prune                 # + delete apps from this file no longer in it
$ 3pm apply eco.toml --prune-all             # + delete every managed app not in the file
$ 3pm apply eco.toml --dry-run               # print the plan, change nothing

$ 3pm runtime eco.toml --env production      # container entrypoint (PID 1), see FEATURES.md
$ 3pm start eco.toml                         # pm2 habit: always (re)starts
$ 3pm start                                  # the cwd's ecosystem.* (first found)
$ cat apps.json | 3pm start -                # the config on stdin (JSON or TOML)
$ 3pm start ecosystem.config.js              # pm2's file, evaluated by your node
$ 3pm stop eco.toml                          # stop / restart / delete accept a file:
$ 3pm restart eco.toml                       # every managed app it declares
$ 3pm stop eco.toml api 3                    # …and several targets at once
$ 3pm restart api --update-env               # re-read the app's own file, same profile
$ 3pm show api                               # `config file` / `profile` rows
$ 3pm ls --json | jq '.[].spec.source'       # {file, profile} per app
```

What `apply` does per app, in plan order:

| Verdict | When | Effect |
|---|---|---|
| **create** | not managed | started |
| **update** | any field differs | restarted on the new spec (the reason names the fields) |
| **scale** | only `instances` differs | rows added/removed, survivors keep their pid |
| **restart** | same spec, currently stopped | started again |
| **unchanged** | same spec, running | nothing, pid kept |
| **prune** | `--prune`: from this file, no longer declared | deleted (hand-started apps never are) |

## Gotchas, in one place

- Times are **milliseconds** with pm2's bare names (`min_uptime = 1000`),
  sizes are human strings (`"200M"`), CPU is a float percent.
- `restart_policy` beats `autorestart` when both are given and agree; a
  contradiction (`"always"` + `false`) is an error at validate time.
- `[defaults]` cannot set `name`, `script`, `profiles` or `env_*` — each
  is a notice, not silently dropped.
- A name declared twice is an error; a broken app outside `--only` never
  blocks the selected ones.
- `env` values must be scalars: tables, arrays and nulls have no shell
  meaning and are rejected.
- `watch = true` on the file's directory *includes* the ecosystem file
  itself — `ignore_watch` it, or point `watch` at `src/`.
- Re-applying **without** `--env` an app that was applied with a profile
  drops the profile and says so.
- `user` needs a **root** daemon that already hosts that account
  (`[users] allow` in the daemon's `config.toml`, then `3pm update`);
  anywhere else it is an error at start, never "runs as the daemon". An
  app a hosted user declares itself always runs as that user, whatever it
  writes here.
- `owner` is not a field you set: the daemon stamps it from the
  connection that started the app (`FEATURES.md` → *Hosted users*). In a
  file it is an unknown key, like any typo.
- `persist = false` keeps the app out of `3pm save` **and** out of the
  implicit save of `3pm update`; `apply` and `restart` still manage it as
  usual. It is not part of the configuration `apply` compares either:
  changing it alone updates the running rows in place instead of
  restarting them.

## CLI flag ↔ field

Every field is settable from `3pm start` too; the spellings follow pm2
(`--max-memory-restart 200M`, `--kill-timeout 1600`, `-i 4`, `-o/-e/-l`,
`--time` = `log_date_format = "%Y-%m-%dT%H:%M:%S"`, `--no-autorestart`
= `restart_policy = "never"`). `3pm start --help` is the authoritative
list; the units are identical. The ones whose spelling differs:

| CLI | Field |
|---|---|
| `3pm start <script> -- a b` | `script`, `args = ["a", "b"]` |
| `3pm start "npm run dev"` | `script = "npm run dev"` |
| `--interpreter none` | `interpreter = "none"` |
| `--interpreter-args "--inspect"`, `--node-args` | `interpreter_args = ["--inspect"]` |
| `--env KEY=VALUE` | `[apps.env]` (with a file, `--env <name>` picks a profile) |
| `-i N` | `instances` |
| `-o` / `-e` / `-l` | `out_file` / `error_file` / `log_file` |
| `--time` | `log_date_format = "%Y-%m-%dT%H:%M:%S"` |
| `--no-autorestart` | `restart_policy = "never"` |
| `--watch` / `--watch=PATH` | `watch = true` / `watch = ["PATH"]` |
| `--user NAME` | `user` (`--user ""` hands the app back to the daemon's account) |
| `--persist false` | `persist = false` |

</document>

<document url="https://3pm.edgecr4ft.net/docs/features.md" title="Features">

# Features

User-facing features, one section each: what you see, how to get it, and
where it comes from. Internals live in
`viking://resources/3pm-docs-archived/architecture/` (OpenViking).

## Start a command from your PATH: `3pm start npm -- run dev`

`3pm start <target>` takes a file (relative to the cwd, or `--cwd`) or,
when there is no such file and the target has no `/`, a command found on
your PATH, as in pm2: `3pm start npm -- run dev`, `script = "npm"` in an
ecosystem file, `pm.start({script: 'npm', args: 'run dev'})` in
`@3pm/api`. The app is named after the command (`npm`). A file in the cwd
wins over PATH, and a miss says where it looked:
`script 'npm' not found (looked at <cwd>/npm, then PATH)`.

**A managed app first.** Before any of that, a bare word — no `/`, no
blank, not a config file, no `-- args` after it — that names an app the
daemon manages, one of its ids, or `all`, restarts it as pm2 does:
`3pm stop api` then `3pm start api` brings `api` back with the
configuration it has (its ecosystem file's `interpreter`, env, …), not a
fresh one read off the cwd. `start` flags typed alongside are folded in
like `3pm restart api [flags]`, except that `--watch` alone sets watch
mode instead of toggling it. The managed name wins even over a file of
that name in the cwd; `./api` always means the file, and
`3pm start api -- …` always (re)defines the app from a script. Code:
`app_word`/`managed_target` in `crates/threepm-cli/src/commands/start.rs`.

**How it works.** The lookup happens in the CLI, against the shell you
typed the command in: the daemon's environment froze when it was
spawned. The app runs the PATH entry itself
(`~/.nvm/versions/node/v25.2.1/bin/npm`), not the file it links to, so an
upgrade that moves that file does not break it. The interpreter is
inferred from the linked file's extension (`npm-cli.js` → `node`,
resolved to an absolute path), so `#!/usr/bin/env node` is never looked
up in the daemon's PATH. A command whose target has no known extension
runs by its shebang, in the daemon's environment; `--interpreter`
overrides it. An interpreter given as a path (`--interpreter /bin/sh`,
`.venv/bin/python`) is kept as given, links included: BusyBox and venvs
depend on the name they are called by. Code: `crates/threepm-cli/src/commands/lookup.rs`,
`crates/threepm-cli/src/commands/target.rs`.

## Start a command line: `3pm start "python app.py"`

A target with a blank in it that is not a file runs under a shell, as in
pm2: `3pm start "npm run dev"` is `bash -c "npm run dev"` (`sh -c` when
your PATH has no bash). The same in a file (`script = "npm run dev"`, or
pm2's `cmd`/`exec`/`command`) and in `@3pm/api` (`{script: 'npm run
dev'}`). The app is named after the command's words: `npm run dev` →
`npm-run-dev`, `python ./src/app.py --port 3000` →
`python-src-app.py-port-3000` (a character a name refuses becomes `-`,
the `-`/`.` at a word's edges go, 128 bytes at most, `app` when nothing
usable is left); `--name` overrides it. The command carries its
arguments inside the quotes: `-- args`, `args`, `--interpreter` or
`--interpreter-args` beside it are errors, where pm2 drops them
silently. A file whose name has a blank still wins, and a command that
merely ends like a config name (`"node x.js --config app.json"`) stays a
command. `stop <file>` and `--only` find such an app by the name its
command implies. Code: `crates/threepm-cli/src/commands/target.rs`,
`threepm_core::process::app_name_from_command` (mirrored in
`packages/node/lib/spec.js`).

## Interpreter: `none`, its own flags, Python's output

- **`--interpreter none`** (`interpreter = "none"`, pm2) runs the script
  itself, by its shebang, whatever its extension says. On `3pm restart
  app --interpreter none` it clears the interpreter the app had, and in
  an app it beats one set in `[defaults]`. A script run without an
  interpreter must be executable: the CLI says so before anything starts
  (`… is not executable and has no known interpreter — chmod +x it or
  pass --interpreter`), for a file's apps as for the command line — a
  file's app used to become an `errored` spawn instead.
- **`--interpreter-args "--inspect --max-old-space-size=64"`** (pm2's
  `--node-args`; `interpreter_args` in a file, or pm2's `node_args` /
  `interpreterArgs`: a string split on blanks, or a list) go between the
  interpreter and the script: `node --inspect app.js`. `""` clears them
  on a restart; without an interpreter they are an error. `3pm show` has
  an `interpreter args` row. They never leave the host over the fleet
  link (they may carry `--inspect=host:port`).
- **Python's output is not buffered.** Python block-buffers stdout when
  it is a pipe, so a `print()` without `flush()` reached the log 8 KiB
  late. The daemon sets `PYTHONUNBUFFERED=1` (pm2 does too) for an app
  whose interpreter's file name starts with `python`, that runs a `.py`
  script by its shebang, or that is a quoted command line whose first
  word is such a python (`3pm start "python3 app.py"`, run by `bash
  -c`) — at spawn time, under the app's `env`
  (`PYTHONUNBUFFERED = ""` there turns it off), never stored in the spec,
  so it follows the interpreter the app runs now.
- pm2's `exec_interpreter` is accepted for `interpreter`.

Code: `threepm_core::spec::interpreter`,
`crates/threepm-daemon/src/spawner.rs` (`argv`, `implicit_env`).

## Config formats: YAML, JavaScript, stdin, the current directory's file

- **Formats.** Every command that takes a config file (`start`,
  `validate`, `apply`, `runtime`, `stop|restart|delete <file>`) takes
  `ecosystem.toml` (native), `.json`, `.yaml`/`.yml` and pm2's
  `*.config.js`, `*.config.cjs`, `*.config.mjs`. JSON, YAML and
  JavaScript also take pm2's other shapes: a bare list of apps, or one
  app alone (an object with a `script` and no `apps`). pm2's `deploy`
  section is ignored, like any unknown top-level key.
- **JavaScript configs are code**: the CLI evaluates them with the
  `node` of your PATH, as pm2 does — in your cwd and environment
  (`env: { PORT: process.env.PORT }` reads your shell's), stdin closed,
  10 s and 16 MiB of output at most. `module.exports`, `export default`
  and named exports (`export const apps`) all work; functions vanish (the
  export crosses as JSON). `restart --update-env` evaluates the file
  again. Without node: `ecosystem.config.js is JavaScript: evaluating it
  needs node in PATH`; a config that throws shows node's error.
- **stdin.** `3pm start -` (and `3pm validate -`) read the config piped
  in: JSON when it opens like pm2's (`{`, or `[` then `{`/`]`), TOML
  otherwise. Relative paths resolve against your cwd. Refused when stdin
  is a terminal, over 8 MiB, and by `apply` (`--prune` needs the file an
  app came from). Such an app has no config file to read again:
  `restart --update-env` says so.
- **No target.** `3pm start`, `3pm validate` and `3pm apply` alone use
  the ecosystem file of the current directory, the first found of
  `ecosystem.toml`, `.json`, `.yaml`, `.yml`, `.config.js`, `.config.cjs`,
  `.config.mjs` (`Using ecosystem.toml`; the others present get a
  warning). None: `no target given, and no ecosystem file in <cwd>
  (looked for …)`.

Code: `crates/threepm-cli/src/config/{formats,js,input}.rs`; YAML via
`serde_norway` (the maintained fork of `serde_yaml`).

## Restart policies

**What you see.** By default every unrequested exit respawns the app
(`always`, pm2 behavior). With `on_failure`, a clean `exit 0` parks the app
in `stopped` — `3pm ls` shows `stopped`, `exit code 0`, no restart counted —
while a non-zero exit, a signal death, or a failed spawn respawns it with
the usual restart pacing and crash-loop cutoff. `never` parks the app on
any exit.
`3pm show` prints the active policy in its `restart policy` row.

**How to get it.**

```console
$ 3pm start job.sh --restart-policy on-failure
```

```toml
[[apps]]
name = "job"
script = "./job.sh"
restart_policy = "on_failure"   # always | on_failure | never
```

pm2's `autorestart = false` / `--no-autorestart` still work and mean
`never`; giving both a policy and a contradicting `autorestart` is a config
error. A `dump.json` written before this field existed decodes its
`autorestart` bool into the matching policy.

**Where it comes from.** `RestartPolicy` in `threepm-core::spec`; the
decision is `restart_policy.restarts_after(exit_code)` inside the pure
policy's `after_death`, so user `stop`/`restart`/`delete`, watch restarts
and the memory guard (which all park an intent before the exit) are never
gated by it.

## Restart pacing: instant by default, backoff on demand

**What you see.** A crashed app respawns at once (pm2 behavior), so a
crash loop runs through its 16 unstable exits in a moment and `3pm ls`
shows `errored` in red, with the cause below the table. `↺` counts the
respawns. Two knobs pace the respawns, both pm2's:

```console
$ 3pm start api.js --restart-delay 500                 # fixed 500 ms pause before each restart
$ 3pm start api.js --exp-backoff-restart-delay 100     # 100 ms, ×1.5 per quick crash, capped at 15 s
```

```toml
[[apps]]
name = "api"
script = "./api.js"
restart_delay = 0                 # ms; fixed pause before every crash restart
exp_backoff_restart_delay = 100   # ms; > 0 switches exponential backoff on
backoff_factor = 1.5              # ┐ tuning, only with exp_backoff_restart_delay:
backoff_max = 15000               # │ growth per quick crash, cap, and the uptime
backoff_reset_after = 30000       # ┘ after which the delay resets to the first one
```

With backoff on, the app sits in `waiting restart` (yellow) between two
attempts; the delay resets to the first one once an incarnation stays up
`backoff_reset_after`. The 0.1 spelling `backoff_initial` /
`--backoff-initial` is still accepted. Neither knob changes the crash-loop
cutoff: `errored` still comes after `max_restarts` consecutive exits under
`min_uptime`, however they are paced.

**pm2 parity.** Same option names (`restart_delay`,
`exp_backoff_restart_delay`, `--exp-backoff-restart-delay`), same growth
(×1.5, capped at 15 s), same `waiting restart` status between attempts,
same default (instant). Two deliberate differences: pm2 only forgets the
grown delay on a manual restart/reload, so an app that crashed once at
boot keeps a long delay for its next crash hours later — 3pm resets it
after `backoff_reset_after` (30 s) of stable uptime; and pm2 hard-codes
factor and cap where 3pm exposes `backoff_factor` / `backoff_max`. When
both knobs are set, pm2 lets `exp_backoff_restart_delay` win; 3pm starts
the backoff at the larger of the two.

**Where it comes from.** `threepm-core::backoff::next_delay`: the fixed
`restart_delay_ms` unless `StartSpec::exp_backoff_enabled()`
(`backoff_initial_ms > 0`, the unchanged wire name), then the pm2 growth
rule. The policy's `after_death` turns it into one `SetRestartTimer`.

## Crash diagnostics: killing signal and stderr tail

When an app dies, 3pm remembers *why* — not just the exit code:

- **`exit_signal`** — the POSIX name of the signal that killed it
  (`SIGSEGV`, `SIGKILL`, `SIGABRT`…). A segfault, an OOM kill and an
  `exit(0)` all have no exit code; the signal is what tells them apart.
- **`stderr_tail`** — the last 30 lines the app wrote to stderr before
  dying (the panic, the stack trace), so you don't have to open the error
  log and find the right spot.

Both sit next to `exit_code` and are cleared by the next successful spawn,
so a running app never shows a stale death.

```console
$ 3pm show worker
┌───────────────────┬────────────────────────────────────┐
│ status            ┆ stopped                            │
│ exit signal       ┆ SIGSEGV                            │
│ name              ┆ worker                             │
│ instance          ┆ 0                                  │
│ pid               ┆ N/A                                │
│ cpu               ┆ 0.0%                               │
│ mem               ┆ 0b                                 │
│ …                                                      │
└───────────────────┴────────────────────────────────────┘
last stderr before exit (2 lines):
  thread 'main' panicked at src/main.rs:42
  called `Option::unwrap()` on a `None` value
```

`3pm show` prints the signal as a row (or `exit code` when the app exited on
its own) and the stderr tail under the table. In machine-readable form:

```console
$ 3pm ls --json | jq '.[] | {name, exit_code, exit_signal, stderr_tail}'
{"name":"worker","exit_code":null,"exit_signal":"SIGSEGV","stderr_tail":["…"]}
```

The same context rides the `exit` lifecycle event on the bus, which
`3pm logs --json` now forwards — an agent or GUI following the stream gets
the cause of death in the same line that announces it:

```json
{"event":"process","id":2,"name":"worker","kind":"exit","exit_signal":"SIGSEGV","stderr_tail":["…"]}
```

Bounds: 30 lines, 1 KiB per line, 4 KiB per process — enough for a stack
trace, small enough that the whole process table stays cheap to ship. The
tail is captured before the exit is reported (3pm waits, briefly, for the
stderr pipe to drain), so the last line — usually the one that matters — is
there. `exit_signal` is distinct from `last error`: the latter is 3pm's own
diagnostic when the app could not even be spawned (missing file, bad cwd).

## Orphans: `3pm ls` names what a dead daemon left running

**What you see.** Kill the daemon uncleanly (`kill -9`, OOM) and its apps
live on, unmanaged: the next daemon starts empty, `3pm ls` shows an empty
table, and the port you are staring at is still taken. Now the table is
followed by a warning that names the survivors:

```
[3PM] 1 process from a previous daemon still running unmanaged: api (pid 4321)
[3PM] They keep their ports. Run `3pm resurrect` (or `3pm start <name>`) to reconcile them.
```

The label is the pid file's stem under `pids/` (`api`, `worker-2` for
instance 2); past six survivors the list is elided with `+N more`.

**How it works.** Every spawn records `pid + process start time` in
`pids/<app>.pid`. `ls` asks the daemon (`orphans` request) for a read-only
scan of that directory: a file no managed row owns whose identity is still
alive is an orphan; a dead or reused pid is unlinked on the way. The scan
runs off the actor thread and the result is re-filtered against the rows
managed at reply time, so a `start` racing the scan is never reported. It
only reports: `resurrect` and `start` are what SIGKILL the orphan's group
and start a fresh copy (never two copies on one port). `ls --json` is
unchanged (a bare array); an older daemon answers `unknown_request` and
`ls` prints nothing extra.

## Watch mode: restart on file change

`--watch` turns 3pm into a dev loop: edit, save, the app restarts.

```console
$ 3pm start server.js --watch --ignore-watch '*.log' --watch-delay 300
$ 3pm ls
┌────┬────────┬──────┬────────┬───┬────────┬──────────┬──────┬────────┬───────┐
│ id ┆ name   ┆ pid  ┆ uptime ┆ ↺ ┆ status ┆ watching ┆ cpu  ┆ mem    ┆ ports │
╞════╪════════╪══════╪════════╪═══╪════════╪══════════╪══════╪════════╪═══════╡
│ 0  ┆ server ┆ 4021 ┆ 2s     ┆ 0 ┆ online ┆ enabled  ┆ 0.0% ┆ 42.1mb ┆ 3000  │
└────┴────────┴──────┴────────┴───┴────────┴──────────┴──────┴────────┴───────┘
```

- **What is watched**: `--watch` alone is the working directory,
  recursively; `--watch=src --watch=config` names specific dirs or files
  (relative to the cwd, must exist). Ecosystem files take `watch = true`,
  `watch = "src"` or `watch = ["src", "config"]`.
- **What is ignored**: dotfiles/dot-dirs (`.git`, `.env`) and
  `node_modules`, always. `--ignore-watch GLOB` / `ignore_watch = [...]`
  *adds* patterns: a bare name or glob (`*.log`, `tmp`) matches any path
  segment; a path (`logs/`, `src/generated`) is anchored at the watched
  root. An ignored directory hides its whole subtree.
- **When it restarts**: after a quiet period of `--watch-delay` ms
  (floor 100ms), measured from the *last* change — a save through a temp
  file or a `git checkout` is one restart. All instances of the app
  restart together (graceful stop, then respawn). Changes that land while
  the app is mid-restart are not queued.
- **How**: OS events (inotify on Linux, FSEvents on macOS, kqueue on
  BSD). If the native watcher can't be set up — typically inotify's
  `max_user_watches` on a big tree — 3pm polls every 2s instead and says
  so under the `ls` table and in `show`:

  ```console
  ↳ server watch: native watcher unavailable (No space left on device (os error 28)); polling every 2000ms
  ```

  `--watch-poll MS` / `watch_poll` forces polling from the start (NFS,
  container bind mounts, anywhere events don't arrive).
- **Toggling**: `3pm restart server --watch` flips watch mode (off → the
  cwd, on → off) and restarts, like pm2. `3pm start server.js --watch` on a
  running app *sets* it (the spec is replaced). `stop` turns watch off,
  as pm2 does: a later `start <name>` or `restart` runs unwatched until
  `--watch` (or a file that sets it) asks again; `save`/`resurrect`
  keep the current setting.

`3pm show` lists the roots, the effective ignore list and the delay:

```console
│ watch & reload    ┆ enabled (/srv/app)                                 │
│ ignore watch      ┆ .*, node_modules, *.log                            │
│ watch delay       ┆ 300ms                                              │
```

Differences from pm2, on purpose: one watcher per app rather than one per
cluster instance; `ignore_watch` extends the defaults instead of replacing
them (pm2's `ignore_watch: ["logs"]` silently re-watches `node_modules`);
`watch_delay` is milliseconds on the CLI too (pm2: seconds on the CLI,
ms in files); and a watcher that could not start is visible, not a line
in the daemon log.


## Logs: destinations, timestamps, hot rotation

Every instance gets its own files (`logs/<app>-out.log`, `-error.log`;
`<app>-<i>-…` from instance 1) holding the app's output byte for byte.
The knobs are pm2's, with the sharp edges filed off:

```console
$ 3pm start server.js --time                 # "2026-08-25T16:43:54: line" in the file
$ 3pm start server.js -l /var/log/app.log    # one merged file for both streams
$ 3pm start worker.js -i 4 -o '/var/log/w/{name}-{instance}.out' -e NULL
```

- `{name}`, `{instance}`, `{namespace}` expand in custom paths. A custom
  path *without* `{instance}` on a multi-instance app is infixed
  `-<instance>` (instance 0 keeps the path as written) and `3pm start`
  says so — sharing one file between instances is never what you meant.
- `NULL` drops the file but not the stream: `3pm logs` still follows it.
  `--disable-logs` drops file, stream and crash tail (the pipe is still
  drained so the app never blocks).
- `--attach` on `start`, `restart` and `stop` (pm2): after the action
  and the table, stream the affected apps' logs live — no history, Ctrl-C
  to leave, clean exit when the daemon shuts down. Conflicts with
  `--json` (`3pm logs --json` is the machine stream).
- `log_date_format` is validated at `3pm start` (a `%Q` fails there, not
  in every log line); there is no CLI-only default, so an ecosystem app
  and a flag-started app behave the same.

**Rotation happens while the app runs.** The daemon counts what it
writes and rotates the file the moment it crosses the threshold, under
the same lock every writer shares — so the line that crossed is in `.1`,
the next one is in the fresh live file, and a follower on the bus sees a
`log_rotated` event between the two. Nothing is ever lost to a restart
that never came.

```console
$ 3pm logs api --size
0|api | out       3.1 KB  /home/u/.3pm/logs/api-out.log  (+2 rotated, 38.2 KB)
0|api | err         0 B   /home/u/.3pm/logs/api-error.log
$ 3pm logs api --lines 500 --rotated     # history continues into .1, .2.gz…
$ 3pm rotate api                         # now, regardless of size
$ 3pm flush                              # empty every file in place
$ 3pm reloadLogs                         # after an external mv (logrotate create mode)
```

Policy layers, most specific first: per-app `log_rotate_*`, then the
daemon's `THREEPM_LOG_MAX_SIZE|_FILES|_DAYS|_COMPRESS`, then `[logs]` in
`~/.3pm/config.toml`, then 20 MiB / 5 files / 14 days / no gzip. `"0"`
switches rotation off. With `compress`, `.2` and older are gzipped and
`.1` stays plain. An hourly sweep applies the count/age limits to files
that rotate rarely. `3pm internals` reports the effective policy, open
files, bytes written and rotations.

Reading has its own timestamps: the bus carries `ts` (arrival, epoch
ms) on every line, so `3pm logs --timestamp[=FMT]` stamps a stream
without touching the files, `--format` gives logfmt (`ts= app= id=
instance= type= message=`), and `--json` gives the wire events — tail
and stream in one NDJSON schema. `3pm logrotate --print` renders a
logrotate(8) drop-in for hosts that centralize rotation there.


## Log shipping (`3pm agent`, `[logs]` in link.toml)

**What you see.** A linked node sends its logs to the fleet backend by
default: no setup, no per-app setting. The backend keeps the counts of
every kind of line and the lines that carry news, and answers them with
`logsQuery` (see the backend's `FEATURES.md`). `3pm agent status` (or
`3pm link status`) has a `logs` row:

```
logs         24 sources (8 app, 15 file, 1 journal), 451 lines read, 33 shipped, 33 templates; batches 16 ok, 0 retried, 0 dropped
```

It adds `; skipped …` (sources not read, by reason), `; lines dropped …`
(by reason: `budget`, `lag`, `ring`, `stale`) and `; last error: …` when
there are any. The running agent writes it to `<home>/agent/logs/status.json`.

**What is shipped.**

- **The 3pm apps' output**, read from the daemon's bus (a second
  subscription, its own cursor: a flood lags only it). ANSI stripped,
  lines cut at 8 KiB. `--disable-logs` apps publish nothing, and the
  agent's own app is never shipped. Every OS.
- **The system journal** (Linux), entries at `info` or worse
  (`host_level`), kernel lines and Docker containers using the `journald`
  driver included. Read by one `journalctl --follow --output=json` child;
  first start from now, then from the cursor of the last batch the
  backend accepted. Entries older than 24 h are dropped on read.
- **Other programs' log files** (Linux), found by their writer: once a
  minute the host agent lists the regular files held open for writing
  with `O_APPEND` (nginx, mysqld, a Python bot writing in its home,
  dockerd's `<id>-json.log`), with the writer's pid, name, systemd unit
  and container, plus `dpkg.log`, `apt/history.log` and
  `unattended-upgrades/*.log` (written and closed at once). When the
  root-installed `automonit` module runs, its `logs` action (asked every
  5 min) adds what a service says about itself: nginx's and Apache's
  configured logs, MySQL's `log_error`, PostgreSQL's log directory,
  MongoDB's `systemLog.path`, Redis's `logfile`, with their format
  (`nginx-combined`, `postgres-prefix`, `mongo-json`…).
  The host agent lists 2 000 files at most, ranked by the writer's real
  uid: root's and the fixed list first, then system accounts' (uid
  below 1 000) outside the daemon's cgroup, then the room left shared
  evenly between users, subordinate uids (100 000 and up) counting as
  one (one user holding thousands of files cannot push `/var/log` or
  another user's files out). A pass reads at most 4 096 descriptors per
  process, root's processes first, and stops after 20 s with what it
  found; one held up by a hung filesystem is served when it ends.
  A file is tailed from its end at first sight (no history), follows
  rotation and `copytruncate`, and is parked after 10 min of silence;
  512 files at most.

**What is not.** The macOS unified log (on a Mac only the apps' output
leaves). Files the daemon's user cannot read: the agent runs as the
daemon's user, an `EACCES` marks the source skipped, and the next pass
(60 s) tries again — a user added to `adm` starts shipping without a
restart. And what the host agent's verdict keeps back:

| verdict | files |
|---|---|
| `duplicate_journal` | written by `rsyslogd`, `syslog-ng`, `systemd-journald` (`/var/log/syslog`, `auth.log`, `kern.log`: the journal, line for line) |
| `duplicate_bus` / `own` | the 3pm apps' log files (already on the bus), `3pm.log`, `hostagent.log` (by path, whoever holds them, a `3pm update`'s new daemon included: the home's `3pm.log`/`hostagent.log` are `own`, anything under its `logs/` or the `logs/` of a home it hosts (`[users] allow`, as the daemon accepts it) is `duplicate_bus`) |
| `binary` | a NUL byte in the first 4 KiB |
| `private` | shell and REPL histories (`.histfile`, and a `history` anywhere under a dot-directory), `.lesshst`, `lesshst`, `.viminfo`, `*.shada`, anything under `.ssh/` or `.gnupg/` |
| `unknown` | a verdict this build does not know |

Only `ship` is read.

**How it is reduced on the node.** Every line goes through, in order:
redaction (secrets replaced by `<REDACTED>`), template mining (Drain: a
line becomes a template with its variables masked, ids per source, kept
across restarts), then the policy. Always sent: each template's text when
it appears or changes, and its count per 5-minute window. A raw line
only when it is news (`why`): a template never seen before (`new`, at
most 30 a minute per source), warning or worse once per template per hour
(`error`), an important template back after 10 min of silence (`back`),
a rate spike (`spike`) or a return after a long gap (`return`), and the
30 s before a new error (`context`). Then a budget per source, 4 KiB/s
with a 256 KiB burst; what is over it is dropped and counted. A batch
leaves every 10 s (gzip NDJSON, `POST /api/buckets/{cluster}/logs`,
never on the WebSocket); while the backend is away, lines wait in a 4 MiB
memory ring. There is no disk spool: the files and the journal are the
spool, and their positions advance only when the backend answers `200`,
so a line read but not accepted before a stop is read again (and stored
once). On the PoC's corpora, 0.4 to 8 % of the raw bytes left a busy
host, 14–20 % a quiet journal.

**Who decides.** Both ends. The backend says `welcome.logs {apps, host,
files}` when it has an events store; without it the agent ships nothing.
The host has the last word in `<home>/agent/link.toml`, every key
optional (a file written before logs reads as the defaults):

```toml
[logs]
apps    = true      # the 3pm apps' output
host    = true      # the journal
files   = true      # other programs' log files
host_level = "info" # journal ceiling: debug, info, notice, warning, err, crit
exclude = []        # globs (`*`, `?`) on app names and file paths
```

A source flows only when both say yes. A misspelt `host_level` closes to
`crit`, a malformed `[logs]` ships nothing (the agent still runs) — an
unknown key counts as malformed, so a misspelt `file = false` closes
every switch instead of leaving `files` on —, and
`3pm link --force` keeps the section. The agent reads it at start:
`3pm restart agent` after an edit.

**Security notes.**

- Redaction is a pattern list: URL passwords, `Authorization` and cookie
  headers, `password=`/`token:`/`"secret":` pairs, command-line
  passwords (`--password`, `mysql -p`, `sshpass -p`, `curl -u`), bearer
  and Basic credentials, JWTs, PEM private keys, known token shapes (3pm
  `3pmu_`/`3pma_`/`3pmk_`, GitHub, Slack, Stripe, npm, AWS, Google), crypt
  hashes, card numbers. Public ids (`3pmn_`, `3pmc_`), numeric token
  counts and plain words after `password` (`password for root`) are kept.
  A secret in a shape the list does not know is shipped, and so is one
  split over lines: redaction reads a line at a time (a YAML env entry
  with `- name: DB_PASSWORD` on one line and `value:` on the next ships
  its value).
- A file is opened `O_NOFOLLOW`, then checked by `fstat` against the
  `(dev, inode)` the host agent declared: a symlink swapped in, or a
  writer in a container naming a host path, gets nothing read. A
  rotation successor must be a regular file of the same owner.
- Only the root-installed `automonit` module is asked (an app a hosted
  user names `automonit` is not), and a file automonit adds alone must
  look like a log (`/var/log`, a `log`/`logs` directory, `*.log`), sit
  outside `/etc`, `/root` and the like, have one link and the owner it
  declared.
- `private` files are never read, whoever writes them.

**Stop.** At a stop the agent closes its open windows and tries a last
batch within 5 s; its row's stop grace is 10 s (`kill_timeout_ms` in the
agent's `module.toml`, raised on existing rows by `3pm update`).

**Where it comes from.** Wire: `threepm-link::logs`,
`docs/backend-agent/link-protocol.md` §6b. Files by writer:
`crates/threepm-daemon/src/hostmetrics/logfiles.rs` (`Request::LogSources`).
The pipeline: `modules/agent/src/logs/` (`docs/backend-agent/agent.md`,
"Logs"). automonit's side: `3pm-automonit/docs/logs.md`.


## Interactive apps: `tty = true` and `3pm attach`

Some programs want a real terminal: `claude`, `codex`, a REPL, `vim`, a
shell. Started with `--tty`, an app runs on a pseudo-terminal the daemon
holds, and `3pm attach` connects any terminal to it, as tmux would —
with everything 3pm already does (restart policy, logs, crash tail,
hosted users, `3pm update` without a restart) unchanged.

```console
$ 3pm start claude --tty                       # or tty = true in ecosystem.toml
$ 3pm attach claude                            # its screen appears, your keys go to it
[3PM] attached to claude (pid 4242, 120x40) — Ctrl-Z detaches
…
[3PM] detached
$ 3pm attach claude --peek                     # the screen as text, then out (--json: a document)
$ printf 'ls\n' | 3pm attach sh                # no terminal: stdin in, the app's bytes out
$ 3pm attach claude --read-only                # watch: no keys, no resize
```

**The app.** `isatty` is true on 0, 1 and 2; the app leads a session of
its own with the terminal as its controlling tty (`setsid` +
`TIOCSCTTY`, `login_tty(3)`'s recipe), so `3pm stop`/`kill` signal its
group as before. The terminal starts at 120×40 with `stty sane`
settings plus `IUTF8` and `VERASE` = DEL; `TERM=xterm-256color` and
`COLORTERM=truecolor` are set unless `env` names them. It follows the
size of the last client that attaches or resizes and may type (bounded
to 2..=1000 each way, resizes coalesced over 50 ms).

**What the daemon keeps.** One screen per `tty` app, with 1 000 rows of
history, fed by everything the app writes, whether a client is attached
or not. It answers the questions an app asks its terminal (cursor
position, `CSI c`/`CSI > c`, XTVERSION, window size, DECRQM, the
default colours `OSC 10/11/12`), so codex and claude start at full speed
with nobody watching. The screen outlives an exit: attaching to an app
that stopped shows its last screen, and the client waits for the next
incarnation (`restarted`, then a fresh screen).

**What a client gets (`rendered`, the default).** Screens the daemon
draws, never the app's bytes: each client has a mirror of its own
terminal and receives the difference, immediately when it is idle, else
every 16 ms, and not in the middle of an app's synchronized update
(`?2026`: held until it ends, 100 ms at most). Rows that scroll off the top of the app's main screen land in
**your terminal's own scrollback** — scroll up through a Claude Code
conversation with the mouse wheel — and on attach, the history held so
far comes first. The alternate screen is entered and left when the app
does (vim leaves your shell's screen as it was); title and bell are
relayed, the cursor shape too. `--raw` relays the app's bytes as written
instead, for what the rendered view does not model (at your terminal's
risk: its answers to the app's queries go back too). `3pm attach
--peek --json` reports `unmodelled`, the count of sequences the screen
did not understand: above 0, `--raw` shows more.

**Leaving.** `Ctrl-Z` detaches, always, and is never sent to the app
(in `--raw` also its kitty-keyboard and modifyOtherKeys spellings, which
an app may turn on). `SIGTERM`, `SIGHUP`, `SIGINT` and `SIGQUIT` sent to
the client detach too. Whatever the way out, the terminal gets back its
termios, scroll region, colours, cursor keys and keypad, mouse modes,
bracketed paste, focus events, keyboard protocols, a visible cursor of
the default shape, and its main screen. Exit codes: 0 detached, 2 no
such app or not a `tty` app, 3 the app was deleted (or re-specified
without `tty`) or the daemon went away, 1 anything else (a client cut
off as too slow, `input_backlog`).

**Without a terminal (pipe mode).** When stdin or stdout is not a
terminal, stdin goes to the app until its end, the app's bytes come out
unrendered, and the command ends once the app had its input and fell
silent for 150 ms (500 ms after it had it at most: a TUI animates
forever), or when it exits. A large stdin goes at the app's pace.
`--paste` sends stdin as one bracketed paste when the app turned
pasting on (a prompt takes several lines as one message).

**Logs.** A terminal has one output stream: the app logs to one file
(`<app>.log`, or the `log_file`/`out_file` you name; `error_file` is
refused), stripped of escape sequences, cut into lines, a move to the
right after text read as a space — `3pm logs claude` reads as text. The
crash tail is the last output (`3pm show`: "last output before exit").
`disable_logs` drops the log of a TUI that redraws too much; the screen
is kept either way. Keystrokes are never logged; what the app displays
is, like any app's output.

**Across `3pm update`.** The terminal crosses the daemon's `execve` like
the pipes do: the app keeps its pid, its terminal and its screen (main
and alternate, ≤ 48 KiB; beyond that the app is asked to repaint by a
resize), and an attached client re-attaches by itself (`[3PM] the
daemon was updated: re-attached`).

`3pm show` has a `tty` row (`yes (1 viewer)`, or how to attach), and
`3pm ls --json` a `tty_viewers` count. The Node SDK has no `attach`
(a terminal session is not a script's call).

**Limits, deliberately.**

- **A `tty` app dies with a daemon that crashes**: the kernel hangs up
  the terminal (`SIGHUP`) when its last master closes. A clean
  `3pm update` keeps it; a future keeper process holding the master is
  the remedy.
- `Ctrl-Z` never reaches the app from a client's keyboard (in pipe mode
  a `0x1A` in stdin is sent as is).
- In `rendered` mode these do not pass: the kitty keyboard protocol and
  modifyOtherKeys (Shift+Enter arrives as Enter in Claude Code), focus
  events (`?1004`), `?2031` colour-scheme reports, OSC 8 hyperlinks,
  OSC 52 clipboard, kitty/sixel graphics. `--raw` passes them.
- The pty has one size: a client smaller than it sees a cut screen
  (the size follows the last client that attached or resized, not the
  smallest; `--read-only` clients never resize).
- A client that stops reading for more than 10 s (a suspended
  terminal) is cut off; so is one that lets 32 lifecycle notes pile up
  (`too_slow`), or types faster than the app reads (256 KiB,
  `input_backlog`).
- `3pm update` does not carry the history (the next rows scroll into
  the client's scrollback as before) nor origin mode (`DECOM`).
- Version skew: a new CLI asking an 0.37 daemon for `--tty` gets an app
  on pipes (the field is unknown there, and ignored); `3pm attach`
  then says to run `3pm update`. Handing a daemon with a `tty` app over
  to an older image is refused at the pre-check (handoff manifest
  version 2) and `3pm update` falls back to a cold restart; any row with
  `tty`, stopped ones included, asks for version 2 (the conservative
  choice).
- First-run dialogs (claude's workspace trust, codex's update prompt)
  wait for a human to attach and answer them.
- Memory: about 3.7 MiB per `tty` app at 120 columns (the history),
  plus about 1 MiB per attached client (its mirror).

**Where it comes from.** Plan `3pm-rd/3pm-tty-attach-plan.md` (v2);
`threepm-core::{spec (tty), ansi, paths::log_targets}`; the wire in
`threepm-protocol::{message (Attach, Attached, NotTty), shared::attach,
live_io}` and `docs/protocol.md` (§ *Attach*); the daemon in
`crates/threepm-daemon/src/tty/` (`pty`, `screen`, `screen_saved`,
`render`, `viewer`, `writer`, `history`, `attach`),
`child_task_tty.rs`, `supervisor/{tty,pumps,adopt}.rs` and
`rpc_server.rs` (the connection takeover); `vt100` 0.16.2 patched in
`vendor/vt100/` (`PATCHES.md`); the CLI in
`commands/attach{.rs,/session.rs,/term.rs}` and `render/attach.rs`.
Tests: `crates/threepm-cli/tests/e2e_tty.rs` (real ptys: a test
terminal, `Ctrl-Z`, signals, pipes, peek, vim and a shell across
`3pm update`), `spawner_tty_tests.rs`, `tty/*_tests.rs` (the responder
table, exact history, a differential proptest of the rendered view
against a large terminal model), `supervisor/tests/tty{,_attach,_handoff}.rs`.

## Start order: `depends_on`, `start_order`, `start_delay`

```toml
[[apps]]
name = "db"
script = "./db.sh"
wait_ready = true            # db says when it is really up

[[apps]]
name = "api"
script = "./api.sh"
depends_on = ["db"]          # spawned only once every db instance is online
depends_timeout = 30000      # ms; 0 = wait forever (default 60000)

[[apps]]
name = "web"
script = "./web.sh"
depends_on = ["api"]
start_delay = 2000           # ms before the first spawn of a start/restart
start_order = 10             # tie-break among apps started together
```

`3pm start ecosystem.toml` orders the file into waves (a dependency
cycle or an unknown name is an error *before* anything starts) and
starts wave n+1 once wave n is online — `online` means past `wait_ready`,
so "started" means "ready", not "forked a millisecond ago" (OxMgr only
sorts the spawn order). `resurrect` does the same at boot.

The daemon enforces it for single starts and restarts too: an app whose
dependency is not online is **parked** in `launching` (no pid, `show`
says `waiting for: db`, `ls --json` carries `waiting_for`) and polled
every 500 ms. Stop and delete act on a parked row immediately; restart
re-parks it. After `depends_timeout` it starts anyway with a notice in
`last error` — a dead database must not keep the dashboard down
forever. `restart db` does not cascade to `api`.

`stop_exit_codes = [0, 3]` parks the app in `stopped` on those codes
whatever `restart_policy` says (a signal death has no code and never
matches). `3pm reset <app|all>` zeroes the ↺ counters without touching
the process.

`stop`, `restart`, `delete`, `reset` and `logs` take **several targets**
at once (pm2: `3pm delete 0 1 api`), mixing names, ids, `all` and
ecosystem files. `all` absorbs the rest of the list and duplicates fold,
so a row is never acted on twice. A target that resolves to nothing does
not cost the others: the batch acts on every target that does resolve,
notices each one that did not, and exits with the failure code at the
end. `logs` is the exception — an unmatched name is a notice and the
rest still stream; only an empty selection is an error.


## Config workflow: profiles, `validate`, `apply`, `--update-env`

```toml
[defaults]                     # every app starts from here
restart_policy = "on_failure"
log_rotate_size = "50M"

[[apps]]
name = "api"
script = "./server.js"
[apps.env]
NODE_ENV = "development"
[apps.env_production]          # pm2 sugar: env only, with --env production
NODE_ENV = "production"
[apps.profiles.staging]        # any field, with --env staging
instances = 2
[apps.profiles.staging.env]
NODE_ENV = "staging"
```

Resolution order: `[defaults]` → `[[apps]]` → `[apps.profiles.<name>]`
→ `[apps.env_<name>]`. Scalars from the later layer win, `env` merges by
key, lists (`args`, `watch`, `depends_on`, …) are replaced. `--env
<name>` selects the profile on `start`, `apply` and `validate`; a name
the file never defines is an error (pm2 silently ignores a missing
`env_production` — a typo would ship dev settings). `--only a,b` limits
any of the three to a subset (a dependency outside it must already be
managed). The chosen profile and the file path are recorded on the spec
(`show`: `config file`, `profile`; `ls --json`: `spec.source`) and
survive `save`/`resurrect`.

`3pm validate eco.toml` parses, resolves the profile and checks the
semantics without a daemon: unknown fields are **errors** here (a
did-you-mean each), plus per-app spec errors, duplicate names,
dependency cycles, and `depends_on` names outside the file (a warning;
`--check-managed` asks the daemon and makes it an error). Exit 1 on any
error; `--json` is the finding list.

`3pm apply eco.toml` reconciles the daemon with the file and is
idempotent: apps not managed are **created**, apps whose configuration
differs are **updated** (restarted with the new spec — the reason names
the changed fields), apps where only `instances` differs are **scaled**
(rows added or removed, survivors keep their pid; `instances = 0` is
compared as the daemon resolves it, one per CPU, so it never reads as a
change), stopped apps with the same configuration are **restarted**,
everything else is **unchanged** and keeps its pid. `--prune` deletes
managed apps that came from this file but are gone from it — an app
that is still in the file but outside `--only` is *not* gone, and an app
started by hand is never pruned (`--prune-all` does that too, and
refuses `--only`); `--dry-run` prints the plan and exits. `3pm start
eco.toml` keeps pm2's behavior — it always restarts — so habits and
scripts don't change; `apply` is the idempotent verb. Re-applying with
no `--env` an app that was applied with a profile drops the profile and
says so.

Loader rules shared by all three: `--only` is decided on the declared
names before anything resolves (a broken app outside the subset never
blocks; a broken selected app reports its own error); a name declared
twice is an error; env values may be bare numbers or booleans (`PORT =
3000`, stringified); a malformed `env_<name>` table is that app's error
only; `[defaults]` cannot carry `name`, `script`, `profiles` or
`env_<name>` (each is a notice, never silent). `validate` needs neither
a daemon nor a socket-capable `THREEPM_HOME` (only `--check-managed`
connects). Values expand `~` (leading) and `$VAR` / `${VAR}` in `cwd`,
`script`, `interpreter`, `args`, `watch`, the log paths and `env` values
(`$$` is a literal `$`); a variable that is not set is that app's error
— never a silent empty string. `stop`, `restart` and `delete` also
accept a file: every managed app it declares is acted on, an unmanaged
name is a notice.

`3pm restart api --update-env` re-reads the file the app came from
(same profile) and restarts it on the fresh spec, always. An app started
from the command line has no file: the error says so.

## Reconfigure on restart: `3pm restart <app> [start flags]`

pm2's attribute update. `restart` accepts every `start` flag; what you
type is folded into the app's stored configuration, and the app restarts
on the result:

```console
$ 3pm restart api --name web            # renamed in place, then restarted
$ 3pm restart 25 -i 4 --max-memory-restart 300M
$ 3pm restart web --env LOG=debug       # added to the environment, not replacing it
```

What the flags do not name is left exactly as it was — script, args,
interpreter, cwd, log destinations, the ecosystem file the spec came
from. `--env K=V` **merges** into the app's environment (pm2), every
other list-valued flag replaces. The new configuration is the app's from
then on: it is what `show`/`ls --json` report and what `save` persists.

The fold is **per app**, never per row: a spec belongs to a name that
owns N instances, so `3pm restart 25 --name web` (id 25 being one
instance of `api`) renames and restarts the whole app — same ids, same
instance numbers, restart counters kept, pid files and log files moved
along with the name (as `3pm rename` does, below). `-i N` reconciles
the count exactly like `start` does: missing instances are created,
surplus ones deleted, survivors restart on the new spec.

Refusals leave everything running as it was: a `--name` whose target
matched several apps (`restart all --name x` would collapse the fleet
into one name), a name another app already carries, a `--name` on an app
a module owns (`install`/`uninstall` match the manifest on that name —
every other flag applies to a module app as usual), and any patch that
cannot become a valid spec (`--kill-signal SIGNOPE`) — checked for every
matched app before the first one restarts.

`--watch` alone keeps pm2's meaning (toggle watch mode: off → the cwd,
on → off); `--watch=PATH` sets the watched paths like `start` does.
`--update-env` restarts on what the ecosystem file says and cannot be
combined with flags — the two ask for different specs.

## Rename in place: `3pm rename <app> <new-name>`

Renames an app **without restarting it**: same ids, same pids, same
restart counters and instance numbers — the process never sees a
signal.

```console
$ 3pm rename api web        # by name
$ 3pm rename 25 web         # by the id of any of its instances: the whole app
$ 3pm rename api web --json # {"affected": [...]}, under the new name
```

What carries the name follows it:

- **every instance** of the app (a spec belongs to a name, never to one
  row);
- **the pid files** (`pids/web.pid`, `web-1.pid`, …);
- **the log files named after the app** — the defaults
  (`web-out.log`, `web-error.log`, `web.log` with `merge_logs`, per
  instance) and a custom path that uses `{name}` — with their rotated
  generations (`.1`, `.2.gz`, …). The file is renamed under the open
  descriptor, so the app writes on into the moved file without a gap
  and `3pm logs web` shows its whole history. A custom path without
  `{name}` is not the name's and stays where it is;
- the name on the event stream (`3pm events`, the `Log` lines of the
  running process included) and in `ls`/`show`;
- watchers and cron restarts (re-armed under the new name; a cron's
  next run is recomputed).

What does **not** follow until the app's next spawn: the `name`
variable the running process was started with, an action call already
in flight, a terminal viewer already attached (its title). Other apps'
`depends_on`, and the ecosystem file the app came from, still say the
old name: `restart --update-env` then looks the new name up in the
file.

Refusals change nothing: a target matching several apps (`all`, a
pattern — refused by the CLI before it asks), a name another app
carries, a module's app (a module names its own apps), an invalid name,
and a log file already at the new name (a deleted app's history left
without `--flush`): nothing restarts to open a fresh file, so moving
onto it would clobber it — remove it first. A hosted user renames its
own apps and they stay its own. `3pm restart <app> --name <new>` takes
the same rename step, then restarts. Node SDK: `client.rename(target,
name)`.

## Event stream: `3pm events`, server-side filters, shutdown announce

**What you see.** One line per daemon event — lifecycle transitions
(with exit code / signal / stderr-tail size), log lines, rotations, lag
markers and the shutdown announce — filtered *by the daemon* before it
serialises anything, so a chatty app costs nothing to a follower that
did not ask for its output:

```console
$ 3pm events -f 'process:*' -p api          # lifecycle of one app only
12:34:56 process         3|api     online
12:35:10 process         3|api     exit code=1 signal=SIGSEGV stderr_tail=2 lines
$ 3pm events -f log:err -f log_rotated       # globs are repeatable
12:36:02 log:err         3|api     warn: slow query
$ 3pm events --json | jq .                   # raw wire events (NDJSON)
$ 3pm kill                                   # in another shell → the follower ends:
12:35:40 daemon_shutdown           reason=kill
[3PM] daemon shutting down (kill)
```

Event names: `process` (also `process:<kind>` — `start`, `online`,
`exit`, `restart`, `stop`, `delete`, `errored`), `log` (also
`log:<stream>` — `out`, `err`; the v1 spellings `log_out`, `log_err`
and `log_*` are accepted as patterns for one release), `log_rotated`,
`lagged`,
`daemon_shutdown`; `*`/`?` globs. `-p` scopes
app events to one name; daemon-level events always pass. `--raw` drops
the time column and chrome; `--timestamp FMT` changes it.

A follower (`events` or `logs`) exits **0** with the reason when the
daemon is stopped on purpose (`3pm kill`, SIGTERM, home removed) and
non-zero with "daemon connection closed unexpectedly" when the daemon
died without announcing — the difference a supervisor script wants.
`3pm logs` uses the same filter under the hood (selected ids + the kinds
its view shows).

**How to get it.** Nothing to configure: any client of `pub.sock` can
send one `EventFilter` JSON line right after connecting (`docs/protocol.md`,
*Subscription filter*); clients that send nothing get everything, as
before.

**Where it comes from.** `EventFilter` in `threepm-protocol::filter`
(pure, `*`/`?` matcher shared with watch ignores), `bus::stream_events`
(handshake + drain-on-cancel) and `Supervisor::begin_shutdown` (the
single announce) in `threepm-daemon`, `commands/events.rs` +
`render/events.rs` in the CLI.

## Application metrics and events (`3pm show`, `3pm events -f app`)

**What you see.** pm2's custom metrics and events, for any language and
any number of processes per app. A process of the app connects to the
control socket, names itself and reports; `3pm show` grows a section:

```console
$ 3pm show saw
 … the usual describe table …
 Custom metrics
┌───────────┬──────────────────┬────────────┬───────────┬──────────────────────┬──────┬─────┐
│ source    │ name             │ labels     │ kind      │ value                │ unit │ age │
├───────────┼──────────────────┼────────────┼───────────┼──────────────────────┼──────┼─────┤
│ cv-engine │ cv.xruns         │            │ counter   │ 12                   │      │ 0s  │
│ cv-engine │ cv.callback      │            │ histogram │ n=48210 p50≈1.2 p99≈4│ ms   │ 0s  │
│ renderer  │ http.requests    │ method=GET │ counter   │ 5321                 │      │ 1s  │
└───────────┴──────────────────┴────────────┴───────────┴──────────────────────┴──────┴─────┘
$ 3pm show saw --json | jq '.[0].app_metrics.metrics[] | .def.name'
$ 3pm events -f app
12:34:56 app             3|saw     [cv-engine] midi.port_lost warn ×3 MidiError: connection lost {"port":"ES-9"}
```

Three kinds — `gauge`, `counter` (cumulative, rates at read time) and
`histogram` (base-2 exponential buckets, quantiles at read time so a
fleet can add them up) — with up to three labels each, a unit and a
description. Events carry a name, a level, flat attributes, an optional
exception with its stack trace (machine paths stripped) and a
fingerprint; a crash loop is rate-limited per fingerprint and shows up
as `×N` on the next event that passes, never as a flood.

A process that makes series from what it finds (one per service a
monitoring module discovers) declares **families** — a name with no
series until a labelled child exists — **removes** a series when its
subject is gone (the daemon drops it after its two-minute TTL), and
reads its own **usage** to stay within the caps (Rust SDK 0.37;
`packages/rust/ROADMAP.md` for the other languages). The daemon keeps a
series' unit and description: it announces `THREEPM_CTL_CAPS=sticky_meta`
to its apps, and an SDK that reads it sends them once per connection.

**How to get it.** Nothing is injected into the child: the app embeds
an SDK — the same surface in four languages (`docs/app-metrics.md`):

```js
const { io } = require('@3pm/api');                 // Node / Electron (packages/node)
io.init({ source: 'worker', builtins: true });
io.counter('http.requests', { labels: { method: 'GET' } }).inc();
io.histogram('http.latency', { unit: 'ms' }).observe(12.5);
io.notifyError(err);
```
```python
import threepm_io as io                             # Python, stdlib only (packages/python)
io.init(source="cv-engine")
io.histogram("cv.callback.jitter", unit="ms").observe(0.42)
```
```cpp
#include <threepm_io.hpp>                           // C++17 header-only (packages/cpp)
auto late = threepm::io::instance().counter("host.late_callbacks");
late.inc();                                         // wait-free, fine on the audio thread
```
```rust
use threepm_io::{io, Init, Metric};                 // Rust, serde only (packages/rust)
io().init(Init::new().source("worker"));
io().counter("http.requests", Metric::new()).inc(1.0);
```

**Install.** The four SDKs are published on `https://3pm.edgecr4ft.net`
only, not on npm, PyPI or crates.io; `sdk/sdk.json` lists every version,
its files and their sha256:

```console
$ npm install https://3pm.edgecr4ft.net/sdk/node/3pm-api-0.64.0.tgz
$ pip install --index-url https://3pm.edgecr4ft.net/simple/ threepm-io
```
```cmake
FetchContent_Declare(threepm_io URL https://3pm.edgecr4ft.net/sdk/cpp/threepm-io-cpp-0.64.0.tar.gz URL_HASH SHA256=…)
FetchContent_MakeAvailable(threepm_io)
```
```toml
[registries.threepm]                                # .cargo/config.toml
index = "sparse+https://3pm.edgecr4ft.net/sdk/cargo/"
[dependencies]                                      # Cargo.toml
threepm-io = { version = "0.64.0", registry = "threepm" }
```

`just publish-sdk` (`distribution/scripts/publish-sdk.sh`) ships each SDK
whose own version is not online yet: it runs the SDK's tests, installs the
fresh archive into an empty project (npm, a venv, a CMake `FetchContent`
build, a cargo build against the sparse registry served locally), uploads
the archives before the indexes and never removes an older version. The archives are reproducible, so a `--force` republish keeps the
sha256 a `URL_HASH` pins. `--index-url`, not `--extra-index-url`: with the
latter pip would also take a same-named package from PyPI.

Outside 3pm every call is a no-op; under 3pm the SDK connects in the
background, sends its whole registry every second, never blocks, never
throws, reconnects with backoff and gives up after five refused `hello`s
(a process that outlived its incarnation). Labels are bounded in the SDK
(3 keys, 50 sets then `overflow="true"`), histograms reproduce the
daemon's bucket grid bit for bit (`packages/exphist-vectors.json`), and
Electron renderers reach the daemon through `io.ipcBridge(ipcMain)` +
`@3pm/api/renderer`. Or speak the NDJSON yourself on
`$THREEPM_CTL_SOCK` — `{"msg":"hello","id":$THREEPM_ID,
"gen":$THREEPM_GEN,"source":"worker"}`, then `{"msg":"metrics","m":[…]}`
every second and `{"msg":"event",…}` when something happens
(`examples/metrics.py`, `docs/protocol.md` "Control channel"). Every
process of a multi-process app (Electron main, a Python engine, a C++
host) opens its own connection with its own `source`; a process that
restarts reconnects under the same source and replaces its predecessor.

**Built-in metrics (Node, Electron).** `io.init({ builtins: true })`
reports the event loop (lag histogram, native p99/max, utilization),
every GC pause by kind, the V8 heap with its limit and the memory
outside it (external, ArrayBuffers), handles, CPU, major page faults and
involuntary context switches (`{ heap_spaces: true }`: the heap per V8
space). In an Electron app, `io.electron(require('electron'))` in the
main process adds every Chromium process by type (count, CPU, working
set), the power state (battery, thermal state, CPU speed limit) and
events for crashed processes, hung windows and sleep/wake;
`createRendererIo(ipcRenderer, { builtins: true })` in a preload adds the
renderer's main-thread lag and long tasks, JS heap, Blink memory, CPU
and DOM size. The full list: `docs/app-metrics.md`, "Node".

**Rust SDK (`packages/rust`, `threepm-io`).** The same metrics, events
and **custom actions** with two dependencies (`serde`, `serde_json`) and
std threads — no async runtime, so it drops into a sync program, a tokio
one or a plugin host alike. `io()` is the process's instance,
`Series` is `Clone + Send + Sync`, and no telemetry call returns a
`Result` or panics: what cannot exist comes back as an inert handle plus
a line under `THREEPM_IO_DEBUG=1`. Actions carry Node's contract —
`ctx.commit_file(ext)` for a result too big for one line, `remote`,
`timeout_ms`, `concurrent`, `output_max_bytes`, a local `ctx.input` — with
the handler's error type being anything that prints. Two things a thread
cannot do that an event loop can: a `metric` callback runs on the flush
thread, and nothing holds the process up, so a short-lived program
flushes before returning and a worker that only answers actions ends its
`main` with `io().serve()`. **Stream sessions** since 0.50.0:
`io().stream(name, Action::new().idle_ms(…), |session, params| …)` serves
one, `io().open(node, app, action, params, Open::new())` opens one, and
`Session` is `send` / `send_frame` / blocking `recv` / `end` / `close` /
`stats` / `result`, over the same `data.sock` legs and abandon sink as
Node's (one chunk per frame, no batching; on a blocked socket a frame
past 100 ms goes on its own age, never the newest, and past 8 MiB the
oldest non-`KEY` frame goes first — only the newest `INIT` is kept);
`io().node()` reads `link.toml`. Not implemented yet: `io.call`, blob
inputs/outputs (a file that crosses the fleet), the built-in
`3pm:profile`, and Node's `builtins`. Surface, bounds and every
difference: `packages/rust/README.md`.

Bounds (1 000 live series per instance, 200 names, 50 label sets then
`overflow="true"`, 10 events/s, 120 s TTL swept every 10 s…) and what happens past them:
`docs/metrics.md`, "Application metrics"; refusals are counted on the
record and shown by `3pm show`. Twenty-three label keys are reserved,
because the fleet backend sets them itself on every series: `cluster`,
`node`, `app`, `instance`, `ns`, `src`, `overflow`, `vmrange`, `le`,
`hostname`, `os`, `arch`, `agent_version`, `mount`, `window`, the host
dimensions `core`, `iface`, `blockdev`, `sensor`, `gpu`, `array`, and
the process-tree ones `proc`, `pid`.

**Where it comes from.** `threepm-protocol`: `app_metrics` (wire types),
`exphist` (the frozen bucket grid), `Request::AppMetrics` (always
answered with the instance's whole state: the daemon cuts no diff),
`Event::App`. `threepm-daemon`:
`app_metrics/{limits,validate,state,answer,events}.rs` (pure: grammars,
caps, the last value of every series under one revision that moves only
when a reader would see a change, token buckets), the control
server (one connection per source, peer pid, validation off the actor),
`supervisor/appmetrics.rs`. CLI: `render/app_metrics.rs` (the section,
approximate quantiles), `render/events.rs` (the `app` line). SDKs:
`packages/node/lib/io/` (`@3pm/api`), `packages/python/threepm_io/`,
`packages/cpp/include/threepm_io.hpp`, `packages/rust/src/` (`threepm-io`,
a standalone workspace), each tested against a fake control socket
(`just sdks`) and under a real daemon
(`crates/threepm-cli/tests/e2e_io.rs`, and `e2e_actions.rs` for the Rust
client's actions).

## The fleet backend's API from Node (`@3pm/api/fleet`)

**What you see.** A script that wants to read a whole fleet — not this
host's daemon, but the backend every node reports to:

```js
const { Client } = require('@3pm/api/fleet');

const api = new Client({ baseUrl: 'https://fleet.example.com', token: '3pmu_…' });
for (const b of await api.listBuckets()) {
  console.log(b.name, (await api.listNodes(b.cluster_id)).length, 'nodes');
}
for await (const [kind, payload] of api.follow(id)) …   // the live feed
```

A method per endpoint, `api.url(name, …)` for the routes that are bytes
rather than JSON, `ApiError {status, code, detail}` with the backend's
own `ERR_*`, and `api.discover()` to follow whichever backend answers.
A token authenticates; there is no browser session here.

**Where it comes from.** The backend declares its routes once
(`3pm-backend/src/api/surface/table.rs`) and `just api` generates the
client from them — twice, from one template: the ES module the backend
serves at `/api/v1/client.js`, and this CommonJS copy at
`packages/node/lib/fleet/api.js`. The template marks its browser-only
blocks (the token prompt, the `EventSource` reader) and the Node build
drops them, so the two can never drift. The generator skips this output
when there is no `3pm` checkout beside the backend.

`@3pm/api` now declares an `exports` map: `.`, `./renderer`, `./fleet`
and `./package.json` — the three entry points that worked before are
unchanged, and deep imports into `lib/` are closed.

## Custom actions and profiling (`3pm trigger`, `3pm profile`)

**What you see.** pm2's custom actions (`io.action`), for 3pm: an app
exposes functions on its control connection, `3pm trigger` runs one on
every instance of a name or on one, and the fleet backend can run the
same through the agent. The SDK's built-in `3pm:profile` action takes a
CPU or heap sampling profile of the process; `3pm profile` wraps it and
prints the hot frames.

```console
$ 3pm trigger api                      # what each source declares
┌──────────┬────────┬─────────────┬────────┬─────────┬────────────────┐
│ instance │ source │ action      │ remote │ timeout │ description    │
├──────────┼────────┼─────────────┼────────┼─────────┼────────────────┤
│ api:0    │ main   │ 3pm:profile │ yes    │ 75s     │ CPU or heap …  │
│ api:0    │ main   │ cache.flush │ yes    │ 5s      │ drop the cache │
│ api:1    │ main   │ 3pm:profile │ yes    │ 75s     │ CPU or heap …  │
│ api:1    │ main   │ cache.flush │ yes    │ 5s      │ drop the cache │
└──────────┴────────┴─────────────┴────────┴─────────┴────────────────┘
$ 3pm trigger api cache.flush '{"all": true}'
api:0 [main] cache.flush ok (12 ms)
  {
    "flushed": 4
  }
api:1 [main] cache.flush ok (9 ms)
  {
    "flushed": 3
  }
$ 3pm profile api --instance 0 --duration 10s
sampling the CPU of api for 10s…
api:0 [main] 3pm:profile ok (10024 ms)
  {
    "kind": "cpu",
    "duration_ms": 10000,
    "sampling": 1000,
    "bytes": 184322
  }
  file: /home/u/.3pm/actions/api/1726300000000-1f.cpuprofile (180.0kb)
 CPU profile: 9500 samples over 10.0 s, 7800 ms busy (79.6% of the time) → /home/u/.3pm/actions/api/1726300000000-1f.cpuprofile
┌──────┬───────────┬─────────────────────┬──────────────────────┐
│ %    │ self (ms) │ function            │ location             │
├──────┼───────────┼─────────────────────┼──────────────────────┤
│ 62.4 │ 6120.0    │ hot                 │ /srv/app/worker.js:6 │
│ 9.2  │ 900.0     │ (garbage collector) │                      │
└──────┴───────────┴─────────────────────┴──────────────────────┘
$ 3pm profile api --heap --duration 30s --src renderer   # an Electron renderer, over its debugger
$ 3pm events -f proc_action                              # the audit line of every call
12:34:56 proc_action     3|api     [main] action cache.flush (local) ok 12 ms
```

The file opens as is in Chrome DevTools (Performance → Load profile)
and in speedscope. Both kinds are *sampling* profiles: `cpu` samples the
stack, `--heap` samples allocations (which functions allocate, not what
the heap holds). There is no heap snapshot: it would freeze the app while
taken, weigh hundreds of MB and carry its data, secrets included.
Locally, `3pm trigger` needs nothing more than the daemon's socket, like
every other command: whoever can talk to the daemon can run any declared
action, `remote: false` included (that flag only closes the fleet path). `--json` prints the outcomes (and, for `profile`,
the summary) for scripts; `3pm show` lists the actions under *Custom
metrics*.

**How to get it.** The Node SDK, the Rust SDK and — since
`[Unreleased]` — the Python SDK (`io.action`, `io.call` with a file in
and a file out, `io.serve`; no stream yet: `packages/python/README.md`);
C++ ignores the messages (it declares nothing yet):

```js
const { io } = require('@3pm/api');
io.init();                                             // source `main`: `3pm:profile` declared by default (`profile: false` to opt out; other sources opt in with `profile: true`)
io.action('cache.flush', { description: 'drop the cache', timeout_ms: 5000 }, async (params) => {
  const n = await cache.flush(params.all);
  return { flushed: n };                               // JSON, ≤ 512 KiB
});
io.action('dump', async (params, ctx) => {             // a bigger result: a file
  await fs.promises.writeFile(ctx.file + '.tmp', JSON.stringify(state));
  return { bytes: await ctx.commitFile('json') };
});
io.action('reset', { remote: false }, () => reset());  // `3pm trigger` only, never the fleet backend
```

A process that declares an action of its own is **held up to serve
it**: a worker whose script is only `io.init()` and `io.action(…)` stays
online instead of exiting once the script has run (the SDK's control
socket is unref'd, so nothing else would hold it). The SDK's own
`3pm:profile` does not count, so a process that only reports metrics
still exits when its work is done; `io.init({ keep_alive: false })` opts
out, and the daemon's `shutdown` lets go. On the calling side, `io.call`
and `io.open` placed right after `io.init()` wait for the control
connection (`connect_wait_ms`, 5 s) instead of failing `node_offline`,
`await io.ready()` says when it is up, and a script awaiting a call is
held until the answer arrives.

An Electron app declares actions in its renderers too (`createRendererIo`
with an `ipcRenderer` that has `on`): each window keeps its own, dropped
when it commits a navigation (a reload too), crashes or closes — a
navigation that never commits (a cancelled `will-navigate`, a download)
keeps them, and the new page is asked to declare its own. A declaration
is acknowledged by the bridge and resent until it is, so one made before
`io.ipcBridge()` still arrives; a renderer's `shutdown()` withdraws it;
renderers cannot take `main`, `pid-*` or `worker-*`; and only the window
a call went to can answer it. The main process's `io.ipcBridge` profiles
a renderer through `webContents.debugger` — DevTools already attached is
an immediate, explicit error, and a source shared by several windows is
refused (give each window its own `source`).

Without a `source`, `io.init()` names the connection `main` in the
process 3pm spawned, `worker-<pid>-<threadId>` in a worker thread and
`pid-<pid>` in a forked child or cluster worker, so they never evict
each other. Two processes that both claim `main` (`sh -c 'node a & node
b'`) settle it: the refused one falls back to `pid-<pid>` (and drops its
default `3pm:profile`). A child started by a non-Node parent otherwise
cannot be told apart — name it.

**The rules.** A source declares its whole list at once (32 at most,
the built-in `3pm:profile` included; names `^[A-Za-z0-9_.:-]{1,64}$`, `3pm:` reserved for the SDK); the
daemon keeps it per connection and sends it again after a reconnect.
A call is bounded: the caller's `--timeout` never exceeds the
declaration's (30 s default, 10 min ceiling); one call at a time per
(source, action) unless it declares `concurrent`, 16 per source, 32
plain calls per daemon (streams counted apart, 16), else `busy`; a name
fans out to 16 instances at most; `params` ≤ 64 KiB, a result ≤ 512 KiB
inline (nested 64 levels at most) or ≤ 64 MiB as a file. Triggered from
the fleet backend the two bounds are the link's: a result travels gzip
inside the reply — 256 KiB of gzip for the whole reply, and a result
over that fails `too_large` saying to return a file instead — while the
file itself is **uploaded to the backend before the reply**, up to the
action's declared `output.max_bytes` (32 MiB when it declares none), and
the reply names the artifact it became. Files live under `~/.3pm/actions/<app>[-<n>]/`
as `<epoch_ms>-<call>.<ext>`, private to the user (`0700`); the daemon
keeps 20 per instance and 200 MB in all, oldest out — never a file whose
call is still gathering, and a file younger than 10 minutes is spared
by the count — and collects what a timed-out call left behind (after
every file, and hourly). A result file must be a regular file: a
symlink is refused. An action never takes the last slot of an app's
control queue, so `shutdown` always gets through. Every call —
succeeded, failed, refused — is one `proc_action` event on the bus with
its origin (`local` / `remote`), never its params or result; a trigger
refused as a whole (ambiguous source, nothing declares the action, more
than 16 instances) leaves one per instance it named. Per-instance
errors are typed (`failed`, `timeout`, `busy`, `gone`, `refused`,
`too_large`) and `3pm trigger` exits 1 when any instance failed, 2 when
nothing declares the action (the message lists what is declared).

Wire contract: `docs/protocol.md` (`dispatch` / `called`, the control
channel's `actions` / `action` / `action_reply`, `proc_action`).

## Zero-downtime update: `3pm update` hands the daemon off in place

```
$ 3pm update
[3PM] Updating daemon v0.9.2 → v0.10.0 (handoff: apps keep running)
[3PM] Saved 12 process(es)
Daemon updated to v0.10.0 in 180 ms — 12 process(es) kept running, 0 restarted
```

The daemon re-executes the new binary **in the same process**: the apps
keep their pids and their pipes, every log line survives (including the
bytes read but not yet written at the switch), an app that exits during
the switch keeps its real exit code, restart timers resume with the time
they had left, `3pm logs -f` stays attached, and clients that talk to the
daemon during the few milliseconds of the switch simply wait. A same-
version `update` is a live reload of `config.toml` (log rotation policy
and the like). `3pm internals` shows the image's `boot id`, how many
handoffs it went through and the last one's outcome.

**Every failure has a floor no worse than before.** The new binary is
run first as a pre-check on a draft of the handoff; a binary that cannot
take over (too old, incompatible, unreadable config, disk full) is
refused and `update` falls back to the cold path (save → kill → respawn
→ resurrect, the apps restart once) — `--no-fallback` exits 3 instead
and leaves the apps alone, `--cold` asks for the cold path outright. A
failed `execve` resumes the old image with nothing lost; a new image
that cannot adopt the apps stops them with their own signals and
restarts them from the saved list; a new image that dies outright is
caught by a small guardian process that stops the apps and restarts the
daemon on the same sockets. `--json` reports `mode` (`handoff`, `cold`,
`emergency`, `recovered`), `kept`, `restarted`, `duration_ms` and, on a
fallback, `fallback_reason`.

Not carried across (documented, bounded): an SDK `ready`/metric/event
sent during the switch (the SDK resends its registry on reconnect; a
`ready` lost is covered by `listen_timeout`), a watched file changed in
the window, a cron tick landing exactly in it. Two hundred apps cross the switch
in about 100 ms on a laptop; an app in backoff, one on its kill timeout
and one waiting to say `ready` each finish their cycle under the new
image at the moment they would have anyway, and a signal aimed at the
daemon mid-switch is held until the new image can act on it
(`tests/e2e_handoff_states.rs`).

**3pm's own processes are restarted at every update.** The apps are
what a handoff keeps running; what runs 3pm's own code is not. The
daemon is re-executed, the host agent is stopped by the old image and
started again by the new one, and every live module app — the fleet
agent, the `3pm serve` sites, a module's `[[apps]]` — is restarted once
the new daemon serves, **whether the release changed it or not**. So a
module never runs code, nor carries process state, older than the daemon
it talks to, and `just bundle && 3pm update` is the whole dev loop. The
price, at every `update` (a same-version one included, which is also the
live reload of `config.toml`): the fleet link drops for the time of an
agent restart — it comes back on its own, no new enrollment — and a
served site blips. `--skip-modules` leaves the module apps alone.

**Modules follow, after the daemon.** The apps of a module keep running
the old code through the switch (same pid), so `update` then refreshes
them, in place — same row, same name, a new `spec.module.version` stamp
— never deleted and recreated. Two populations: an installed copy
(`~/.3pm/modules/<name>/`) whose bundled version changed is replaced by
a staged copy and its rows restarted (the manifest apps on their new
spec, the sites it serves and the apps started by hand from its files
as they are; a `[[apps]]` entry the release added is started, one it
removed is deleted); an app running straight from the bundle without an
install — the agent after `3pm link`, a site served with `3pm serve`
before any install — is restarted on the new binary and re-stamped when
the `module.toml` next to its script moved on. A row already at the
bundle's version is restarted as it is — same spec, same stamp, nothing
reconciled: an app of the manifest that you deleted stays deleted, and
a module app whose every instance is **stopped** stays stopped (stopped
on purpose; an update is no reason to bring it back). The decision is
per row, never per module: an installed copy whose files are current but
whose rows are not (a refresh that failed after the copy) is re-stamped
by the next `update`. `--skip-modules` leaves them all alone. The cold
path restarts every app anyway: only the module apps behind their stamp
are restarted once more to carry it, never the current ones. A module
app whose directory vanished is left as is, with a notice.

```
Daemon updated to v0.10.0 in 180 ms — 12 process(es) kept running, 0 restarted
Module [serve] 0.9.2 → 0.10.0 reinstalled — 2 app(s) restarted
Module [agent] 0.9.2 → 0.10.0 refreshed from the bundle — 1 app(s) restarted
```

and, when the release did not touch a module (or on a same-version
`update`):

```
Module [agent] 0.10.0 — 1 app(s) restarted
```

`--json` adds `modules: [{name, from, to, source: "installed"|"bundle",
restarted, started, deleted}]` — `from == to` for a module restarted as
it is; absent under `--skip-modules`, when no module app is live, and on
a cold path with nothing to re-stamp. The module restarts are counted
there, not in `restarted`.

**Where it comes from.** OV `3pm-docs-archived/architecture` §4.7 and the plan in
`3pm-handoff-plan.md` in `~/3pm-rd-doc-2026-09-09.tgz`; daemon
side `crates/threepm-daemon/src/handoff/`
(`orchestrate`, `exec`, `guardian`, `inherit`, `adopt_main`, `emergency`,
`manifest`, `check`) and `bootstrap/daemon_main.rs`; actor side
`supervisor/{quiesce,handoff,manifest,adopt}.rs`; client side
`crates/threepm-protocol/src/client/handoff.rs`; CLI
`commands/{daemonctl,update_handoff,module_refresh}.rs`,
`render/update.rs`. Tests: `crates/threepm-cli/tests/e2e_handoff.rs`
(real daemons, a second copy of the binary standing in for the new
version, test knobs `THREEPM_HANDOFF_{HOLD_MS,CRASH_AT,EXEC_FAIL}` read
only under the test marker), the guardian as a real forked process in
`handoff/guardian_tests.rs`, the actor's extraction/adoption under paused
time in `supervisor/tests/{handoff,adopt}.rs`, the modules step against
a bundle bumped in place in `tests/e2e_modules.rs` (both populations,
`--skip-modules`, the cold path, a vanished directory) and against the
real agent in `modules/agent/tests/e2e.rs`, the transient states and the
200-app switch, the exit and the held signal inside the window in
`tests/e2e_handoff_states.rs`.

## Boot in your own session: `3pm startup --level user`

**What you see.** `3pm startup` installs a system service (root, the
daemon running as you). A desktop app that must live in your GUI
session — macOS AudioUnits, the keychain, anything that checks the Aqua
audit session — needs the daemon there too: `3pm startup --level user`
installs a LaunchAgent under `~/Library/LaunchAgents` (`systemctl
--user` on Linux) without root, and `3pm unstartup --level user` removes
it. `3pm save` records what comes back at login.

**Where it comes from.** `commands/startup.rs` (`ServiceLevel::User`
through `service-manager`), `cli.rs` (`--level`).

## Hosted users: one root daemon, per-user sockets

**What you see.** A root daemon can **host** the accounts you declare, so
a non-root user drives its own apps — start, stop, logs, metrics — without
a daemon of its own, and root still sees the whole machine in one
`3pm ls`. Two lines in root's `config.toml`, then a restart of the daemon:

```console
# /root/.3pm/config.toml
[users]
allow = ["srt"]              # optional: dir = "/var/lib/3pm/users", max_apps = 32

$ sudo 3pm update            # 3pm.log: "hosting user" user=srt home=/var/lib/3pm/users/srt
```

`srt` now has its own `rpc.sock`, `pub.sock` and `ctl.sock` in
`/var/lib/3pm/users/srt/`, and its CLI finds that home on its own — no
`THREEPM_HOME` to set, no daemon to start:

```console
srt$ 3pm start cam.py --name tabloid-cam --persist false
srt$ 3pm ls                            # its own rows, nothing else
srt$ 3pm logs tabloid-cam              # /var/lib/3pm/users/srt/logs/…, then the stream
srt$ 3pm start x --user root
[3PM][ERROR] cannot run as root: a hosted user runs its apps as itself
srt$ 3pm start kiosk                   # a name root already holds
[3PM][ERROR] `kiosk` belongs to root
srt$ 3pm save
[3PM][ERROR] forbidden: `save` is a daemon-level command (hosted user)
srt$ 3pm ls                            # with the root daemon stopped
[3PM][ERROR] this home (/var/lib/3pm/users/srt) is hosted by the root 3pm
daemon, which is not running — a hosted user never starts a daemon of its own
```

Root administers everything, and can run an app **as** a hosted account
without that account being involved at all:

```console
$ sudo 3pm start api.js --user srt     # or `user = "srt"` in an ecosystem file
$ sudo 3pm ls                          # every row, srt's included
$ sudo 3pm show api                    # the account it runs as, its owner, whether it persists
$ ps -o user,pid,cmd -p $(sudo 3pm show api --json | jq .[0].pid)   # srt
```

**Two axes, on purpose.** `user` is *who the app runs as*: the uid, gid and
supplementary groups of the child, and the home its files go to. `owner` is
*whose connection started it*: what `3pm ls` shows that caller, and what it
may stop, restart or delete. Root starting an app as `srt` produces
`user = srt`, no owner: root's app, running as `srt`, its logs in `srt`'s
home. `srt` starting an app produces both — the daemon stamps `owner` from
the connection and forces `user` to the same account. `owner` is never a
field you set: a client that sends one is overwritten, silently and on
every path (`start`, `apply`, a patched `restart`, a rename, `scale`).

**What the daemon holds to.**

| | |
|---|---|
| Identity per connection | each user's three sockets are chowned to it (0600) and peer-uid checked against *that* uid; the daemon knows who is calling before it reads the request |
| Root's commands stay root's | `save`, `resurrect`, `kill`, `internals`, `orphans` and the `3pm update` handoff answer `forbidden` to a hosted caller (V1: `save` would let one account decide what root relaunches at boot) |
| The host is everyone's | `3pm hostmetrics` (and the SDK's `hostMetrics()`) answers a hosted caller: CPU, frequencies, memory, network, sensors are the machine's. What names the other accounts is dropped from its reply — `top_processes` (their processes, root's included) and `sessions` (who is logged in) |
| Nothing of root's environment | a hosted child's environment is built from scratch: `PATH`/`LANG`/`TZ` carried over, the account's `HOME`/`USER`/`LOGNAME`/`SHELL`, the app's own `env`. Root's `3pm link` credentials and `SUDO_*` never cross |
| Nothing of root's descriptors | the daemon's sockets are never inherited by a hosted child, including one spawned by the image that just took over an in-place handoff |
| Files root writes, the account reads | the hosted home and its `logs/`, `pids/`, `actions/` are `root:<gid> 0750`; logs, rotated generations and pid files are `root:<gid> 0640`, set explicitly rather than left to the umask. An app of a hosted user may not choose its log paths (`out_file`, `error_file`, `log_file` are refused): root would open — and `delete --flush` unlink — whatever it named |
| No way back to root | the child drops groups, gid then uid, proves `setuid(0)` fails, and on Linux sets `PR_SET_NO_NEW_PRIVS` |
| Bounded | at most `[users] max_apps` rows per account (default 32), 64 live connections per hosted socket, and `watch` roots that stay inside the app's `cwd` |
| The fleet is not a privilege | a hosted app calls other nodes (`io.call({node})`, `3pm call --node`), sends and receives files, and opens streams exactly as a root app does — no setting, nothing to grant. Its blobs travel on `agent/data-<uid>.sock`, which the agent binds for its uid alone (0600) |
| Names stay global | a name another owner holds is refused on both sides; root is told how to free it (`3pm delete <name>`) |

**`persist = false`.** A new per-app field (`--persist false`, or in an
ecosystem file), for apps that must not come back by themselves: the app is
supervised, restarted and carried across `3pm update` exactly like any
other, but it is left out of `dump.json` — so out of `3pm save`, out of the
implicit save `3pm update` does, and absent after a reboot. It is what an
ephemeral per-frame backend wants: at boot, the app that owns the camera
starts it, not the resurrect list. And it is not part of the configuration
`3pm apply` compares: changing `persist` alone updates the running rows in
place, so the backend holding the camera is never cycled for it.

**Edge cases worth knowing.**

- **Name squatting.** App names are one global namespace. A hosted account
  can take `kiosk` before root does; root, who sees every row, frees it
  with `3pm delete kiosk` (the refusal says so). V1 has no `--force`:
  taking a running app away from its owner is a decision, not a flag.
- **`forbidden` exits 1**, like any other failure — only `not_found` has
  its own code (2). Scripts read `code` from `--json`.
- **Adding or removing an account takes a daemon restart** (`3pm update`):
  `[users]` is read once, at start. There is no `3pm users add` in V1.
- **A daemon that is not root ignores `[users]`** entirely, with a warning
  in its log: it could not become another user anyway. `--user` on such a
  daemon is refused at start rather than quietly running as the daemon.
- **The account needs a private primary group** (`useradd -m -U srt`).
  A shared one (`users`) is refused: every member of the group would read
  the app's logs. So is uid 0, and a second name for a uid already allowed.
- **`PR_SET_NO_NEW_PRIVS` breaks setuid binaries** — chromium's setuid
  sandbox among them. An app that needs one stays a root app dropping its
  own privileges (`setpriv`), not a `user =` app.
- **A hosted client is cut for a moment by `3pm update`.** Handoff V1
  rebinds the hosted sockets in the new image instead of carrying them
  across the exec; a hosted client waits up to 5 s for the socket to answer
  again and reconnects (`3pm logs --follow` included). Apps keep running.
- **`3pm orphans` scans root's `pids/` only**, so a hosted app left behind
  by a dead daemon is not listed there.

**Where it comes from.** `crates/threepm-daemon/src/users/` (`config` reads
`[users]`, `home` prepares and binds, `caller` is the per-connection
identity, `env` the clean environment, `file_owner` the 0640 files, `gate`
the connection cap), `supervisor/admit.rs` (stamping and every refusal),
`spawner_user.rs` (the privilege drop), `bootstrap/services.rs` (one accept
loop set per user) and `threepm-core/src/paths/hosted.rs` +
`threepm-protocol/src/client/home.rs` (the marked home a client finds and
never spawns into). Wire contract: `docs/protocol.md` → *Hosted users*;
files and `[users]`: `docs/logs.md`; the fields: `docs/ECOSYSTEM.md`.
Tests: `crates/threepm-cli/tests/e2e_users.rs` and
`e2e_users_multi.rs` (root-gated, `just e2e-users`: the lane creates the
accounts it hosts; the second suite is the boundary between three of
them), `supervisor/tests/users.rs` under paused time, and the unit suites
of each `users/` module.

## Containers: `3pm runtime` (PID 1, auto-exit)

**What you see.** One foreground process that owns the apps, streams
their output to the container's stdout/stderr, drains on SIGTERM, and
**exits when there is nothing left to supervise** — so the
orchestrator's restart policy is the outer loop:

```console
$ 3pm runtime ecosystem.toml --env production     # every `start` flag works
[3PM] 3pm runtime: daemon pid 7, home /root/.3pm
[3PM] App [api] launched
0|api | listening on :3000
[3PM] App [api] exit code=1 signal=SIGSEGV
[3PM] App [api] errored
[3PM] nothing left to supervise: stopping the daemon (exit 2)
```

| Exit | When |
|---|---|
| `0` | SIGTERM/SIGINT → every app drained through its policy |
| `2` | nothing left to supervise: zero apps selected, or every app `stopped` / `errored` (a crash loop counts — the container restart policy takes over). `--no-auto-exit` disables |
| `1` | the daemon died without announcing, or a start failed |
| `130` | a second signal forced the daemon down |

`--raw` gives bare lines (pm2-runtime's default; 3pm prefixes `id|name |`
because containers usually run several apps), `--format` logfmt,
`--timestamp [FMT]`, `--json` the wire events. Log *files* are off
(`NULL`) unless `--log-files` or an explicit `-o`/`-e`/`-l`. `--delay
SECS` waits before starting. Unlike `oxmgr runtime`, the apps live in a
real daemon: `docker exec c 3pm ls|logs|events|restart api` all work.

**How to get it.**

```dockerfile
FROM node:22-slim
COPY --from=build /src/target/release/3pm /usr/local/bin/3pm   # or a release tarball
WORKDIR /app
COPY . .
CMD ["3pm", "runtime", "ecosystem.toml", "--env", "production"]
```

Full example: `examples/Dockerfile`; cheatsheet: `docs/RUNTIME.md`.

`3pm runtime` refuses to start if a daemon already owns `THREEPM_HOME`
(it must own the daemon to own the exit code).

**Where it comes from.** `commands/runtime.rs` (one `select!` over the
bus, signals and a `waitpid(-1)` reaper; `nothing_to_supervise()` is the
pure auto-exit predicate), `threepm-protocol::spawn_foreground`
(`SpawnMode::Foreground`: same session, inherited stdio, caller-reaped)
and `3pm __daemon --foreground` in the daemon.

## Resource limits: CPU ceiling, cgroup enforcement, GPU denial

**What you see.** `max_memory_restart` has two siblings. `max_cpu_percent`
recycles the app the moment one CPU sample crosses the limit — `3pm show`
counts a restart and the daemon log says `max_cpu_percent exceeded`. Both
are soft guards: the app is *observed* and gracefully restarted, and both
judge its **whole process tree** — the RSS sum and the CPU sum of the
spawned pid and every descendant, what `3pm ls` shows — unless
`limit_scope = "leader"` narrows them to the spawned pid (children that
are meant to grow: a build, a browser). On
Linux, `cgroup_enforce = true` additionally makes them **hard**: each
incarnation runs in its own cgroup v2 leaf under `/sys/fs/cgroup/3pm/`
with `cpu.max` (throttled by the kernel) and `memory.max` (OOM-killed by
the kernel) set from the same two fields. `deny_gpu = true` hides GPUs
from the app through the CUDA/ROCm visibility variables.

A guard restart is paced and counted like a crash: after the graceful
stop the row emits `restart` and sits in `waiting restart` for
`restart_delay` / the exponential backoff, and a trip within
`min_uptime` + 10 s of the incarnation going `online` (or 3 sampler
periods, when `THREEPM_METRICS_INTERVAL_MS` makes that longer; under
`wait_ready`, `online` is the ready signal or `listen_timeout`) counts
toward `max_restarts`. An app
over its limit from the start therefore ends `errored` (16 trips with
the defaults) instead of cycling forever, while one that runs a while
between two trips never spends its budget. A user `restart` / `stop` /
`delete` during the guard's stop starts a new series. `max_restarts = 0`
ends the app `errored` on its first trip, as on its first crash.
`restart_policy = "never"` and `stop_exit_codes` do not prevent a guard
restart — they judge the exits an app makes on its own, and a guard
restart is the daemon's recycle, as in pm2 — so leave the guard off on
an app that must never be restarted.

**How to get it.**

```console
$ 3pm start train.py --max-cpu-percent 150 --max-memory-restart 4G --cgroup-enforce --deny-gpu
```

```toml
[[apps]]
name = "train"
script = "./train.py"
max_cpu_percent = 150       # percent of one core; >100 on multi-core hosts
max_memory_restart = "4G"
limit_scope = "tree"        # "tree" (default): the sums over the process tree; "leader": the spawned pid alone
cgroup_enforce = true       # Linux only: hard cpu.max / memory.max
deny_gpu = true             # CUDA_VISIBLE_DEVICES="" NVIDIA_VISIBLE_DEVICES=none …
```

`max_cpu_percent` must be a positive number; a spec that says otherwise
is rejected at the CLI edge and again at the daemon boundary.
`cgroup_enforce` on macOS, or on a Linux host without cgroup v2 at
`/sys/fs/cgroup`, fails the spawn *before* the app starts — the row goes
`errored` and `3pm show` names the reason — so an enforce spec never runs
unenforced. A permission error tells you to run the daemon as root or
delegate `/sys/fs/cgroup/3pm` to its user. `deny_gpu` is best effort: an
app that unsets those variables can still open `/dev/nvidia*`. Full
reference and cheatsheet: `docs/resources.md`.

**Where it comes from.** The policy's `CpuSampled{gen, tree_cpu,
leader_cpu, sample_period_ms}` and `MemorySampled{gen, tree_rss,
leader_rss, sample_period_ms}` inputs carry both figures and the
sampler's period (the supervisor's `metrics_interval`); in
`threepm-core/src/policy/guards.rs`, `scoped` picks the one
`limit_scope` names inside `step`, `on_resource_sampled` is shared by
the two guards (gen-guarded, online rows only, never during daemon
shutdown), and `after_guard_stop` respawns through the crash path
(`respawn_or_errored`), the trip's uptime (since
`ProcState::online_at`, carried by the handoff manifest as
`online_at_age_ms`) judged against `min_uptime` +
`settle_margin_ms(period)`: `GUARD_SETTLE_MS` (10 s) or
`GUARD_SETTLE_PERIODS` (3) periods, whichever is longer.
`threepm-daemon::cgroup` owns
the cgroup file protocol against an injected root (unit-tested on every
platform, `cgroup_tests.rs`): `CgroupFs::prepare` makes the
incarnation's leaf (`<name>-<id>-<instance>-<lineage>-g<gen>`) and
writes its limits before the spawn, the child writes itself into the
leaf's `cgroup.procs` between fork and `exec` (`spawner_cgroup.rs`) — a
failed join fails the spawn and `discard` removes the leaf, as `prepare`
does itself when a limit cannot be written —, and
`CgroupFs::cleanup`, off the actor on `ChildExited` / delete, kills what
is left in the leaf (`cgroup.kill`, else SIGKILL member by member with
the leaf frozen, so no pid is reused between the read and the kill),
removes the cgroups the app made below it and the leaf, then the
instance's older empty leaves. The log files are opened before all of
it: one that cannot be opened fails the spawn before any process or
leaf exists.
`deny_gpu` is `spawner::GPU_DENY_ENV`, applied after the user's `env` so
it wins.

## Limits and scheduling: `ulimit_*`, `nice`, `ioprio`, `oom_score_adj`, `cpu_affinity`

**What you see.** An app that runs its threads in `SCHED_FIFO` (real-time
audio: a PipeWire client, a plugin host, a DSP) needs a real-time
priority limit and room to lock its buffers in RAM. A 3pm daemon started
by systemd has neither (`Max realtime priority 0`, `Max locked memory`
8 MB: a system unit never goes through PAM's `limits.conf`), and every
app inherits them. `ulimit_rtprio` and `ulimit_memlock` set the app's
`RLIMIT_RTPRIO` / `RLIMIT_MEMLOCK` (soft = hard) right before `exec`;
`/proc/<pid>/limits` shows them, the app's whole tree inherits them, and
`3pm show` lists them as `rtprio limit` / `memlock limit`. 3pm does not
change any scheduling itself: the app asks for `SCHED_FIFO` for the
threads that need it.

**How to get it.**

```console
$ 3pm start ./onyx --user srt --ulimit-rtprio 95 --ulimit-memlock unlimited
```

```toml
[[apps]]
name = "onyx"
script = "./onyx"
user = "srt"
ulimit_rtprio = 95          # Linux only, 0–99
ulimit_memlock = "4G"       # a size, or "unlimited"
```

From the Node SDK: `start({ script, ulimit_rtprio: 95, ulimit_memlock: '4G' })`.

Who may ask for what is judged at `start`, never by a restart loop:

- a **root** daemon grants any value — it sets the limit before the drop
  to `user`, so past its own — provided it holds `CAP_SYS_RESOURCE`
  (root in a stock container does not: it is then judged like the next);
- a daemon that is **not root** grants up to its own hard limit, and
  refuses more (`ulimit_rtprio 95: above this daemon's hard limit (0);
  only a root daemon (with CAP_SYS_RESOURCE) can raise it`);
- an app a **hosted user** owns (started over its own sockets, like the
  frame backends tabloid starts as `srt`) gets at most what the root
  daemon's `config.toml` grants; when nothing is granted it may still
  keep or lower what the daemon runs with (`ulimit_memlock = "0"`), never
  raise it:

```toml
[users]
allow = ["srt"]
max_ulimit_rtprio = 95
max_ulimit_memlock = "4G"     # or "unlimited"
```

`ulimit_rtprio` on macOS is refused (no `RLIMIT_RTPRIO`); `ulimit_memlock`
works there too. `no_new_privs`, set on a hosted user's apps, does not get
in the way: an unprivileged process may take `SCHED_FIFO` up to its
`RLIMIT_RTPRIO` whatever it says (PipeWire runs that way). A new value
takes effect at the next spawn (`apply` restarts a changed app); a
handoff (`3pm update`) keeps the running apps' limits. The rules above
judge `start`, `apply`, a patched `restart` and a rename; a resurrect
(`dump.json`, boot), a crash respawn and a plain `3pm restart` re-run
the admitted spec as it is, like `max_apps`: a lowered `[users]` grant
reaches an app already admitted only once it is `3pm delete`d and
started again. A
daemon older than the field ignores it: update 3pm before relying on it.
With `cgroup_enforce` on a kernel built with `CONFIG_RT_GROUP_SCHED`
(rare: Ubuntu and the Surface kernels leave it off), the app's own
cgroup has no real-time budget and `SCHED_FIFO` fails whatever the
limit.

**The rest of a desktop session's scheduling.** A desktop session also
hands its apps a nice limit (PAM, rtkit), and the frame's processes use
it: Chromium lowers its audio threads to nice -8, the plugin host and the
MIDI scheduler fall back to -11 when `SCHED_FIFO` is refused. Under a
systemd-started daemon `Max nice priority` is 0 and those calls fail
silently. Six more fields, same model — set by the child on itself
before `exec`, judged at `start` — fill that in and add the guards a
kiosk wants:

| Field | Flag | Sets | Range |
|---|---|---|---|
| `ulimit_nice` | `--ulimit-nice` | `RLIMIT_NICE`: a nice down to `20 - v` | 0–40, Linux |
| `ulimit_rttime` | `--ulimit-rttime` | `RLIMIT_RTTIME`, µs a real-time thread may run without blocking before the kernel kills it (SIGKILL, no SIGXCPU warning: soft = hard) | > 0, Linux |
| `nice` | `--nice` | the app's nice value (`setpriority`) | -20–19 |
| `ioprio` | `--ioprio` | the I/O class: `idle`, `best-effort[:0-7]`, `realtime[:0-7]` | Linux, BFQ / mq-deadline |
| `oom_score_adj` | `--oom-score-adj` | the OOM killer's bias, -1000 = never | -1000–1000, Linux |
| `cpu_affinity` | `--cpu-affinity` | the CPUs it may run on (`sched_setaffinity`) | `"0-3,6"`, Linux |

```toml
[[apps]]
name = "frame"
script = "./frame"
user = "srt"
ulimit_nice = 31            # its threads may go to -11
nice = -5
oom_score_adj = -900        # the last app the OOM killer picks
ulimit_rttime = 200000      # a FIFO thread hogging 200 ms is killed

[[apps]]
name = "indexer"
script = "./indexer"
nice = 10
ioprio = "idle"             # disk only when nobody else wants it
cpu_affinity = "2-3"
```

`3pm show` lists them (`nice limit`, `rttime limit`, `nice`, `io
priority`, `oom score adj`, `cpu affinity`). The `pre_exec` order is the
limits first, then nice, I/O class, OOM score and CPUs, so a `nice` below
the daemon's can rest on the app's own `ulimit_nice`. A refused step
names its field: `nice -11: cannot set it: Permission denied`.

Who may ask for what, judged at `start` like the limits:

- **anything that only gives up** — a higher nice or OOM score, the idle
  or best-effort I/O class, a lower limit, a subset of CPUs — needs no
  privilege and is never refused;
- a daemon **without the capability** refuses: a `nice` below its own
  without `CAP_SYS_NICE`, unless `20 - nice` fits the app's `ulimit_nice`
  (or the daemon's soft `RLIMIT_NICE`); an `oom_score_adj` below its own
  or a limit above its hard one without `CAP_SYS_RESOURCE`; the realtime
  I/O class without `CAP_SYS_NICE` / `CAP_SYS_ADMIN`;
- a **CPU** the daemon may not run on is refused (`cpu_affinity 2-5: CPU
  4-5 not available to this daemon (it runs on 0-3)`), not left to an
  `EINVAL` restart loop;
- an app a **hosted user** owns gets a nice below the daemon's only
  down to `20 - max_ulimit_nice`, an `oom_score_adj` below the daemon's only down
  to `min_oom_score_adj`, never the realtime I/O class (it starves every
  other app's disk), and a `ulimit_rttime` no higher than the daemon's:

```toml
[users]
allow = ["srt"]
max_ulimit_nice = 31          # nice down to -11
min_oom_score_adj = -900
```

All but `nice` are refused off Linux. `ioprio` is honoured by the BFQ
and mq-deadline disk schedulers; `none`, the NVMe default, ignores it.
`nice` ranks an app against the processes of its own CPU group only: a
`tty` app (its own session, so its own autogroup under
`CONFIG_SCHED_AUTOGROUP`, the Ubuntu default) or a `cgroup_enforce` app
(weighed by `cpu.weight`) is not ranked against the others by it. A
value this 3pm cannot read in a `dump.json` or from a newer peer
(`ioprio = "deadline:9"`) fails that app's spec only, not the dump.

**Where it comes from.** `StartSpec.ulimit_*`, `nice`, `ioprio`,
`oom_score_adj`, `cpu_affinity` (`threepm-core/src/spec.rs`; the value
types in `spec/ulimit.rs` — a number of bytes or `"unlimited"` on the
wire, never `u64::MAX` — and `spec/sched.rs` — `IoPrio` and `CpuList`
cross the wire as their text); the `pre_exec` step `spawner_rlimit.rs`
(after the cgroup join and the terminal login, before `spawner_user`'s
drop, each step's errno offset by `RLIMIT_BASE + step × 0x1000`);
`check_ulimits` in `supervisor/admit_ulimits.rs`; the caps in `[users]`
(`config_file.rs`, `users::UlimitCaps`).

## Cron restarts

```toml
[[apps]]
name = "nightly"
script = "./job.sh"
cron_restart = "0 2 * * *"     # pm2's 5 fields; "30 0 2 * * *" for seconds
```

`--cron-restart "0 2 * * *"` (alias `--cron`) on `start`. Five fields
(minute first, pm2) or six (seconds first, OxMgr); lists, ranges, steps,
month/day names, `7` = Sunday, and when both day-of-month and
day-of-week are restricted either matches (Vixie cron). The expression
is validated at `start`/`validate`, stored in its 6-field spelling, and
`show` prints it with the **next fire** in local time (`ls --json`:
`next_cron_epoch_ms`). Each tick restarts the app's **online** rows,
exactly like a file change: a `stopped` app is never revived by its
schedule, a row mid-restart skips the tick, and the scheduler is
dropped while every row is stopped. Local time zone, DST-aware
(re-evaluated at every fire).

## Target patterns: `3pm logs 'victoria*'`

**What you see.** Anywhere a command takes a process, a target holding
`*` or `?` is a **pattern** rather than a name — no flag, no mode:

```
$ 3pm logs 'victoria*'          # the three victoria apps at once
$ 3pm restart 'worker-?'        # worker-1 … worker-9
$ 3pm delete 'bench*'
$ 3pm show 'api*'
$ 3pm events --process 'web-*'
```

`*` matches any run of characters (possibly empty), `?` exactly one —
the same vocabulary as `ignore_watch` and `3pm events --filter`. The
match is **whole-string** and case-sensitive: `victoria*` selects
`victoria-logs`, and the substring form is spelled `*victoria*`. A
pattern is matched against the same two fields a literal name is — the
app name **and** the namespace — so a namespace pattern sweeps
everything filed under it. Ids are never matched: a numeric target is an
id, and `1*` is a name pattern.

Quote it. An unquoted `victoria*` is the shell's to expand, and it
silently becomes a filename (or nothing) before 3pm ever sees it — the
"no process matches" notice says so.

**Where it applies.** `logs`, `stop`, `restart` (including
`--update-env`), `delete`, `reset`, `show`, `sendSignal`, `flush`,
`rotate`, and `events --process`. `scale` refuses one: an instance count
belongs to a single app.

A pattern that selects nothing is a **notice** when other targets
matched (`3pm logs api 'ghost*'` still follows `api`) and the run's
error when nothing is left to act on — for a mutation the error comes
*before* the first request, so nothing is half-done.

**How it works.** Patterns are resolved in the CLI, against the process
list the daemon just returned, and only the names they selected go on
the wire. The protocol carries no glob: no `Target` variant to grow, and
no daemon that could resolve a pattern differently from the one that
printed the list you were looking at. `events --process` is the
exception — it is a server-side filter, so the glob travels and the
daemon matches it.

Module rows (`3pm install …`) are shielded from a pattern on a mutation
exactly as they are from `all`: `3pm stop 'a*'` never stops the agent,
only its spelled-out name does. Reads hide nothing — a module's logs are
logs like any other.

## `sendSignal`

`3pm sendSignal SIGUSR2 api` (aliases `send-signal`, `signal`; bare
names like `usr2` are accepted) delivers a signal to the running process
of each matched row — the process itself, as pm2 does, or its whole
process group with `--group`. It is not a lifecycle operation: nothing
about restart accounting changes, and if the signal kills the process
that is an ordinary exit for the restart policy. A stopped app is an
error (`exit 2`), an unknown signal name gets a did-you-mean.

## pm2's process dumps: `3pm jlist`, `3pm prettylist`

For scripts written against pm2, `3pm jlist` prints the process list as
`pm2 jlist` does (one line of JSON) and `3pm prettylist` as `pm2
prettylist` does (Node's `util.inspect`): `{pid, name, pm2_env, pm_id,
monit}` per process, with pm2's keys in `pm2_env` (`status`,
`restart_time`, `NODE_APP_INSTANCE`, `exec_mode`, `watch`,
`pm_out_log_path`, …) and the app's variables at the top of `pm2_env`.
A field appears as often as in pm2's output, so `grep -c` counts match.
Module apps are left out. `exec_mode` is derived: `cluster_mode` for an
app of several instances, `fork_mode` otherwise. Both are hidden from
`3pm --help`; `3pm ls --json` stays the native format.

## `3pm cd`: go to an app's working directory

A command cannot move the shell that ran it, so `3pm cd` answers in two
ways, chosen by where its stdout goes:

```
$ 3pm cd api                  # a terminal: a shell in the app's cwd
[3PM] /bin/zsh in /srv/api (exit to return)
$ cd "$(3pm cd api)"          # piped: the path, for this shell
$ 3pm cd api --json
{"name":"api","cwd":"/srv/api"}
```

The shell is `$SHELL` (`/bin/sh` when unset), exec'd in place: no 3pm
process waits behind it. A folder that no longer exists is an error
rather than a shell somewhere else. The target is one name or id (the
instances of an app share its cwd); `all` and patterns are refused. For
a real `cd` without the subshell, a shell function does it:

```sh
3cd() { cd "$(3pm cd "$1" | cat)"; }
```

## Host metrics agent (`3pm hostmetrics` served from a separate process)

**What you see.** `3pm hostmetrics` answers from a cache in ~10ms and
its overview starts with a `sampled` row (`2026-08-28T15:02:48Z (3s
ago)`; `collected_at_epoch_ms` in `--json`). `3pm ls` — and every
command that reprints the table — ends with pm2's one-line host summary,
`host metrics | cpu: 12% 45º | load: 1.23 | ram: 60% | gpu: 22% 1199/11538mb
| en0: ⇓ 1.2mb/s ⇑ 300kb/s | disk: ⇓ 512kb/s ⇑ 2.0mb/s /data 85%`
(thresholds colored like pm2; virtual/loopback NICs and idle segments
hidden; the throughput segments appear from the agent's second fast
sample on — about 2 s — since they are rates over its 1 s window).
`3pm internals` shows a
`host agent` row — the agent's pid and how many times the daemon had to
respawn it — and an `internal processes` table listing every process 3pm
runs for itself (the daemon, the agent) with pid, status, uptime, cpu,
RSS and respawns (`internal_processes` in `--json`; the agent's gauges
are sampled on the daemon's metrics tick and never carried over from a
previous incarnation). If the agent is down or wedged, the command fails fast with
`err{unavailable}` and a message naming `hostagent.log` — the daemon never
collects inline.

**How to get it.** Nothing to enable: the daemon spawns
`3pm __host-agent` at boot, restarts it with exponential backoff (1s →
30s) when it dies, and terminates it at `3pm kill`. The agent samples in
cost tiers, each on its own ticker: cheap gauges (cpu, memory, load,
PSI, NIC/disk counters) every second, walks and forks (process table,
socket table, sensors, `nvidia-smi`/`ioreg`, `statvfs`) every 10s, the
NTP probe and identity every 5 minutes — a hung GPU tool or a stalled
NFS mount never delays the cheap ones. `THREEPM_HOSTMETRICS_INTERVAL_MS`
(min 200), `THREEPM_HOSTMETRICS_MEDIUM_MS` and
`THREEPM_HOSTMETRICS_SLOW_MS` override the periods (each floored by the
previous). Its log is `~/.3pm/hostagent.log`. An agent that is alive
but no longer answers (stopped, deadlocked) is pinged every 5s
(`THREEPM_HOSTAGENT_HEALTH_MS`) and replaced after 6 misses in a row
(~30s); one answering with another protocol is left alone and named
once in `3pm.log` (run `3pm update`). On Linux, once a new binary was
installed over the old one, a respawn re-runs the daemon's own image
(`/proc/self/exe`), so it still works and the two stay the same
version until `3pm update`.

```console
$ 3pm ls | tail -1
host metrics | cpu: 38.8% 62.8º | load: 7.42 | ram: 63.5% | gpu: 22% 1199/11538mb | en0: ⇓ 1.2mb/s ⇑ 300kb/s | disk: ⇓ 512kb/s ⇑ 2.0mb/s
$ 3pm hostmetrics | head -3
 box — Linux 6.8 (x86_64)
┌─────────┬───────────────────────────────────┐
│ sampled ┆ 2026-08-28T15:02:48Z (3s ago)     │
$ 3pm internals | grep 'host agent'
│ host agent  ┆ pid 50632, 0 restarts │
$ 3pm internals | tail -6
 internal processes
┌────────────┬──────────────────┬───────┬────────┬────────┬──────┬────────┬───┐
│ role       ┆ command          ┆ pid   ┆ status ┆ uptime ┆ cpu  ┆ mem    ┆ ↺ │
│ daemon     ┆ 3pm __daemon     ┆ 50631 ┆ online ┆ 2D     ┆ 0.4% ┆ 26.7mb ┆ 0 │
│ host-agent ┆ 3pm __host-agent ┆ 50632 ┆ online ┆ 2D     ┆ 0.2% ┆ 35.9mb ┆ 0 │
└────────────┴──────────────────┴───────┴────────┴────────┴──────┴────────┴───┘
```

**Where it comes from.** Host-wide collection (`/proc` parsing,
`sysinfo`, an NTP probe, `nvidia-smi`/`ioreg` subprocesses) is the code
most exposed to platform edge cases and third-party leaks; running it in
its own process means a panic, a leak or a hang there can never take the
supervisor — and your apps — down. Rates (`rx_bytes_per_s`, `tx_bytes_per_s`,
`errors_per_s` per NIC; `read_bytes_per_s`, `write_bytes_per_s` per
mount; `rate_window_ms`) are derived agent-side from two consecutive
fast samples, so `--json` and a future fleet poller get them too; the
fast collector keeps its previous CPU ticks so only its very first
sample pays the 200ms warmup. `sampled_at_epoch_ms` carries each tier's
stamp (the overview adds `sampled (medium)` / `sampled (slow)` rows
only when they lag). The `ls`
line is best effort under a 500ms budget: no agent, no sample yet or a
wedged agent means no line, never an error. The agent speaks the daemon's own
NDJSON RPC on `host.sock` (`ping` + `host_metrics`), the daemon's
`Request::HostMetrics` handler is a pure proxy with a 2s connect window
(rides out a respawn) and a 3s request bound. Lifecycle mirrors the
daemon's: `flock` singleton (`hostagent.lock`), readiness byte, pid file,
self-termination when its parent or the home directory disappears.
Details: OV `3pm-docs-archived/architecture` §8.2.

## Batteries in `3pm hostmetrics` (`batteries[]`, `on_ac`)

**What you see.** On a laptop or a tablet, `3pm hostmetrics` gains a
`batteries` section — one row per pack, titled with whether the host is
on mains — and `3pm ls` gains a `bat:` segment on its host line, red
under 10% and yellow under 25% (a charge is the one gauge on that line
where *low* is the bad end), with a `⚡` while charging. Nothing appears
on a machine without a battery.

```console
$ 3pm hostmetrics | tail -6
 batteries — on mains
┌───────────────────────┬──────────┬────────┬────────┬─────────────┬───────────────┬────────┬──────┐
│ battery               ┆ state    ┆ charge ┆ health ┆ energy      ┆ power         ┆ cycles ┆ temp │
│ BAT1 (Li-ion M1009169)┆ Charging ┆ 84%    ┆ 99%    ┆ 37.3/44.5Wh ┆ 15.9W @ 8.71V ┆ 65     ┆ N/A  │
└───────────────────────┴──────────┴────────┴────────┴─────────────┴───────────────┴────────┴──────┘
$ 3pm ls | tail -1
host metrics | cpu: 8.1% 44º | load: 0.42 | ram: 51% | bat: 84% ⚡ | wlp1s0: ⇓ 12kb/s ⇑ 4kb/s
$ 3pm hostmetrics --json | jq '.batteries[0].power_w, .on_ac'
15.918
true
```

**How to get it.** Nothing to enable, and nothing to run as root: the
host agent reads `/sys/class/power_supply` on its medium tier, next to
sensors and the GPU tools. Linux only — macOS would need IOKit and the
section stays absent rather than faked.

**Where it comes from.** The attributes are world-readable files, so the
collector is a directory walk. What it has to get right is that hardware
disagrees about what it exposes: the kind of a supply comes from its
`type`, never its name (`BAT0`, `ACAD`, `axp20x-battery`,
`macsmc-battery`); a `scope=Device` supply is a mouse or a headset and is
skipped, while an **absent** `scope` means System (inverting that drops
the real pack on most laptops); a gauge that reports amp-hours has no
`energy_now` at all and is converted through its voltage; the draw is
carried unsigned because gauges disagree on the sign, with `status`
giving the direction; and `temp` is in tenths of a degree and usually
missing.

Above all, **an absent reading is absent, never zero**. A full pack on
the dock draws a real `0 W` and a pack without a sensor has no
temperature: collapsing the two is exactly how a charged battery comes
out reading empty. Every field is an `Option` on the wire and renders
`N/A`, and health (`full_wh / design_wh`) is left unclamped — a fresh
pack can legitimately hold more than its design capacity. Field list and
platform blanks: `docs/metrics.md`, *Power*.

## 3pm's own metrics in `3pm hostmetrics` (`3pmDaemon`)

**What you see.** `3pm hostmetrics` ends with two tables about 3pm
itself: `3pm processes` — the daemon and the host agent with pid,
uptime, cpu, RSS, open descriptors and threads — and `3pm runtime` — the
daemon's counters: version, uptime, `boot_id`, handoffs; apps by status;
restarts (unstable ones in yellow); armed timers and bus subscribers;
log sinks, rotations and bytes written; `RLIMIT_NOFILE`; agent respawns;
and when the part was sampled. Errored apps turn the `apps` row red.
`--json` carries it as `3pmDaemon: {at_epoch_ms, processes[], runtime?}`,
so anything that already reads the host snapshot — the fleet link
included — gets 3pm's own health for free.

```console
$ 3pm hostmetrics | tail -22
 3pm processes
┌────────────┬───────┬────────┬──────┬────────┬─────┬─────────┐
│ role       ┆ pid   ┆ uptime ┆ cpu  ┆ mem    ┆ fds ┆ threads │
│ daemon     ┆ 50631 ┆ 2D     ┆ 0.4% ┆ 26.7mb ┆ 41  ┆ 9       │
│ host-agent ┆ 50632 ┆ 2D     ┆ 0.2% ┆ 35.9mb ┆ 18  ┆ 7       │
└────────────┴───────┴────────┴──────┴────────┴─────┴─────────┘

 3pm runtime
┌────────────┬─────────────────────────────────────────────────────┐
│ daemon     ┆ v0.9.3, up 2D, boot 0x1f3a, 1 handoff               │
│ apps       ┆ 12 (11 online, 1 errored)                           │
│ restarts   ┆ 37 (2 unstable)                                     │
│ timers/bus ┆ 4 armed, 1 subscriber                               │
│ logs       ┆ 12 sinks, 3 rotations, 1.1gb written; 3pm.log 4.0kb │
│ nofile     ┆ 1024 / 1048576                                      │
│ host agent ┆ 0 restarts                                          │
│ sampled    ┆ 2026-08-28T15:02:40Z (11s ago)                      │
└────────────┴─────────────────────────────────────────────────────┘
$ 3pm hostmetrics --json | jq '."3pmDaemon".processes[0].mem_bytes'
26700000
```

**How to get it.** Nothing to enable. The section rides the medium
period (`THREEPM_HOSTMETRICS_MEDIUM_MS`, 10s); it is absent until the
tier has landed once and from a daemon predating it.

**Where it comes from.** The host agent measures 3pm the way it measures
the box — from the outside. Two sources, kept apart so either can fail
alone: the OS for the process rows (the daemon is the agent's parent
pid, the agent is itself; descriptors and threads of a foreign pid are
Linux-only, on macOS the daemon row's `open_fds` comes from the daemon's
own count and `threads` is absent — never faked), and the daemon's
`Request::Internals` over `rpc.sock` for the counters nothing else can
see, reduced to a health subset (no paths, no configuration — it leaves
the box). One connection per tick, 2s bound: a daemon that is down,
wedged, mid-`3pm update` or version-skewed blanks `runtime` for that
tick (`counters unavailable` in the table) while the rows keep coming;
the previous part is served meanwhile. The part never delays the first
snapshot, and `3pm internals` keeps its own 1s self-sample — it does not
depend on the agent. Details: `docs/hostagent.md` → *3pm's own metrics*,
OV `3pm-docs-archived/architecture` §8.2.

## Extended Monitoring (`3pm hostmetrics --extended`, PSS in `3pm show`)

**What you see.** `3pm hostmetrics --extended` swaps the overview for the
kernel's own counters: one table per family (`cpu`, `meminfo`, `vmstat`,
`netstat`, `cpuidle`, …, ~1 200 values on a 32-core Linux box), one line
per row and field, the family's kind in its title, and last the sources
that could not be read and why (`permission`, `error`, eBPF's
`not_root`…). A family only partly readable says `(partial: <reason>)`
in its title. `3pm show` puts the PSS sum beside the RSS sum, and its
` Processes` table gains `pss` (Linux) or `footprint` (macOS), `threads`
and `fds` — `-` for a member that could not be read. The other per-process
counters (faults, I/O, context switches) are in `--json`.

```console
$ 3pm hostmetrics --extended | head -12
 labomatic — Ubuntu Linux (Ubuntu 20.04) (5.4.0-153-generic, x86_64)
 extended monitoring — 23 families, 1228 values

 cpu — counters since boot, 32 rows
┌─────┬────────────┬────────────┐
│ row │ field      │      value │
├─────┼────────────┼────────────┤
│ 0   │ guest      │          0 │
│ 0   │ guest_nice │          0 │
│ 0   │ idle       │ 4031282977 │
│ 0   │ iowait     │     143096 │
$ 3pm show tree | grep -E ' mem |pss|sleep'
│ mem               │ 4.4mb RSS · 949.0kb PSS (3 processes, sums) │
│ pid     │ ppid    │ name  │ uptime │ cpu  │ mem     │ pss     │ threads │ fds │ ports │
│ 1316237 │ 1316236 │ sleep │ 4s     │ 0.0% │ 580.0kb │ 149.0kb │ 1       │ 5   │       │
```

**How to get it.** On by default (`[hostmetrics] extended` in
`config.toml`, `THREEPM_HOSTMETRICS_EXTENDED` above it). With the mode
off, `--extended` says so in one line and `show` is exactly what it was.
On macOS the host map is present and empty (`--extended` says the
platform has no extended source); per process, macOS has `footprint`
instead of PSS. What it costs on the link, measured on edgecr4ft.net
(2026-09-29, two nodes side by side, same 4 apps, 14 min): 777 B/s up
with the mode on against 529 B/s off, **+248 B/s** — only the fields
that moved leave, compressed; `extended = false` is per node.

**Short-lived processes (eBPF).** A sampler never sees a process that
lives less than its period — on a busy Linux server, most of them. With
the agent running as root, two kernel tracepoints count every exec and
exit exactly: `shortlived` (execs, exits, exits under 1 s, total
lifetimes, exits whose exec was not seen since the agent started — a
fork without exec such as a Postgres backend or a prefork worker, or a
process older than the agent —, and what a full kernel map refused,
`by_comm_full` / `starts_full`), `shortlived_by_comm` (exits and lifetime
per process name — the kernel's 16-byte name, never a path — for the 10
busiest names of each 10 s window, the rest under `other`; only processes
whose exec was seen, so fork-only workers show in the totals alone) and
`shortlived_by_comm_max` (the longest life per name in the window). No
clang, BTF or headers on the node: one prebuilt object ships inside
`3pm`. Not root (the agent's `CapEff` holds neither `CAP_SYS_ADMIN` nor
`CAP_BPF` + `CAP_PERFMON`, checked before anything is loaded; those are
necessary, not sufficient — tracefs permissions, Ubuntu's
`perf_event_paranoid = 3` and, before Linux 5.11, `RLIMIT_MEMLOCK` can
still refuse at the try, with the same answer): `unavailable` says
`not_root`, and nothing is retried before the agent restarts; `load_failed` (with the reason in the daemon
log) or `bpf_disabled` otherwise. Cost: not measurable over 5 000
fork+exec, one 50 ms load.

```console
$ sudo 3pm hostmetrics --extended | grep -A11 '^ shortlived — '   # a root agent
 shortlived — counters since the agent started
┌──────────────────────┬──────────────┐
│ field                │        value │
├──────────────────────┼──────────────┤
│ by_comm_full         │            0 │
│ execs                │         2402 │
│ exits                │         3446 │
│ exits_under_1s       │         2200 │
│ exits_without_exec   │         1153 │
│ lifetime_ns          │ 315147170152 │
│ lifetime_under_1s_ns │  36298126357 │
│ starts_full          │            0 │
$ 3pm hostmetrics --extended | tail -3                              # the same host, agent not root
├────────────┼──────────┤
│ shortlived │ not_root │
└────────────┴──────────┘
```

**Where it comes from.** Families print in the collection table's order
(`threepm_daemon::host_extended_family_order`), each `<family>_gauge`
right after its family, a family the table does not name last; rows sort
naturally (`2` before `10`). Values are shown as read: bytes for
`meminfo` (its `HugePages_*` lines are page counts), whole numbers
plainly, fractions to three decimals, sysfs units untouched (µs, kHz,
m°C). Nothing is summarized or hidden — `--json` carries the same map.
Details: `docs/hostagent.md` → *Extended Monitoring*, `docs/metrics.md`.

## Top processes (`3pm hostmetrics`)

**What you see.** Three sections at the end of the host part of `3pm
hostmetrics`: `top processes — cpu`, `— memory`, `— network`, ten rows
each (`pid`, `name`, `user` when known, then the section's two figures), for every
process of the host, whoever owns it. `--json` carries the one table
they are cut from, `top_processes`.

**How to read it.**
- `cpu` is CPU time over the last window: 100 % is one core busy.
- Memory is headed `footprint` on macOS: what Activity Monitor calls
  Memory, compressed pages and graphics memory included, from one `top`
  the host agent keeps running (about 1.8 % of a core at 5 s). A
  process `top` has not listed yet (started since its last sample)
  shows `N/A` there and has no place in the memory section. If `top`
  cannot run, the column falls back to `rss`.
- It is headed `rss` on Linux: the resident set, where shared pages
  count in every process holding them. An app 3pm manages has its PSS
  in `3pm show`; this table does not.
- `in/s` and `out/s` are bytes per second (`kb` = 1 024 bytes). `N/A` is
  a direction the platform could not measure for that process, not
  silence. On Linux they are TCP only, and cover other users' processes
  only when the daemon runs as root; without `ss` there is no network
  section.
- A section lists only processes with a figure above zero; one with
  none is not drawn.
- A process appears once it has been seen twice, so the first five
  seconds after the daemon starts show no table, and a process that
  lives less than a period never shows.
- Names are the executable's, never a path or arguments.

**Turning it off.** `[hostmetrics] top_processes = false` in
`config.toml` then `3pm update`, or `THREEPM_HOSTMETRICS_TOP_PROCESSES=0`
in the daemon's environment then a daemon restart (`3pm kill`, then any
command: a handoff keeps the old environment).
`THREEPM_HOSTMETRICS_TOP_MS` moves the period (5 000 ms).
A linked node sends the table to its backend: the names and owners of
the heaviest processes of the box.

**Limits.** No per-process disk I/O, no history, no UDP on Linux. On
macOS a long name whose executable was deleted since it started (a
rebuilt binary still running) shows cut at 16 bytes.
Details: `docs/hostagent-tiers.md` §3.8, `docs/metrics.md`.

## Network traffic per app (`net ↓/↑` in `3pm ls`)

**What you see.** `3pm ls` has a `net ↓/↑` column: what each app
received and sent per second over the last five seconds
(`1.2mb/s 310kb/s`). `3pm ls --tree` gives each process its own, a
`×N` group the sum of its members. `3pm show` adds a `net` line with
the totals since the app started (`↓ 1.2mb/s ↑ 310kb/s (total 4.1gb /
880mb)`), and a `net ↓/↑` column to its ` Processes` table (empty for a
process that moved nothing; the first column to go on a narrow
terminal). `--json` carries `net` on every record and tree member
(`rx_bytes_per_s`, `tx_bytes_per_s`, `rx_bytes`, `tx_bytes`,
`partial`), and `3pm internals` says how it is measured. A linked
node sends it to its backend, which keeps the history
(`process.net.received` / `.sent`, per process group
`subprocess.net.*`).

```console
$ 3pm ls
┌────┬────────┬──────┬────────┬───┬────────┬──────────┬──────┬────────┬─────────────────┬───────┐
│ id │ name   │ pid  │ uptime │ ↺ │ status │ watching │ cpu  │ mem    │ net ↓/↑         │ ports │
├────┼────────┼──────┼────────┼───┼────────┼──────────┼──────┼────────┼─────────────────┼───────┤
│ 0  │ api    │ 4000 │ 2h     │ 0 │ online │ disabled │ 3.1% │ 84.2mb │ 1.2mb/s 310kb/s │ 8000  │
│ 1  │ worker │ 4012 │ 2h     │ 0 │ online │ disabled │ 0.3% │ 18.0mb │ 0b/s 0b/s       │       │
└────┴────────┴──────┴────────┴───┴────────┴──────────┴──────┴────────┴─────────────────┴───────┘
```

**How to read it.**
- An app's figure is what it exchanged **with the outside of the app**:
  its whole process tree, each socket counted once however many members
  hold it, and a connection between two of its own processes (Electron
  and its `node`, `npm` and its server) left out — each of its bytes
  would count twice, once per end. Traffic with another app or a local
  database counts. A member's own figure counts every socket it holds,
  those connections included, so the members may add up to more than
  their app (an inherited socket, a conversation inside the app) or less
  (below).
- Short connections count. On Linux the daemon reads the host's TCP
  sockets every 5 s (`sock_diag`, no fork, no root) and hears every
  socket's close from the kernel with its final counters; on macOS it
  keeps one `nettop` running, which reports the closed flows too.
- **Linux, one blind spot**: an *outbound* connection opened and closed
  between two walks of the app's processes (10 s; 30 s for an app
  holding thousands of sockets) is never seen alive, and a closed socket
  no longer says whose it was: it is not counted. A longer one is found
  at the next walk and counted whole. An inbound one (a server
  answering) is always the app's: its local port is one the app listens
  on. When several apps share that port (`SO_REUSEPORT`), it is nobody's
  in particular — `3pm internals` counts it as `unattributed`.
- **What it costs is your apps'**, not the host's: the kernel filters
  both what the daemon reads and the close notifications it hears on
  your apps' ports, so a busy nginx next door costs nothing; an app
  holding 10 000 connections costs about 1 % of a core.
- Linux counts TCP, "sent" being what the peer acknowledged; macOS
  counts what `nettop` does, TCP and UDP, loopback included.
- Empty cell: no figure yet (the first ten seconds of an app, or of the
  daemon) or no source on this platform — never a guessed `0`. `≥`
  before a figure: the kernel dropped some notifications that window
  (`3pm internals` counts them), the figure is a lower bound.
- The totals restart from zero with the app, and with a daemon update.
- The column is drawn once some app has a figure (not on a platform with
  no source, nor from an older daemon), and goes right after `ports`
  when the terminal is narrow.

**Limits.** No UDP on Linux; the outbound short connections above (and
a connection inside an app that opens and closes between two windows
counts once, on its server side); UDP inside an app not paired; nothing
on other platforms. eBPF would close the
Linux gaps, as root only: not done.
Details: `docs/metrics.md` ("Network traffic per app").

## Responsive `3pm ls` (table and host line fit the terminal)

**What you see.** The process table and the `host metrics` line under it
adapt to the terminal width instead of wrapping cells (`statu`/`s`,
`watch`/`ing`). Columns are dropped whole, lowest priority first —
`ports`, `watching`, `pid`, `uptime`, `↺`, `mem`, `cpu` — until the
natural widths fit; `id`, `name` and `status` are never dropped, and only
a very long `name` may still wrap (never below 8 columns). Widths are
terminal columns, so a full-width name (CJK, emoji) gets the column it
needs on screen. The host line sheds segments the same way: `disk`, then
NICs, `gpu`, `load`, `ram`; `cpu` always stays. Piped output (`3pm ls |
grep`) is not a tty and is never cut: every column, every segment, whole
names, no color — the same for `3pm fleet`'s listings. A
reader that stops early (`3pm ls | head`, `3pm logs | head -20`) ends
the command silently with exit 0 — every 3pm command survives a closed
pipe, streamers included.

```console
$ 3pm ls          # 60-column terminal
┌────┬──────┬──────┬────────┬───┬────────┬──────┬────────┐
│ id ┆ name ┆ pid  ┆ uptime ┆ ↺ ┆ status ┆ cpu  ┆ mem    │
╞════╪══════╪══════╪════════╪═══╪════════╪══════╪════════╡
│ 0  ┆ api  ┆ 4000 ┆ 2h     ┆ 0 ┆ online ┆ 0.3% ┆ 42.1mb │
└────┴──────┴──────┴────────┴───┴────────┴──────┴────────┘
host metrics | cpu: 12.3% 52.4º | load: 1.23 | ram: 60.1%
```

**Where it comes from.** `render/table_layout.rs` measures every
column's widest header/cell (one `Col::text` source shared with the
painter, so measuring and rendering can never disagree) and picks the
columns; `render/table.rs` pins each kept column to that width so
comfy-table never squeezes them. `render/hostline.rs::fit_segments`
drops ranked segments by `visible_width` (ANSI-stripped). Property test:
for any width ≥ 26 and any names, no rendered line exceeds the terminal.

## Live table: `3pm ls --watch`

**What you see.** pm2's `pm2 ls --watch`, without the blink: the table
(link line, orphan notices and `host metrics` line included — exactly
what `3pm ls` prints) is redrawn every 300ms under a `Last refresh:`
stamp, **in place, where the cursor is**. Whatever was in the terminal
above stays there; the counters, statuses and pids change under your
eyes; Ctrl-C leaves the last table on screen with the prompt under it.
`--interval <MS>` sets the period (100ms floor; 1 s by default with
`--tree`, the daemon's tick — a tree reply that overruns the frame's
budget is left out of that frame, with the host line and the orphan
notice). The terminal is
re-read at every tick, so the table follows a resize (columns dropped or
regained). Off a tty (`3pm ls --watch | head`) there is no cursor
control at all: frames follow each other, a blank line between them,
and a reader that leaves ends the command with exit 0. `--json --watch`
prints one JSON array per tick, one line each (NDJSON, like `logs
--json`). The command ends on Ctrl-C, a closed pipe, or a daemon that
goes away.

```console
$ 3pm ls --watch --interval 500
Last refresh: 2026-09-07T11:02:17+02:00
┌────┬──────┬──────┬────────┬───┬────────┬──────┬────────┐
│ id ┆ name ┆ pid  ┆ uptime ┆ ↺ ┆ status ┆ cpu  ┆ mem    │
╞════╪══════╪══════╪════════╪═══╪════════╪══════╪════════╡
│ 0  ┆ api  ┆ 4000 ┆ 2h     ┆ 0 ┆ online ┆ 0.3% ┆ 42.1mb │
└────┴──────┴──────┴────────┴───┴────────┴──────┴────────┘
host metrics | cpu: 12.3% 52.4º | load: 1.23 | ram: 60.1%
```

**Where it comes from.** `render/watch.rs::Screen` is pure: given the
frame text and the terminal size it decides how to reach the previous
frame — cursor up by the previous frame's height (`ESC[nA`), every line
closed by an erase-to-end-of-line, the last by an erase-below — and
remembers the height and width it painted for. Two cases cannot climb:
a frame taller than the screen (the terminal scrolled: repaint from the
top of the visible screen, pm2's shape minus the blink) and a width
change (the old lines rewrapped: print a fresh frame below, resume in
place from it). The climb is counted in terminal rows, not lines: a
failure note under the table carries the raw error text and may wrap,
and every row it occupies is climbed (`rows_of`: visible width over
columns) — counting lines instead leaked one stale `Last refresh:`
above the frame at every tick. `commands/ls.rs` composes one `Listing` per tick — the same
body as a plain `ls`, so the two can never disagree — and, when the
host-line reply timed out (the connection may still carry it),
reconnects instead of reusing the client.

## Process trees: an app is every process it spawned (`3pm ls --tree`)

**What you see.** The `cpu` and `mem` of an app are those of its whole
**process tree** — the spawned pid and every descendant — not of the
spawned pid alone: an `npm run dev` that starts Electron and a dozen
helpers reads `62.2% / 3.0gb`, not `0% / 71mb`. `3pm ls` writes `saw
+13` when an app has other processes; `3pm ls --tree` (`-t`) draws them
under the row, compacted — a chain of single children is one line (`npm
› bash › node › Electron`), twins are grouped (`Electron Helper ×4`) —
and `--all` unfolds everything, one row per process. `⚠` marks a
process that left the app's process group, which `3pm stop` will not
reach. `3pm show` says `cpu 62.2% (14 processes)`, `mem 3.0gb (14
processes, RSS sum)` and lists every process under ` Processes`.
`--json` gains `proc_count` and `tree_rev` on every record and, with
`--tree` or in `show`, a `tree` object per app. The resource guards
judge the tree too (`limit_scope`, see *Resource limits*).

```console
$ 3pm ls --tree
┌────┬──────────────────────────────────┬───────┬────────┬───┬────────┬──────────┬───────┬─────────┬────────────────┐
│ id │ name                             │ pid   │ uptime │ ↺ │ status │ watching │ cpu   │ mem     │ ports          │
├────┼──────────────────────────────────┼───────┼────────┼───┼────────┼──────────┼───────┼─────────┼────────────────┤
│ 1  │ saw +13                          │ 12865 │ 5m     │ 2 │ online │ disabled │ 62.2% │ 3.0gb   │ 5002,6789,9222 │
│    │ └─ npm › bash › node › Electron  │ 12888 │ 5m     │   │        │          │ 1.4%  │ 323.1mb │ 5002,9222      │
│    │    ├─ Electron Helper (Renderer) │ 12963 │ 5m     │   │        │          │ 19.4% │ 777.0mb │                │
│    │    ├─ Electron Helper (GPU)      │ 12964 │ 5m     │   │        │          │ 10.2% │ 131.0mb │                │
│    │    ├─ Electron Helper ×4         │       │ 5m     │   │        │          │ 0.3%  │ 230.0mb │                │
│    │    ├─ ONYX Instrument Host       │ 12975 │ 4m     │   │        │          │ 25.1% │ 1.3gb   │ 6789           │
│    │    ├─ node                       │ 12990 │ 4m     │   │        │          │ 0.8%  │ 105.0mb │                │
│    │    ├─ python                     │ 12991 │ 4m     │   │        │          │ 0.6%  │ 121.0mb │                │
│    │    └─ ONYX Plugin Host ⚠         │ 13002 │ 4m     │   │        │          │ 4.4%  │ 90.2mb  │                │
└────┴──────────────────────────────────┴───────┴────────┴───┴────────┴──────────┴───────┴─────────┴────────────────┘
$ 3pm show saw | sed -n '/^ Processes/,$p'
 Processes (14, RSS summed: 3.0gb, cpu 62.2%)
┌───────┬───────┬────────────────────────────┬────────┬───────┬─────────┬───────┬────────────┐
│ pid   │ ppid  │ name                       │ uptime │ cpu   │ mem     │ ports │            │
├───────┼───────┼────────────────────────────┼────────┼───────┼─────────┼───────┼────────────┤
│ 12865 │ 4242  │ npm                        │ 5m     │ 0.0%  │ 71.3mb  │       │            │
│ 12880 │ 12865 │ bash                       │ 5m     │ 0.0%  │ 2.0mb   │       │            │
│ 12887 │ 12880 │ node                       │ 5m     │ 0.4%  │ 47.8mb  │ 5002  │            │
│ 12888 │ 12887 │ Electron                   │ 5m     │ 1.0%  │ 202.0mb │ 9222  │            │
│ 12963 │ 12888 │ Electron Helper (Renderer) │ 5m     │ 19.4% │ 777.0mb │       │            │
│ 12964 │ 12888 │ Electron Helper (GPU)      │ 5m     │ 10.2% │ 131.0mb │       │            │
│ 12965 │ 12888 │ Electron Helper            │ 5m     │ 0.2%  │ 61.0mb  │       │            │
│ 12966 │ 12888 │ Electron Helper            │ 5m     │ 0.1%  │ 58.4mb  │       │            │
│ 12967 │ 12888 │ Electron Helper            │ 5m     │ 0.0%  │ 56.1mb  │       │            │
│ 12968 │ 12888 │ Electron Helper            │ 5m     │ 0.0%  │ 54.5mb  │       │            │
│ 12975 │ 12888 │ ONYX Instrument Host       │ 4m     │ 25.1% │ 1.3gb   │ 6789  │            │
│ 12990 │ 12888 │ node                       │ 4m     │ 0.8%  │ 105.0mb │       │            │
│ 12991 │ 12888 │ python                     │ 4m     │ 0.6%  │ 121.0mb │       │            │
│ 13002 │ 12888 │ ONYX Plugin Host           │ 4m     │ 4.4%  │ 90.2mb  │       │ ⚠ detached │
└───────┴───────┴────────────────────────────┴────────┴───────┴─────────┴───────┴────────────┘
 ⚠ ONYX Plugin Host (13002) left the process group: `3pm stop` will not reach it
$ 3pm ls --json | jq '.[0].proc_count'
14
```

**How to get it.** Nothing to enable: every app has it once the daemon
runs a version with trees (`3pm update`; an older daemon renders `ls`
and `show` without trees and says `run \`3pm update\``). `-t`/`--tree`
and `--all` on `ls`, with `--watch` too (its period becomes 1 s, the
daemon's tick). `THREEPM_PROCESS_TREE=off` in the daemon's environment
restores the spawned-pid-only meaning for every app; `limit_scope =
"leader"` per app keeps the guards on the spawned pid while the figures
stay the tree's (an app saved by an earlier 3pm, in `dump.json` or
across `3pm update`, reads back as `leader`: an upgrade never widens a
limit). Which processes are an app's, what cannot be seen
(another user's processes, the environment of Apple platform binaries
on macOS, a nested daemon), the caps and every field: `docs/metrics.md`
→ *Process trees*.

**Where it comes from.** The daemon's metrics tick reads the process
table once per second (`sysinfo`, cpu and memory in one pass, off the
runtime; the whole table every thirtieth tick, in between only the pids
born or gone, the leaders, the members and 3pm's own — `metrics/table.rs`)
and attributes every process to an instance by, in order, the cgroup
leaf (`cgroup_enforce`), the parent pid, the leader's real process
group (orphans reparented to a subreaper), the memory of the members
already seen, keyed by `(pid, start time)` under `(id, gen)`, and the
environment the spawner injected (`THREEPM_ID`/`THREEPM_GEN`/
`THREEPM_CTL_SOCK` and the daemon's lineage `THREEPM_BOOT` — the classic
double-fork daemon, never a survivor of an earlier daemon of the same
home); then every descendant of a member, and never a process older
than the leader, nor 3pm's own daemon or host agent — `metrics/tree.rs`,
pure and table-tested, with a bench on a thousand rows. The totals land
on the record through `apply_metrics`; the tree itself stays on the
runtime and travels only on request (`process_trees {ids, gen?}`, 256
lines per tree and 2 000 per reply shared between the trees that list
members, `omitted` counting the rest, the totals always over every
member) — `supervisor/trees.rs`. The policy
receives both figures (`MemorySampled{gen, tree_rss, leader_rss}`) and
picks by `limit_scope` inside `step`, so the decision stays pure. The
CLI compacts through `threepm_protocol::tree::compact` (pure, in
`threepm-protocol` for the fleet dashboard to reuse; proptest: sums
conserved, every pid in exactly one row, `⚠` never hidden) and renders
in `render/tree.rs` (`custom_styling` for the dimmed `+N`,
`unicode-width` for the cut at the column's width, labels widening it
by up to 32 columns); snapshots from 40 to 120 columns, e2e `tests/e2e_tree.rs`
on a real shell with two sleeps. `3pm internals` shows `last metrics
tick` (`metrics_tick_ms`): about 12 ms for a thousand processes.

**A stuck process does not stop the sampler.** Reading the environment
of a process (Linux `/proc/<pid>/environ`) or its `io` /
`smaps_rollup` (Extended Monitoring) waits on the process's own
memory-map lock: on a process stuck in `D` state (a hung NFS mount, say)
the read never returns, and the whole tick — every sample, so
`max_memory_restart` and `max_cpu_percent` for every app — used to wait
with it, silently. These reads now run on a reader thread under a
500 ms deadline, within a 1 s budget per tick (`metrics/risky.rs`,
`metrics/guarded.rs`): a read that misses is retried on that process
with a backoff (1 to 60 ticks), not at all while its reader is still
stuck in the read, and the reads over the budget wait for a later tick
— meanwhile the process is not adopted through its environment and
keeps its last extended counters. The tick goes on, and at most four
readers are ever left stuck, daemon-wide, before both reads are
switched off until one returns (logged once each way). A probe not
made is no answer: the process is asked again on the next tick. The
refresh itself, and `process_start_time` on the supervisor, no longer
ask sysinfo for `io`, `exe` or tasks (on macOS `process_start_time`
skips sysinfo, whose refresh reads a new pid's `KERN_PROCARGS2`
unguarded — in the sweep, the stall report covers it). One sweep runs at a time; a sweep
that still runs past `max(5 × period, 5 s)` is logged as an error and
shown by `3pm internals`:

```text
│ last metrics tick     │ 12ms                                             │
│ metrics sweep         │ stalled for 83s — no samples, no resource guards │
```

(`DaemonInternals.metrics_stalled_ms`, counted from when that sweep
began, absent while healthy; the line goes when the sweep returns). The
loop keeps waiting for that sweep and stays cancellable. Tests:
`metrics/guarded_tests.rs` (deadline, reader replacement, the shared
cap, a panic told apart), `metrics/risky_tests.rs` (budget with reads
just over and under the deadline, backoff, quarantine while stuck),
`metrics/tests/hung.rs` (a real
tree with one member's read hanging: the others keep their figures; a
hung sweep reported stalled then recovered), the supervisor's
`internals_report_how_long_the_metrics_sweep_has_been_stalled`, and the
`internals` snapshots.

## Node.js client: `@3pm/api` (`packages/node`)

**What you see.** A zero-dependency Node library with pm2's programmatic
shape — `connect`, `start`, `list`, `describe`, `stop`, `restart`,
`delete`, `scale`, `save`, `launchBus` — promise-based, talking straight
to `rpc.sock` / `pub.sock`, auto-spawning the daemon when none is up:

```js
const { ThreePM } = require('@3pm/api');
const pm = await ThreePM.connect();
await pm.start({ script: 'app.js', name: 'api', instances: 2, max_memory_restart: '200M' });
const bus = await pm.launchBus({ process: 'api', subscribe: ['process:*', 'log:err'] });
bus.on('process:exit', (e) => console.log(e.exit_code, e.stderr_tail));
bus.on('log:err', (e) => console.error(e.line));
bus.on('daemon:shutdown', (e) => console.log('bye', e.reason));
```

Start options use pm2 names and units (`min_uptime`, `kill_timeout`,
`watch: true`, `interpreter` inferred from the extension and resolved to
an absolute path client-side, unknown keys rejected with a did-you-mean);
targets are ids, names or `all`. Bus events come as pm2-style aliases
(`log:out`, `process:online`, … — the two built from a field of the
event, `log.stream` and `process.kind`) *and* wire names, plus a raw
`event` catch-all so a newer daemon's kinds are never lost; the server-side
`EventFilter` handshake is used so unwanted events never cross the
socket. Daemon errors surface as `ThreePMError{code}`; `connect()`
checks `PROTOCOL_VERSION` and tells you to `3pm update` on skew.

**Where it comes from.** `packages/node/lib/{rpc,bus,spec,client}.js`;
`npm test` runs unit tests, decodes the frozen `fixtures/wire/v2`
fixtures of `threepm-protocol`, and drives a real daemon in an isolated
`THREEPM_HOME` (leak-proof, like the Rust harness).

## Modules

**What you see.** Optional features shipped with the install as
*modules* — directories with a `module.toml` — enabled per machine like
`pm2 install`: the bundled ones without any download, the others from
the download site. `3pm module list` shows what is
installed and what this 3pm bundles; `3pm install <name>` copies a
bundled module (or any local directory holding a manifest) into
`~/.3pm/modules/`, starts its `[[apps]]` and saves the process list —
on a module already installed it swaps the files and restarts its rows
in place (the sites it serves included; nothing is deleted and
recreated), and `3pm update` does the same for every installed module
whose bundled version changed, plus the apps running from the bundle
without an install (see *Zero-downtime update*); `3pm uninstall`
deletes its apps (and any app running from its files) and the
directory. `3pm <module> …` runs a module's command — the
installed copy, else the bundled one, so command-only modules need no
install at all — and unknown subcommands get a did-you-mean over
built-ins and modules. `3pm` passes `SIGINT`, `SIGTERM`, `SIGHUP` and
`SIGQUIT` on to the module and exits with its status; a `3pm` killed
outright (SIGKILL) leaves no orphan behind it — the module sees its
death as the EOF of an inherited pipe (`THREEPM_PARENT_FD`, every unix;
`PR_SET_PDEATHSIG` too on Linux) and stops through its own `SIGTERM`
path (`3pm fleet ssh -R` still closes the node's port). Module apps show in a second **Modules** table in
`3pm ls`, are targetable by name, and are skipped by `stop|restart|delete
all`, `delete --flush all`, `apply --prune-all` and `restart
--update-env`: install/uninstall own them.

**A module the release installs by itself.** `[module] auto = true` in
a **bundled** manifest means the module is part of 3pm the way a
built-in command is — `autoupdate` is the first one, so that any node
can be updated from the dashboard without someone ssh-ing in. 3pm then
does what `3pm install` would have done (copy, start the apps, save) at
the two moments it has a reason to look: on a daemon the CLI just
spawned — the first command on a fresh machine, and every boot without
a startup service — and after a `3pm update`, which is how a host
already running learns that a release added one. It says so once:

```console
[3PM] Module [autoupdate] installed (v0.32.0): it ships with this 3pm — `3pm uninstall autoupdate` if you do not want it
```

**And the host has the last word.** `3pm uninstall <name>` writes the
name into `<home>/modules.off` and nothing puts the module back — not
the next update, not the next boot — until `3pm install <name>` asks
for it again, which forgets the refusal. A module already installed is
left to the ordinary refresh; one that will not install is a warning,
never a failed command (`3pm ls` on a cold start still prints its
table); and a 3pm that predates the key ignores it.

**Modules that are not bundled come from the download site.**
`3pm install automonit` — a name that is neither a path, nor bundled,
nor installed — reads `<site>/modules/automonit/latest/version.json`
(`$THREEPM_CDN_URL`, `https://3pm.edgecr4ft.net` by default, the site
`autoupdate` reads), takes the archive built for this platform, checks
its size and its sha256 before opening it, unpacks it with every entry
checked (no absolute path, no `..`, nothing outside `automonit/`, no
link leading out, a `module.toml` that names `automonit`), and installs
it as `3pm install ./dir` would. `automonit@0.42.0` pins a version,
`automonit@latest` unpins, and `3pm update` takes a newer release of
every unpinned site module, in place; a site out of reach is a warning.
Only the configured site, only https (a `file://` or loopback mirror
aside), and the archive URL is rebuilt from the site, never taken from
the index. Downloads go through `curl`, else `wget`: no HTTP stack in
the binary.

```console
$ 3pm install automonit
[3PM] Downloaded automonit v0.42.0 (linux-x86_64) from https://3pm.edgecr4ft.net, sha256 verified
[3PM] Module [automonit] installed (v0.42.0) at /home/me/.3pm/modules/automonit
$ 3pm install automonitt
[3PM][ERROR] no module named `automonitt` (did you mean `automonit`?) — available: automonit; not on https://3pm.edgecr4ft.net either (cannot read https://3pm.edgecr4ft.net/modules/automonitt/latest/version.json: download failed (exit status: 22): curl: (22) The requested URL returned error: 404 Not Found); or pass a directory holding a module.toml
```

**How to get it.**

```console
$ 3pm module list
$ 3pm install serve               # or: 3pm install ./my-module
$ 3pm install automonit           # from the site; automonit@0.42.0 pins
$ 3pm serve ./dist 8080           # runs the bundled command even before install
$ 3pm uninstall serve
$ 3pm install --json serve        # {"module":{name,version,path,action},"affected":[…]}
```

```toml
# module.toml
[module]
name = "my-module"
version = "1.0.0"
command = "bin/my-module"         # optional: `3pm my-module …`

[[apps]]                          # optional: started by `3pm install`
name = "worker"
script = "bin/worker"
```

**Where it comes from.** `threepm-core/src/module.rs` (provenance,
naming rules, bundle roots), `StartSpec.module` (daemon-stamped, like
`source`), `threepm-cli/src/modules/` (manifest loader reusing the
ecosystem resolver, registry over installed + bundled, atomic install
copy, `3pm <module>` dispatch via clap's external subcommands),
`commands/module.rs`, `commands/module_refresh.rs` (the in-place
refresh shared by reinstall and `update`), `render/modules.rs`; daemon
shielding in
`supervisor/records.rs::resolve_mutation_target` and
`persist.rs::load_dump`. Contract and authoring guide: `docs/modules.md`.
Covered by unit/proptest at every layer, supervisor component tests
(`supervisor/tests/modules.rs`), `tests/e2e_modules.rs` with the fixture
modules under `tests/fixtures/modules/`, and snapshots of the Modules
table and `module list`.

## Static file server (`3pm serve`)

**What you see.** pm2's `serve`, as the first bundled module: a directory
served over HTTP by a supervised app that is *online* once its socket is
bound (`wait_ready`), listed with its port under **Modules** in `3pm ls`
(below the apps started by hand; `stop|restart|delete all` skip it, name
it to act on it), restarted like any app. Serving again on the same name
or port restarts the site with the new options. Without a port, 8080 or
the next free port above it is used. The command ends with the links to
open — `Local: http://localhost:<port>` and, when bound to every
interface, `Network: http://<LAN ip>:<port>` for a phone on the same
Wi-Fi — and a QR code of the network link (`--no-qr`). No process table.
`3pm serve page.html` serves the file's directory with that page as the
homepage, opens it in your browser (from a terminal; links only, no QR
code) and stays in the foreground: Ctrl-C stops the server and deletes the app (`--detach` to
leave it running; `--attach` gives a directory the same session
behavior, and the command also leaves when the app is stopped elsewhere). `--spa` answers unknown routes with the homepage (pm2's two-step
fallback: same basename at the root, then the homepage), `--ftp` lists
directories (index.html first, directories before files, dotfiles hidden,
a trailing-slash redirect so relative links work), basic auth challenges
with a 401, a `404.html` at the root replaces the plain 404, responses
carry pm2's CORS headers, HEAD works, other methods get a 405, bodies are
streamed. Path traversal is a 403 and a malformed percent-escape a 400
(GHSA-9prm-75wm-xpg6) — never a crash. A symlink that resolves outside
the served directory is a 403 too (and hidden from listings; one that
stays inside is served like its target). This is enforced by the open
itself, not by a check before it: the served directory is held open and
every file is resolved from that descriptor one component at a time
(`openat` + `O_NOFOLLOW`, `readlinkat` for the links met on the way,
`..` never past the root, an absolute target only when it spells a
location under the root — and then re-walked from the root, never
trusted), and the bytes streamed are read from the handle that walk
produced. Swapping a link between the router's decision and the read
changes nothing. A port already in use is reported
by the command instead of by a restart loop a minute later. Ecosystem
files written for pm2 keep working: `PM2_SERVE_*` env is honored.

**How to get it.**

```console
$ 3pm serve                              # ./ on 8080 (or the next free port), links + QR code
$ 3pm serve report.html                  # its directory, report.html as homepage, opened in the browser, Ctrl-C stops it
$ 3pm serve ./dist 8080 --spa --name site
$ 3pm serve --port 0 --ftp               # kernel-chosen port, directory listing
$ 3pm serve . --basic-auth-username admin --basic-auth-password s3cret
$ 3pm serve ./dist 8080 --host 127.0.0.1 --homepage app.html --namespace web
$ 3pm --json serve ./dist 0              # {"affected":[…]}
```

**Where it comes from.** `modules/serve` (own workspace and lockfile:
hyper 1, mime_guess, percent-encoding, base64): `options.rs`
(`THREEPM_SERVE_*` > `PM2_SERVE_*` > pm2 defaults), `router.rs` (pure
request → decision over an injected filesystem probe, property-tested to
never leave the root, lexically or through a symlink), `lookup.rs` (the
probe for real: a descriptor-relative walk with rustix — `openat`/
`readlinkat` — whose resulting handle is what gets served),
`listing.rs`, `server.rs` (hyper, streamed files,
clean exit on SIGTERM/SIGINT, readiness and `shutdown` through
`threepm_protocol::module::ctl`),
`start.rs` (the `3pm serve` side: `SpecPatch` → `Request::Start`,
busy-port check, pm2's output). Covered by unit/proptest per file and
`modules/serve/tests/e2e.rs` against a real daemon (real HTTP on a
kernel-chosen port: types, CORS, 400/403/404/405, escaping symlinks, SPA, ftp, auth,
`--json`, busy port, pm2 env).

## Fleet link agent (`3pm agent`)

**What you see.** The second bundled module links this 3pm to a fleet
backend (`3pm-backend`, the server side, its own repository). You paste one line from the
backend's bucket page — `3pm link` and `3pm unlink` are built-in sugar
over `3pm agent link` / `3pm agent unlink`: every argument is handed to
the bundled module untouched (`3pm link --help` is the agent's help),
the module owns the exit code, `--json` on either side of the verb
reaches it, and without an `agent` module the command fails with a
reinstall hint before any daemon is spawned:

```
$ 3pm link 3pmc_<cluster> 3pmk_<token> wss://fleet.example.com
$ 3pm link 3pmc_<cluster> 3pmk_<token>   # no server: the hosted backend, wss://fleet.edgecr4ft.net
$ 3pm link                    # the link: cluster, node, server, app, `link connected for 3m 2s`; exit 0 only when established
$ 3pm link stop               # pause: stop the agent app, keep the enrollment (saved, survives a reboot)
$ 3pm link start              # resume on the existing enrollment (registers the app again if it was deleted)
$ 3pm link status             # = 3pm agent status
$ 3pm link delete             # = 3pm unlink: disconnect, delete the app, forget the enrollment (also `rm`, `remove`)
$ 3pm unlink                  # = 3pm agent unlink
$ 3pm agent link 3pmc_<cluster> 3pmk_<token> wss://fleet.example.com
$ 3pm agent status            # cluster, node, server, dashboard, the app's row
$ 3pm agent unlink            # delete the app, save, forget the enrollment
$ 3pm agent status --json     # {"linked":…,"cluster_id":…,"node_id":…,"process":{…}}
```

`3pm link` alone (no positional, no `THREEPM_LINK_*` in the environment)
is a check: the status view plus a `link` line — `connected for 3m 2s
(session 2, server 0.1.0)`, with `, 3 unreadable frames` when this
backend and this agent have disagreed about a field (`frames_invalid`,
zero on a healthy link and then unsaid) — or `NOT established — <why>` (not linked, app
stopped or not registered, daemon unreachable, no session reported yet,
`link down for 12s (closed 4409 ERR_SESSION_REPLACED), retrying`,
`parked: the node was revoked`) — and exits 0 only when the link is up,
1 with `link not established: <why>` on stderr otherwise (`--json` adds
`session: {connected, for_ms, sessions, retries, parked,
last_disconnect?, server_version?, frames_invalid}`). The running agent writes
`<state dir>/session.json` on every change of its session and removes
it on a clean exit; the reader trusts it only when its `pid` is the
app's current one.

**The link line above the table.** A host that is linked opens `3pm ls`
with one line, pm2's `⇆ PM2+ activated | …` (a 3pm with no backend prints
nothing at all, and `--json` is untouched):

```
⇆ 3pm link | node 3pmn_wdcaepioyd6kflr5 | dash: http://127.0.0.1:9616/b/3pmc_dnhiue6kcwdsi4hn
⇆ 3pm link | node 3pmn_wdcaepioyd6kflr5 | link down for 5s (transport error), retrying
```

Green when the link is established, red with the reason when it is not —
the same verdict as bare `3pm link` (`threepm_link::local::status`, one
definition for the three readers). The address is the backend's page for
this cluster, derived from `link.toml`'s server (`ws://` → `http://`,
`wss://` → `https://`, path `/b/<cluster-id>`); `3pm agent status` prints
it as its `dashboard` line and `3pm link <…>` ends on it, so the bucket
page is one click from the terminal. The CLI reads `link.toml` and
`session.json` itself — no module spawned, no extra request, the app's
row comes from the list the table just printed — and drops the address
first on a narrow terminal, the reason last.

`link stop` and `link start` never run on a host that is not linked
(they point at the pasted line instead); `stop` on an app that was
deleted by hand says so and points at `link start`, which re-creates it
from `link.toml`. `--json`: `{"stopped":true,"app":…,"affected":[…]}` and
`{"started":true,"app":…,"registered":bool,"affected":[…]}`.

A provisioning script sets `THREEPM_LINK_CLUSTER` / `THREEPM_LINK_TOKEN`
/ `THREEPM_LINK_SERVER` (/ `THREEPM_LINK_INSECURE`) instead — the
environment **wins** over the positionals. The server is optional: named
neither by the third positional nor by `THREEPM_LINK_SERVER`, it is the
hosted backend `wss://fleet.edgecr4ft.net`, so `THREEPM_LINK_SERVER`
alone points every link of a machine at another backend. `ws://` to a
host that is not loopback is refused with `ERR_INSECURE_SERVER` unless
`--insecure`; a malformed token is reported without echoing it. `--name`
(default `agent`), `--cap
metrics:read|proc:restart|proc:stop|proc:action` (repeatable, default
all; a `link.toml` written by an earlier build (`version = 1`) holding
exactly the first three reads as all, since `version = 2` the list is
literal; a remote `restart`/`stop` never reaches a module app, the agent
included), `--no-env-keys`, `--force` (enroll again over an existing
link). What a call from the backend may do on this host is that `--cap`
list and nothing else: it is declared in `hello.caps` so the backend can
show it, and enforced here — the backend grants no capability. The
supervised app is configured by `THREEPM_AGENT_STATE_DIR` (default
`<home>/agent`) and `THREEPM_AGENT_NAME`; the pace of the link is the
backend's (`welcome.pace`: the state tick, the application metrics and
the process trees, each mandatory, each floored at a second), so the
`THREEPM_AGENT_INTERVAL_MS` / `THREEPM_AGENT_FULL_EVERY_S` an older
`3pm link` left on the app are not read, whatever they say; identity and server live in `link.toml`,
never in env (`3pm show` prints env).

`link` generates an Ed25519 node key and enrolls it in one request:
`POST /api/v1/enroll` (JSON over the same HTTPS the fleet routes use,
ten seconds for the whole of it) carrying the cluster, the token, the
public key, what this host is, the link versions this build speaks and a
proof signed over the token id and that key. The answer names the node
(`node_id`, `state`, the link version both sides share, the server's
clock) or refuses with one `ERR_*` code and its HTTP status. Nothing is
written before the answer is checked: a version outside this build's
range, or clocks more than five minutes apart (`ERR_CLOCK_SKEW`, the
delta shown), stops there — the operator learns of a skewed clock at
`3pm link` time rather than at the first handshake. Then `node.key`
(owner-only) and `link.toml` under `<home>/agent`, the supervised
`agent` app started (`wait_ready`, this binary as `__run`) and the
process list saved. A refusal (`ERR_TOKEN_EXHAUSTED`,
`ERR_TOKEN_INVALID`, a dead port…) exits 1 with nothing written. The
node's identity is that key and nothing else: no certificate, no cluster
CA — the backend holds the public half on the node's row and checks
every proof against it, and a host that was linked before this keeps its
node and simply drops the two files it no longer needs. An existing link
is kept unless `--force` (a fresh key, hence a new node id; the app is
restarted in place): the same line again — same bucket, same server —
says `Already linked … — kept` and exits 0 (`{"kept": true, …}` in
JSON), a link to another bucket or server exits 1.

**Install and link in one line.** The installer takes the same words
after `link`, and runs `3pm link` with them once 3pm is installed (by
its full path, so the shell's PATH does not matter). Optional: without
`link` it only installs or updates. Re-run as the updater, the same line
keeps the link:

```
$ curl -fsSL https://3pm.edgecr4ft.net/install.sh | sh -s -- link 3pmc_<cluster> 3pmk_<token> wss://fleet.example.com
$ curl -fsSL https://3pm.edgecr4ft.net/install.sh | sh -s -- v0.35.0 link 3pmc_<cluster> 3pmk_<token>
$ curl -fsSL https://3pm.edgecr4ft.net/install.sh | THREEPM_LINK_CLUSTER=3pmc_… THREEPM_LINK_TOKEN=3pmk_… sh
```

A refused link leaves the install in place and exits 1 with the command
to retry; the token is never echoed. `distribution/install.sh`. A reusable or long-lived token yields a *pending*
node, said so on the command line.

**musl hosts (Alpine).** The same line installs there: the installer
detects musl (no `getconf GNU_LIBC_VERSION`, a musl `ldd` or loader)
and downloads the static `3pm-linux-<arch>-musl` tarball, modules
included; `THREEPM_LIBC=glibc|musl` overrides the guess. A musl build
updates to musl through the autoupdate module. Not on musl: the
logged-in-users host metric (no utmp, always 0) and the agent's glibc
malloc tuning.

**Where it comes from.** `modules/agent` (lib + bin, own workspace and
lockfile; deps on `threepm-core`/`threepm-protocol`/`threepm-link`):
`cli.rs`, `inputs.rs` (pure resolution, env > positionals, the insecure
guard), `options.rs` (`from_env`/`to_env` round trip), `state/`
(`StateDir`: `node.key` created 0600 before any byte lands, temp + fsync
+ rename for every file; the *shape* of the two readable files and the
verdict over them live in `threepm-link`'s `local` module — `LinkConfig`
TOML with unknown keys ignored, `SessionFile`, `not_established` — since
the CLI reads them too for its link line, the agent's own half of
`session.json` (the fleet bearer, the count of unreadable frames) being
flattened next to it), `identity.rs` (the key and
the two proofs it signs), `tls.rs` (clear text or `webpki-roots`;
`--insecure` over TLS refused until the hardening step), `transport/`
(bounded `WebSocketConfig`, sub-protocol `3pm.link` — no version in the
name: `link_min..=link_max` is the whole of it, and a peer outside that
range is a 400 at the upgrade — HTTP rejections mapped to `ERR_*`), `codec.rs` (one deflate stream per connection,
`Sync` flush per frame), `enroll.rs` (the POST and what it makes of the
answer, 10 s), `start.rs` (`SpecPatch` → `Request::Start` → `Save`),
`link_cmd.rs`, `status_cmd.rs` (pure `view`/`render` + `Describe`),
`unlink_cmd.rs` (`Delete` tolerant of NotFound, `Save`, remove),
readiness and `shutdown` through `threepm_protocol::module::ctl`,
`run.rs` (`__run`: ready as soon as the identity is loaded — a backend
outage never makes the app `errored` — then the session until a
shutdown), `transport/handshake.rs` (three frames — challenge → hello →
welcome, the signature riding the `hello` it authenticates — 10 s,
version range and ±5 min skew checked before signing),
`transport/writer.rs` (the one writer: every frame — the node's state
included — queues on one bounded channel and is never dropped for being
late; a tracked frame tells its sender whether the bytes went out on the
socket, and one nobody waits for any more is skipped; ping every 20 s,
flush every 5 s, a full write buffer is a dead link),
`transport/reader.rs` (never blocks: a `call` is handed to the session,
which acknowledges it as its first act — so a call nobody could take is
never acknowledged — `error` kept for the close, a tag this build does
not know skipped, a body it cannot read counted (`frames_invalid`) and,
when it was a `call`, answered `err malformed` instead of being left to
die on its deadline; 60 s of silence — pongs count — is a dead link), `session.rs` (connect → handshake → pump → classify →
`next_delay_ms` with full jitter; a stable session resets the attempt;
revoked parks with an hourly probe; shutdown sends `Close(Normal)` and
waits 2 s for the echo; `THREEPM_AGENT_LOG` picks the log level),
`node_state.rs` (the one task that carries the node, see "The node's
whole state as diffs on the link" below: one `state` frame every
`welcome.pace.state_s` seconds — a second at the least, the wire's own
floor — from `List` + `HostMetrics`, then `AppMetrics` and
`ProcessTrees` for what moved, paced by `pace.metrics_s` and
`pace.trees` (`off` / `shape` / `figures {every_s}`), counted in ticks
by `node_state/paces.rs`;
records are redacted before they are shaped; while the daemon is down
the frame still goes, with an empty table and no `daemon`, and nothing
is said gone), `stream.rs` (what that task is made of: the tick clock,
what wakes the loop — the tick, a session status change (a `welcome`,
the link up or down), `state_resync`, shutdown — and the writer's
confirmation of a frame, waited 5 s), `daemon.rs` (the long-lived daemon
client of the node-state task, and of each of the commands' pool, one
connection each: every call bounded to 3 s, reconnection 1 → 30 s),
`events.rs` (the daemon bus filtered to `process`, `daemon_shutdown` and
`app`, decoded leniently, forwarded as `emit` without the stderr tail; a
`daemon_shutdown` is forwarded and then the agent closes politely;
resubscription 1 → 30 s). Covered by unit tests per file (writer and
reader under paused tokio time), `modules/agent/tests/session/` (the
session in process against the mock: handshake, reconnection ≤ 3 s after
a cut, 4403 revoked ⇒ parked, 403 at the upgrade ⇒ probe on the stop
schedule, ack before reply, a `call` this build cannot read refused
`malformed` and counted, enrollment refused for its own reason
(token, clock skew, a backend of another link generation), polite close)
and `modules/agent/tests/e2e.rs`: a real 3pm dispatching to the staged
module against a mock backend (`tests/support/mock_backend.rs`, its own
thread, per-instance cluster, enrollment over HTTP, link route with the
node looked up from the `Authorization` header, every agent frame
recorded — a frame link 4 cannot read fails the test where it happens —
orders: cut / close with a code / push a frame or a raw line) — enroll,
app online and in `dump.json`, state files and modes, `status`, refusal
without state, pending + `--force`, unlink, and the running app: the
signed `hello` seen, reconnect after a cut with the same boot id, clean close on
`stop` (exit path, no SIGKILL), new boot id on `restart`, parked when
revoked, and the node's state with a real sleeper: ≥ 3 `state` frames
under one `gen` with a clock that never goes back, every section whole
and with its CRC the first time it is seen, the host and the table
rebuilt from the frames alone, the daemon's pid and protocol version,
the sleeper with its env key *names* and no env value / home path /
script path anywhere in the captured frames, `emit {process}` on
`restart`, `emit {daemon_shutdown}` then Close 1000 on `3pm kill`. The
shared wire vocabulary is `crates/threepm-link`.

**The backend side (`3pm-backend`, step 11).** `GET /api/v1/link` admits
a node before any frame flows: `Authorization: 3pm-node
<cluster-id>:<node-id>` — both public, a claim and not a credential —
names the row, whose key and state are read from SQLite on every
handshake (401 `ERR_NODE_UNKNOWN`, 403 `ERR_NODE_PENDING` /
`ERR_NODE_REVOKED`), then challenge → hello → welcome (4401 / 4426 /
4408 / 4400 on refusal), the `hello`'s signature verified against that
row's public key. The session feeds an in-memory hub: presence `online` →
`stale` (3 s without a frame) → `offline` (30 s, tunable), the last
snapshot per node, a 300-point `{t_s, cpu, rss_kb}` ring per process, a
second session of the same node closing the first 4409, frames over 1
MiB closed 1009, 60 s of silence closed. `GET /api/buckets/{id}/events`
is the dashboard's SSE feed — `init` (the same shape as `GET
/api/buckets/{id}`: bucket, advertise, nodes with their displayed state,
snapshots, history), then `node` / `telemetry` / `emit`, monotonic ids,
`Last-Event-ID` replay of the last 4 096 events or a `reset`. The
dashboard (step 12) is three embedded files: `/` lists and creates
buckets; `/b/<id>` mints a token into the line to paste, shows every
node with its displayed state and, per node, the process table at 1 Hz
with the raw `telemetry` JSON under a `<details>`; `reset` reloads, the
browser reconnects by itself. The backend's `tests/it/real_agent.rs`
proves the whole vertical with a real `3pm`. Node API (step 13): `GET /nodes`, `POST
/approve`, `POST /revoke` (the live session is closed 4403 and the agent
parks itself); a host that knocks while pending or revoked leaves
`last_auth_error` on its row, so the page shows who is waiting and one
click admits it. See OV `3pm-backend-docs-archived/backend`.

### The node's whole state as diffs on the link (`state`)

**What you see.** Nothing on the host. On the wire, **one mechanism**:
the agent sends one `state` frame a second carrying only what moved in
the node — the host, the process table, each instance's application
metrics, each process tree — and an empty frame when nothing did: the
heartbeat. Measured on two real 10-minute traces (payload bytes on the
link, everything included): **357 B/s** on a dev Mac with 11 apps,
**463 B/s** on a Linux server with 8 — where the host and the table
alone cost 525 and 755 B/s as whole frames. It replaces the three
streams the link carried before (`telemetry`, `app_metrics`, `trees` —
and `host_patch`, a fourth that never left development builds), each
with its own revisions, gap detection,
resync message and safety net: they are gone, and so is link version 1
— an agent and a backend on different sides of that line refuse each
other at the handshake with `ERR_LINK_VERSION` ("update 3pm").

**Under the hood.** The daemon hands over **complete states** and cuts
no diff (its `app_metrics` answer is always whole; `app_metrics_rev`
and `tree_rev` only say whether asking again is worth it); the agent is
the only diff engine; the backend has one reader.
`crates/threepm-link/src/node_state/`: a node's state is a set of
**sections** (`host`, `procs`, `app/<id>`, `tree/<id>`), each an
independent JSON document with its own revision — a gap resends one
section, a section's full state fits one frame (nothing is chunked).
`mod.rs` is the engine, schema-less: `NodeWriter` (cut against the
state last **committed**, commit on the socket writer's confirmation,
`cut_gone`, `resync(secs)`), `NodeReader` (a patch applies only under
its `gen` and onto `rev - 1`; a `StateGap` leaves the section
untouched; a full patch with a wrong CRC is accepted and turns the
check off until the next session). `json.rs`: the canonical form
(`null` members dropped), tables `{"~r", "~o"?}`, rounding through the
decimal text, JSON Merge Patch, CRC-32. `host.rs` and `shape.rs` turn
the typed snapshots into sections and back: the host keyed by a fixed
path table at 3 significant digits; the process table keyed by id
**without `uptime_ms`** — the start is worked out once per
`(id, gen, pid)` (`Starts`; `created_at_epoch_ms` is the row's
creation, not the spawn) and the reader's uptime is the frame's clock
minus it — and without `app_metrics_rev`; metrics keyed by `sid` and
histogram buckets by index, **never rounded**; trees keyed by pid.
Wire: `AgentFrame::State(StateFrame {ts, gen, secs})`,
`ServerFrame::StateResync {secs}`, `welcome.pace`, the `state_frame`
fixture family (fixtures under `tests/fixtures/link/v4/`).
`modules/agent/src/node_state.rs`: one task, one tick, one daemon
connection — `List` + `HostMetrics`, then `AppMetrics` for the
instances whose `(gen, app_metrics_rev)` is not the one the backend's
section was shaped from, and `ProcessTrees` for those with more than
one process whose `(gen, tree_rev)` moved (their figures when
`pace.trees` is `figures {every_s}`, their shape alone when it is
`shape`, nothing at all when it is `off` — and then the sections held
are said gone). Everything is owed in full on a new session, on
`state_resync`, and when the daemon was replaced; a section carries its
CRC on every full patch and once a minute, each section at its own
moment of it (the ones that move every tick are not all checked on the
same one); what is no longer there (an
instance deleted, a tree back to one process, the host agent down) is
said `gone`, but nothing is while the daemon itself is down; frames
stay under the 1 MiB cap, a section never split. Cost at the most a
node manages (20 instances at the 1 000-series cap, all moving): 3.5
MB/s of whole states from the daemon at 0.28 ms an answer, ≈ 60–70 ms
a tick in the agent, 176 KB/s on the link; 20 × 50 series: 3 ms,
8 KB/s. Design and measures: `docs/backend-agent/link-protocol.md` §6
(the R&D plan that produced them is no longer in the tree). Tests: `node_state::`
(engine, JSON primitives, shapers — proptests through text), the
agent's `node_state::` (one frame a tick, a frame not written, resync,
gone, the cap) and `tests/session/state.rs` (the mock backend rebuilds
the node from the frames alone).

### Link traffic by category (`traffic`)

**What you see.** `3pm agent status` has a `traffic` row — `↑ 612 B/s
1.4 msg/s · ↓ 40 B/s 0.1 msg/s · overhead ~42 B/s est. · host 310 est.,
extended 170 est., procs 70 est., logs 12 … (as of 40s ago)` — and
`--json` the whole table (`traffic.counters`, `traffic.rates`), read from
`<state>/traffic.json`, which the running agent rewrites **at most once a
minute** (never `session.json`; removed by an unlink). The backend gets
the same counters as the `traffic` section of the node's state, every 5 s
(`max(1, 5 / state_s)` ticks), only the counters that moved; the `logs`
row gains `B sent`.

**What is counted** (`modules/agent/src/link_traffic.rs`, process-wide
relaxed counters, kinds and directions from `threepm_link::traffic`):
WebSocket up, a frame's compressed payload (what the backend's
`rx_bytes` counts) — the `hello` included; a `state` frame split over
its patches pro rata of their JSON length (`extended` and `top` carved
out of `host`, tree members' `x` out of `trees`), one message per patch,
a heartbeat one `control` message; down, a Text frame's payload by what
it decoded to (`calls` or `control`); pings one `control` message of 0
bytes, pongs not at all. HTTP: the body, one message per request (every
retry) and per answer — `logs`, `artifacts` (profiles, dumps, ssh
recordings), `blobs` (the relay's legs, per chunk), `api` (the rest of
`FleetHttp`). The per-frame **total is exact, the split an estimate**:
measured on a real node (`rd/logs-bytes-counter/lot0-preview.md`), ±5–10 %
on host and procs, ±25–40 % on the carved-out sub-values. Overhead
(28 B a message, WS header + TLS record) is shown, never counted. A
`3pm agent fleet …` command line counts nothing that is shipped.

**Tests.** The agent's `link_traffic::` (a host+tree frame's shares sum to
the bytes written; heartbeat, pings, hello, a decoded server frame; the
file at most once a minute), `http::logs::` (a retried POST counts
twice), `node_state::traffic_cut`, `status_cmd` (the row, `--json`) and
`tests/session/traffic.rs` (the section arrives; Σ up WebSocket bytes ==
what the mock backend received).

### The agent's threads: four workers, sixteen for blocking work

**What you see.** `3pm-agent __run` idles at **five threads** on any
machine, instead of one worker per core — 33 of them on a 32-core host.
`modules/agent/src/runtime.rs` names both figures. **Four tokio
workers**, because a worker can be *in* a transfer (the relay's copy,
the `data.sock` server, the WebSocket writer) and a node answers up to
`CALLS_IN_FLIGHT_MAX` calls at once. And a **blocking pool capped at
sixteen** instead of tokio's 512: the module itself hands that pool one
kind of work, `commands::pack` (an action's file read by 64 KiB steps
and gzipped, one bounded pass per call, cancelled when the call goes
away), but the pool is shared with the libraries underneath, and DNS
lives there — `tokio::net::lookup_host` on a cold dial, and hyper-util's
resolver under every HTTP client. A `spawn_blocking` cannot be
cancelled, so a lookup against a silent nameserver holds its thread for
as long as the resolver takes; sixteen keeps a burst of those from
queueing the packing of every call behind them. Five at rest is not a
ceiling: the pool spawns its threads on demand and retires them ten
seconds later, so the worst case is 1 + 4 + 16. The doc comment on the
constant carries the inventory, and a test walks `src/` and fails if a
new caller appears without being added to it. The host metrics agent
did the same thing first, 38 threads to 8
(`crates/threepm-daemon/src/hostagent/mod.rs`).

**What it is not.** It is not the memory fix. The resident set that
used to follow the bytes relayed was glibc's arenas, and
`MALLOC_ARENA_MAX=2` on the agent's own spec is what settled it (*The
agent runs with `MALLOC_ARENA_MAX=2`*, `src/arenas.rs`); pinning tokio
to four workers was measured there and moved nothing. What this buys is
the threads: a footprint that does not depend on the host's core count,
and 58 MB less virtual memory on a 32-core box — 28 thread stacks of
2 MiB. Throughput is unchanged: the relay is bound by the network long
before it is bound by four cores.

## Update 3pm from the dashboard (`autoupdate`)

**What you see.** A third bundled module, `autoupdate`, whose only
surface is two custom actions: a node updates itself when someone calls
`update` on it — from the fleet backend, from `3pm trigger`, from
`io.call` — instead of someone ssh-ing in to paste the install line.

```console
$ 3pm trigger autoupdate            # what it declares (nothing to install: see below)
$ 3pm trigger autoupdate check      # {"current":"0.32.0","latest":"0.33.0","outdated":true,…}
$ 3pm trigger autoupdate update     # {"started":true,"pid":40881,"from":"0.32.0","to":"0.33.0",…}
$ 3pm trigger autoupdate update '{"version":"0.33.0"}'   # or a named one (newer only)
$ tail -f ~/.3pm/autoupdate.log     # what the installer said
```

`update` installs a **signed release** and nothing else. It downloads
`v<X>/release.txt` — the version, then the sha256 of every archive and of
that release's own `install.sh` — and `v<X>/release.sig`, its ed25519
signature, and checks it against the two public keys built into the
module (`RELEASE_KEYS` in `modules/autoupdate/src/manifest.rs`: the key
`just publish` signs with on the publishing Mac, and an offline rescue
key). The manifest must name the version asked for, which must be
strictly newer than the running one; then the archive for this platform
and `v<X>/install.sh` come down and must match their signed sha256. Only
then does it run that `install.sh` (written to disk, never piped into a
shell) with `THREEPM_ARCHIVE` pointing at the archive already checked, so
nothing is downloaded twice. A bad signature, another version, a wrong
hash, a target the manifest does not list: nothing runs, and
`autoupdate.log` says why. Whoever controls the download site can
withhold a release, not forge one. The installer replaces `~/.3pm/bin/3pm` and its bundled
modules and ends on `3pm update`: the daemon hands off to the new binary
in place, the apps keep running, the modules are refreshed. `check`
reads `latest/VERSION` from the same site, the file `just publish`
writes last, and compares it with the version these module files shipped
with. `$THREEPM_CDN_URL` moves both to another site (the knob
`install.sh` itself honours), and `curl` — or `wget` — does the fetching:
the module carries no HTTP stack, as the installer carries none.

**The reply does not wait for the update.** It cannot: the update's last
step restarts every module row, this app included, so a handler that ran
`install.sh` in the app's own process group would be killed by the
command it launched and the call would answer `gone`. So the action
*starts* an updater and answers at once with its pid and the log path.
The updater calls `setsid` first thing, leaves the app's process group,
and outlives the restart — 3pm's `killpg` deliberately does not follow a
process that left (*`3pm stop` does not chase a `setsid` escapee*), and
this module's app sets no `cgroup_enforce`, the one thing that would
take it along. A `flock` on `~/.3pm/autoupdate.lock` is held by the
updater process itself, so a second `update` while one runs is refused
with the log path rather than doubled, and a crashed updater leaves no
stale marker behind.

A version named by the caller is checked before it reaches anything
(`^v?[0-9][0-9A-Za-z.+-]{0,31}$`): `{"version":"1.2.3; reboot"}` is a
failed call, not a shell. The log is appended to, rolled to
`autoupdate.log.1` past 512 KiB.

**Every hour, by itself.** The app does not wait to be asked: it reads
`latest/VERSION` every hour and, when a newer 3pm is out, starts the same
updater the action would — to that exact version, so a release published
between the check and the download cannot change what gets installed.
The first check comes within five minutes of the app starting (a jitter,
so a fleet restarted together does not reach the site in the same second),
a check is skipped while an update runs, and every `just publish` reaches
the fleet within the hour. `3pm logs autoupdate` says what it decided
(`0.35.0 is out (running 0.34.0): updating (pid 40881)`) and an
`autoupdate.started {from, to}` event goes to the feed.

A published version is **tried once**: the version is written to
`<home>/autoupdate.last` before the updater starts, and never tried again
by the timer. `current` is the version of the module's own files, so an
update that does not change them — it failed, or `install.sh` could not
hand off a daemon run from another binary (a development checkout) —
would otherwise re-run the installer every hour. The next release is a new
version and gets its own attempt; the `update` action ignores the record,
because a caller who asks, forces. `THREEPM_AUTOUPDATE_EVERY` in the
daemon's environment changes the interval (`30m`, `90s`, bare seconds) or
turns the timer off (`off`); the actions stay either way.

**How to get it.** Nothing: `module.toml` says `auto = true`, so the
module installs itself (*A module the release installs by itself*,
above) — on a fresh machine with the first daemon 3pm spawns, on a host
already running at its next `3pm update`. The install is saved, so the
app comes back at the next boot, and `3pm update` refreshes it with the
rest. A host that does not want it says so once, with `3pm uninstall
autoupdate`; `3pm install autoupdate` brings it back.

**How it works.** `modules/autoupdate/`, a standalone workspace whose
whole runtime is `threepm-io` (the Rust SDK — the module is an ordinary
3pm app using `io().action`), `rustix` for `setsid`/`flock` and the
installer's own `curl`: `src/worker.rs` (the supervised app: init,
declare, `io().serve()`), `src/actions.rs` (what the two calls answer),
`src/updater.rs` (the detached role, the lock, the log), `src/schedule.rs`
(the hourly check: `every`, `decide`, `tick`) and
`src/release.rs` (the site, the versions, `curl`/`wget`). Covered by
unit tests per file and `modules/autoupdate/tests/e2e.rs` against a real
daemon, with a release site on disk — `curl` fetches `file://`, so the
e2e drives the real code path and downloads nothing, including the test
that restarts the app under a running updater and still sees the install
finish.

## Fleet RPC (`3pm call`, `io.call`, `3pm fleet`)

**What you see.** A custom action (`io.action`, `3pm trigger`, see
*Custom actions and profiling* above) can now take bytes and answer with
bytes, and can be reached on another node of the bucket a linked host
belongs to — not just on this machine.

**The wire and the actions.**

- An app calls an action on its own behalf (`io.call`), of this host or
  of another node of the bucket. A call of this host sends a file by
  path and nothing is copied, so the path is the contract: when the call
  is placed the daemon opens it and judges what it finds — no link, no
  fifo, no device, no directory, no relative path, no file of another
  account, and a size the action accepts — then hands the handler that
  very path. It is an entry control, not a promise about the file's
  future: a caller that replaces its own file afterwards changes only
  what its own handler reads. A call that carries bytes to another node
  names the instance they go to, and never a path of this host.
- An action declares what it accepts: `input {max_bytes, ext}`, `stream`,
  and `concurrent` (1 by default, 16 at most) — and the most its file may
  hold, `output {max_bytes}` (1 byte to 256 MiB): past it `commitFile`
  fails `too_large` with both sizes before a byte leaves, and a caller
  on the fleet reserves that much of its blob quota instead of 256 MiB. The daemon runs at most
  `concurrent` calls of one action per source, 16 per connection, and —
  counted apart — 16 streams or 32 plain calls per daemon; a refused call
  names the rule it met. A source that declares no `input` is never given
  one, and the refusal says so.
- `3pm call` / `io.call` place a call with bytes attached, on this host
  or on another node of the bucket. The bytes travel as a blob, never on
  the 1 MiB control line; the transfer budget is the size at 128 KiB/s,
  clamped to [60 s, 24 h].
- `3pm link` gives the daemon a call gate: the agent's own app line gets a
  32-byte secret in its environment at every launch (never in the spec,
  never in `describe`, never in a log), and only a control connection that
  sends it back opens the gate. No gate, no route for the bytes: a call
  that names a node is refused at once, and the calls in flight when it
  closes end `gone`.
- A `live` blob is framed: 20-byte big-endian header (`len`, `flags`
  KEY/END/INIT/DISCONT, `hdr`, `seq`, `ts_us`), messages of at most 1 MiB,
  a longer header accepted and skipped. Frozen vectors:
  `crates/threepm-protocol/tests/wire_samples/live_frames.json`.
- **What one stream carries over a distance.** The agent relays a leg
  over one TCP connection to the backend, which makes a second one to
  the node at the other end. Measured under `tc netem delay 30ms` —
  60 ms of round trip per hop — one stream carries **5 MB/s at the
  latency floor**: a hundred 50 KiB images a second arrive with a p50 age
  of **61.0 ms** and a p90 of 61.3 against a floor of 60, ten of 1 485
  lost; at thirty a second, p50 61.2, p90 61.5, nothing lost. Above that,
  what the kernel's send buffer allows is **up to about 43 MB/s where the
  path permits it** — the buffer Linux sizes for that round trip
  (2 626 560 bytes here) divided by it, read off the socket and never
  measured end to end; the highest measured is the 5 MB/s above, which is
  all the camera offered. Until this lot the agent pinned that buffer at
  256 KiB to keep a frame's latency down, and the same stream capped at
  **4 MB/s**: the same hundred images a second delivered 4 of an offered
  5, p50 216 ms, 18 % of them dropped. `TCP_NOTSENT_LOWAT` keeps the
  latency now (`modules/agent/src/data/outbound.rs`) and nothing keeps
  the ceiling. On a platform without that knob (macOS, the BSDs, Windows)
  the buffer and the 4 MB/s both stand. Both ends carry the change: a
  fleet only half updated is still capped by the other one. The cost is
  kernel memory and not resident memory — `sk_wmem_queued` against
  `tcp_mem`, three tuned TCP sockets per node (the link session and two
  warm relay connections), auto-tuning settling near twice the
  bandwidth-delay product rather than at the `tcp_wmem` ceiling.

**The Node SDK.**

- The Node SDK places calls: `io.call(node, app, action, params, {file,
  output, instance, source, timeout_ms})` on another node of the bucket
  or on this machine (`node: null`, where the file is taken by path and
  nothing leaves the host). Bytes travel as a blob over the agent's Unix
  socket, hashed before the call and checked on arrival; failures are
  typed (`e.code`: `node_offline`, `too_large`, `daemon_too_old`, …).
- An action declares what it takes — `io.action(name, {input: {max_bytes,
  ext}}, handler)` — and its handler gets the file ready as `ctx.input`;
  `await ctx.commitFile(ext)` sends its own output back the same way,
  bounded by `output: {max_bytes}` when the action declares it.
- Streams: `io.stream(name, opts, (session, params) => …)` and
  `io.open(node, app, action, params)` exchange framed messages over two
  legs, with `io.mjpeg(readable)` cutting JPEG images as they arrive.
  Writing never blocks the producer: past a 100 ms window the oldest
  message that is not a KEY is dropped and counted (`session.stats`).
- `io.node` is this host's own node (`{id, cluster_id, server,
  linked_at_ms}` from the agent's `link.toml`, `null` when not linked).
  A live session cannot stay on the host, so `io.open(null | 'self', …)`
  goes through the fleet to that node (`node_offline` when unlinked);
  `io.call('self', …)` does the same, `io.call(null, …)` stays local. A
  `CallError` prints as `code: message`.

**From the command line.**

- `3pm call <app> <action> [params]` places one call on one instance and
  waits for the one answer it owns: `--file` attaches a file of this host
  (nothing is copied — the handler is handed that very path), `--out DIR`
  puts whatever the action produced where you want it, and `--node`
  places the call on another node of the bucket. `--json` prints one
  document: `{call_id, ok, node?, results, output?}`.
- `3pm trigger --file` is the same call with the trigger's own report, so
  it reaches one instance (`--instance`, 0 by default) instead of every
  one; `3pm trigger --out DIR` needs no `--file` and delivers the file of
  every instance that left one. The listing (`3pm trigger <app>`, and
  `3pm show`) now says what each action takes: `kind` (`call`, `stream`,
  `×N` when several may run at once) and `input` (the size it accepts and
  the extensions it lists).
- `3pm fleet nodes | apps | inspect | call | top` is sugar over the
  bundled agent (like `3pm link`): the arguments travel untouched, the
  module owns the exit code, and 3pm refuses early and plainly when there
  is no agent module or no link to read this host's node identity from.

**Fleet identity.**

- A linked node gets its own bearer for the backend's HTTP routes in
  every `welcome` (`3pma_<10 hex>_<64 hex>`, re-minted per handshake,
  valid for the session plus a grace window): `3pm fleet …` and a
  node-placed call authenticate with it, so the owner's account token
  never has to sit on a server. The token never reaches a log — its
  `Debug` shows `3pma_<id>_<redacted>` — and the backend keeps only
  `sha256(token)` in memory, so a revoke bites on the next request.
- A remote action can carry bytes: the link names blobs
  (`action {input, output, stream}`), never the bytes themselves, and
  the file an action produced comes back as `output {blob, bytes, ext,
  sha256}` for the caller to pull.
- `call_result` reports a call THIS node placed on another node of the
  bucket; the calls that settled while the agent was away come back as
  `welcome.pending_calls`.
- `3pm fleet nodes` (alias `3pm fleet ls`) `| apps [node] | inspect
  <node> [app]` on a linked
  host, authenticated as *that node*: the fleet's hosts with their
  state and how many apps each runs, every process of the bucket (or of
  one node) with its cpu, memory and restarts, and one node in full —
  identity, capabilities, its own grant, its process table. `--json`
  everywhere gives a stable document, not the backend's own.
- The `nodes` table also says **whose** each box is and where it is
  reached: `user` (the account that node's daemon runs as), `public`
  (the address its link comes from, as the backend sees it) and
  `private` (its interfaces' addresses, container bridges left out).
  `—` means the backend did not say, which is what a node whose agent
  predates the columns reads. Every fleet table is `3pm ls`'s
  (`threepm-table`): box style, colored state, and on a narrow terminal
  the lowest-priority columns drop first — `os/arch`, `seen`, `apps`,
  `user`, `private`, then `public` for nodes; `pid`, `uptime`, `inst`,
  `↺`, `mem`, `cpu` for processes — while `node`, `host` and `state`
  (`name`, `status` and the node) stay. Only when those alone overflow
  is the `3pmn_…` id (the app name) cut, with a `…`, to no fewer than
  twelve (eight) columns. Widths are terminal columns, so a full-width
  name gets its on-screen width rather than a second line, and a
  box-drawing glyph in a name a neighbour chose prints as `·`, never as
  a border. A pipe or a file is never cut — every column, whole names,
  no color — like `3pm ls`; `--json` carries
  `user`, `peer_ip` and `private_ips` untruncated, plus `sessions` —
  the accounts logged in on that node — which the backend sends only to
  a node granted `fleet:read`, and `inspect` shows `user` on its card. A node
  granted nothing reads nothing: the first answer names the missing
  scope and where its owner grants it. With `fleet:read` the card also
  carries the host, and an app's own series and declared actions.
  `--ca-file <PATH>` adds a certificate authority to the built-in roots,
  for a private CA or a root this build is too old to know.

**`3pm fleet top`.**

**`3pm fleet top`** — the bucket live in two panels (nodes, then every process
of the fleet), fed by the backend's `events` stream with the node's own bearer
and `fleet:list` alone. The column names and units of `3pm fleet nodes` /
`apps`, fewer columns (no addresses, no `pid` or `uptime`),
redrawn once a second whatever the stream is doing; `q`, `Esc` and Ctrl-C
leave. Reopens the stream on a `reset` or a cut, takes one second chance at a
bearer the resident agent has since refreshed, and closes naming the missing
scope when the backend refuses for good. Refuses `--json` / `THREEPM_JSON` and
a stdout that is not a terminal, before opening anything.

**The data socket and remote calls.**

**The data socket** — an app of this host that calls another node never
speaks to the network. It `PUT`s its input and `GET`s its output on
`agent/data.sock`, with a token the daemon put in its ticket, and the agent
moves the bytes: TLS, the backend's address and the node's bearer stay in the
agent. A token is worth one blob in one direction until its call settles; it
is checked when the request opens and never again, so a day-long tail
survives a rotated one. Nothing is buffered and nothing is hashed on the way
through — a hundred megabytes cross for 240 KiB of the agent's memory — and
the head of a `GET` is flushed before the first byte, so the receiver knows
it is connected before the sender has started.

A call that names its target by **hostname** rather than by node id is
resolved in the fleet gate, and the answer is kept for ten seconds: one
name costs one read of the bucket instead of one per call, which is what
`io.open` pays inside its establishment latency. An id the backend says
it does not know is dropped at once, so a node that re-enrolled under
the same hostname costs exactly one refused call.

**What the socket is, and is not.** There is one `data.sock` per agent,
mode 0600 in a 0700 directory: only a process running as the daemon's
user reaches it. A root daemon that hosts users (`[users] allow`) adds
one socket per hosted user, `agent/data-<uid>.sock` in that user's
hosted home (`/var/lib/3pm/users/<name>/`), owned by the user, 0600, in
a directory root:gid 0750: the daemon names them on the fleet gate each
time it opens (`hosted_socks`), the agent binds them on the same server
as `data.sock` — same tokens, same relay — and the SDK of a hosted app
finds its own with no `THREEPM_DATA_SOCK`. The SDK finds the hosted home
under the default `/var/lib/3pm/users` only (or `$THREEPM_USERS_DIR`,
which a hosted app does not inherit): with a custom `[users] dir`, set
`THREEPM_DATA_SOCK` in the app's `env`. An app run as another account
by a daemon that does not host it (a non-root daemon's `user:`) still
has no data plane and places no call that carries bytes.
The data sockets keep 512 requests in flight **together**, `data.sock`
and the hosted ones sharing one descriptor budget: the hosted sockets
take at most half of it between them (256) and 128 each, so `data.sock`
keeps its half and one hosted user leaves the others theirs. A caller
past the ceiling reads `503`; at most 64 of those answers are written
at once, past which the connection is closed unanswered, and a failed
`accept` (out of descriptors) pauses 100 ms rather than spinning.
Connections that have not yet spoken a header are counted apart, and
past 64 of them the agent's patience for a head drops from ten seconds
to a quarter of one — a crowd of silent connections costs its author
forty times the reconnects, and never costs an honest caller its request. Under systemd,
give the agent `LimitNOFILE` (or `ulimit -n`) of at least 4096: the 512
plus the warm connections plus the link. The SDK keeps three agents:
every write, sized or chunked, goes on one with no socket ceiling that
keeps its connection alive (so a refusal that arrives mid-body keeps
its reason instead of dying with the reset); a read of a blob the
caller does not know settled takes a connection of its own; and the
keep-alive pool of eight serves what fits in one round trip. A `PUT` is
answered when its body ends, so it has no
header budget at all — sixty seconds without a byte moving on the way
out ends it instead. When the backend's name resolves to several
addresses the relay gives each a quarter of a second before the next,
so a published address that is not routed costs that and not the whole
dial; on a link slower than that round trip with several addresses, a
dial can cost up to half a second more, once. Opening a call is about
seven round trips cold (TCP and TLS, `nodeAction`, the relays composing
theirs, the head of the `GET`) and three warm; a `live` frame, once the
session is open, crosses the relay in two round trips and under half a
millisecond, which is the figure the plan quotes. The `?k=` a `viewBlob`
mints is a bearer for sixty seconds, bound to one blob and nothing else
— not an address, not a session.

**Concurrency.** An agent runs at most four plain commands at once and,
counted apart, sixteen sessions — the daemon's own
`STREAMS_PER_DAEMON_MAX`, so a session this agent admits is never one
the daemon then turns away. Each refusal names its own budget ("4
commands already in flight", "16 sessions already in flight");
`CALLS_IN_FLIGHT_MAX`, the wire figure the backend sizes a session's
blobs on, is unchanged at four. A session whose call is refused after
the ticket ends on that reply: both legs close, `session.stats` keeps
its last values, and `frames()` throws the backend's `CallError` —
waiting for the reading leg's idle guard used to turn every late refusal
into a sixty-second timeout.

**Streams across the fleet** — `io.open` on one node and `io.stream` on
another exchange framed messages through the same socket: a chunked
`PUT` for what a side sends, a `GET ?live=1` for what it receives, two
per side and no state anywhere else. The agent cuts the stream it
relays on **message** boundaries, never on a byte count, so a reader is
never handed half a header; and when the link to the backend cannot take
the frames as fast as the app writes them, the agent throws its oldest
ordinary messages away rather than queue them — keeping the `INIT`, the
last `KEY` and the `END`, which are what a reader joining later needs,
and never the only message it is holding, however big: the queue's
ceiling bounds a backlog, not a frame. That is the second abandon sink
of a live stream (the SDK's is the first, at the source): in `live`
there is no back-pressure, by design, and a hundred megabytes of frames
cross the agent for no more memory than a hundred megabytes of file.
For that sink to see the congestion, nothing below it may hold much: a
`live` leg's upload rides **a connection of its own kind**, never one a
file used, with 16 KiB unsent in the kernel (`TCP_NOTSENT_LOWAT`, Linux
and macOS; a 64 KiB `SO_SNDBUF` where it is missing or refused), one TLS
record in rustls and hyper's 8 KiB minimum write buffer — against
64 KiB, 64 KiB and ~400 KiB for a file, which keeps its throughput. On
Linux it also asks for BBR (taken when the agent may: `CAP_NET_ADMIN`
or BBR allowed), whose window does not fill a modem's queue. A LAN
neighbour's direct leg is served the same way. Measured in process
(Linux, 1 MiB/s offered to a 100 KiB/s reader): the oldest frame to
arrive was 380 ms late, against 1.14 s on a file's connection — 30 s at
the 5 kB/s of a congested hotspot. The backend dates a message on
arrival, so this is where a frame late from upstream can still be told
late and dropped.
Bytes that are not `live` messages, or that stop in the middle of one,
are the writer's own mistake and are refused `422` — never `503`, which
would blame a node that is answering perfectly well. A session's tokens
live as long as the session, the 24 hours the backend holds it open,
whether or not the backend named the deadline itself; and a session
never carries an `output`, because its answer is its second leg — the
SDK, the agent and the backend all refuse one.

In the Node SDK a session ends the way an application expects: `close()`
cuts the reading leg and a `for await` over it simply returns, `end()`
half-closes (sends `END`, keeps reading until the other side's), and
`send(readable)` feeds a source — `io.mjpeg(ffmpeg.stdout)` — and ends
the leg cleanly when that source ends, is stopped or is killed, never
with an uncaught error. A received frame can be written straight onto
another leg, flags and capture time intact. Because frames are dropped
at three places and only the first one is local, the reading side counts
what never arrived from the holes in `seq` (`s.stats.missing`), and
measures latency from each frame's capture stamp (p50/p95/max over the
last 256 frames, clock skew included between two hosts); the sink's
`drop` event lets a producer lower its rate the moment it is too fast.

**`3pm agent fleet call <node> <app> <action>`** — one call on another node
from the command line, with this node's own identity: `--file` sends bytes,
`--out` keeps what came back and checks its digest, `--timeout` bounds the
wait. It talks to the backend directly, because the command line has no data
socket for the daemon to mint it a ticket on, and the call shows as
`node:<id>/-`. A hostname is resolved here; `--stream` is not from
here. The output lands through a temporary
beside it that only this call can have made (`O_EXCL`): a link someone
else left at that name is refused rather than written through, and so
is the leftover of a crash.

**One pass over the bytes instead of three (`opts.sha256`, `input.stream`).**
A hundred-megabyte call spends most of its time on SHA-256 in JavaScript
and on a copy to disk, not on the road: measured end to end, the source is
read once by `io.call` to find its digest (about 200 ms), written once by
the callee into a staging file, and read back a third time by a handler
that wants the bytes. Two opt-ins, both off by default, remove two of
those passes and leave the wire untouched.

*The caller that already holds the digest* passes it: `io.call(node, app,
action, params, { file, sha256, bytes })` declares those instead of
reading the file to find them again. The size is checked against the file
— one `stat`, and it catches the one mistake this invites, a digest kept
after the file changed under it. The digest is taken at the caller's word:
verifying it is the very work being skipped, and the callee checks it
against what lands anyway. Both or neither, lower-case hex, refused as
`refused` otherwise.

*The handler that reads the body* declares `io.action(name, { input: {
max_bytes, stream: true } }, …)`: nothing is staged and
`ctx.input.stream()` yields the bytes once, as they arrive from the
agent. A handler that never calls it never fetches them at all. The
declaration the daemon reads is unchanged — what the caller may send is
the same — because the choice is about this process, not about the
contract. The body is read once, a second `stream()` being `refused`.

Three differences to know. **`ctx.input.path` is `null` on a call from
another node**, there being no file; a call placed on this host still
names one, and `stream()` reads the daemon's own file, so one handler
serves both. **The size and the digest the call declared are checked at
the end of the body**, so a truncated or altered input throws out of the
iteration rather than refusing the call before the handler ran — and
nothing has vouched for the bytes until that iteration ends, so a
handler that stops early has used unchecked bytes (the SDK says so on
its debug channel). **The transfer counts against the handler's
`timeout_ms`**, which starts before a byte has moved, where the default
path hands the handler its whole budget with the bytes already in.

### `3pm fleet ssh`: a login shell on a node

**What you see.**

```
$ 3pm fleet ls
$ 3pm fleet ssh envision-surface-3997     # or 3pmn_…, or a unique prefix
root@envision-surface-3997:~# htop        # full-screen programs, colours, resize
root@envision-surface-3997:~# exit 3
$ echo $?
3
```

The login shell of the target agent's own unix user (root on the
tablets), on a real pty, in its home. `~.` after Enter disconnects
(`~~` types a `~`); the exit code is the shell's, `128 + signal` when a
signal ended it, 255 when the session ended any other way, with one
line on stderr saying how. Who may open one: the bucket's owner, or a
node holding `invoke` on the target whose `callers` rules — when it
has any — name the action `3pm:shell` (app `-`); `*` never covers it.
A target refuses with `node:shell` taken out of the `caps` of its
`link.toml`. Every opening and end is written to VictoriaLogs by the
backend, every command line by the agent (`kind = shell`).

**How it works.** `nodeShell` places a call whose answer names two
`pipe` blobs, lossless and back-pressured: the command line `PUT`s the
keyboard into one and `GET`s the screen from the other (one reader,
never retried), the target the reverse; both carry `attach` messages in
`live` framing. Either leg ending ends the session — the target hangs
the shell up, the command line exits 255. The command line pings every
20 s (a keyboard leg silent for 90 s is a dead one to the target), and
never awaits the keyboard leg: messages queue, the terminal is read no
more past 64 of them, and the screen keeps flowing, so a paste into a
shell that is not reading cannot wedge the two legs against each
other. While the screen leg is silent, the call is read back every 3 s:
a target that refuses (no `node:shell`, an agent too old, sixteen shells
already open) never writes on it.

**What the node does under a shell.**

- *The link reconnects* (backend restarted behind a proxy, a network
  blink): the shell is not touched. Its legs are HTTP requests of their
  own, not the WebSocket; output keeps flowing, and its end is answered
  on the new connection.
- *`3pm restart agent` or `3pm update` typed in the shell* goes through:
  the daemon restarts the agent, and the agent, before it closes its
  link, hangs up every shell it carries (`SIGHUP`, then `SIGKILL` 0.5 s
  later for one that ignores it — not the 5 s of a leg that ended: the
  shell is a session of its own, the daemon's kill of the agent would
  not reach it) and delivers their end. The session
  ends as a hang-up — the command line exits **129** (`128 + SIGHUP`),
  the call settles `exited` with signal 1 — and the next `fleet ssh`
  opens on the restarted agent. The shell's own `3pm` command does not
  print its result: it dies of the hang-up with its shell, after the
  daemon took the order. The wait is bounded (1.2 s, under the daemon's
  1.6 s kill timeout); an agent killed outright (`SIGKILL`) closes the
  pty and the kernel hangs the shell up, and the command line exits 255.
- *A macOS target* works like a Linux one (the account's login shell,
  zsh by default).
- *Between LAN neighbours* the session's bytes may go agent to agent
  instead of through the backend. The command line asks for it when
  its host's agent runs (it finds that agent's `data.sock`); the backend
  agrees only when both nodes are set to `direct: all`. A session
  sealed end to end (the default, below) is recorded by the target's
  agent, so it goes direct **recorded or not**; a session in clear
  (`--no-e2e`, an old agent) is recorded off the relay, so on a bucket
  that records shells it always takes the relay. The command line then hands its
  two legs to its agent (`POST /legs` on `data.sock`, the agent's own
  socket, never a hosted user's) and `PUT`s/`GET`s them there; the
  agent dials the target for the screen, the target dials it for the
  keyboard, each with a 1 s hold before the relay takes the leg —
  whole, never split. The call, its audit and `exited` stay on the link
  as always; the call says `path: direct` once a leg went that way.
  When the backend named a neighbour but this host's agent would not
  take the legs (an agent too old for `POST /legs`, one with no LAN
  listener), the command line says so once on stderr and goes through
  the relay; `3pm fleet ssh -v` says which way every session's legs go,
  and why the relay (no socket at its path, no neighbour named, the
  agent's refusal).
  Same for `-- <cmd>`, `fleet attach` and `fleet cp`; `-L` forwards
  keep the relay for now.

**Where it comes from.** Command line: `modules/agent/src/fleet/ssh.rs`
(placement, legs, the raw terminal with its panic hook and the
signals that end a session from outside, refusals by code), `fleet/ssh/session.rs` (the loop, with no HTTP and no tty, which
the tests drive on a paused clock), `fleet/ssh/escape.rs`. Target:
`modules/agent/src/shell/` (`host.rs` permits and replies, `open.rs`
the login, `pty.rs`, `pump.rs`, `audit.rs`, `legs.rs` the LAN way of
the legs). LAN direct, caller side: `fleet/ssh/lan.rs`, `data/legs.rs`. Wire: `node:shell`,
`AgentRequest::Shell`, `AgentResponse::Exited`, `AgentEvent::Shell` in
`threepm-link` (`docs/backend-agent/link-protocol.md`).

### `3pm fleet ssh` end to end: the backend relays what it cannot read

**What you see.**

```
$ 3pm fleet ssh web-1
The authenticity of node 'web-1' (3pmn_…) can't be established.
Node key fingerprint is SHA256:Qm9v….
Continue connecting (yes/no)? yes
root@web-1:~#
$ 3pm fleet ssh web-1 --accept-new -- uptime   # scripts: pin without asking
$ 3pm fleet known-nodes ls                      # id, fingerprint, name, date
$ 3pm fleet known-nodes rm web-1                # forget it (ssh-keygen -R)
$ 3pm link status                               # on the node: "shell key SHA256:…"
```

Every mode — the shell, `-- <cmd>`, `fleet cp`, `fleet attach`, `-L` —
is sealed between this 3pm and the target's agent by default: the
backend relays keystrokes, screen, `cp` files and forwarded bytes it
cannot read, and cannot change a byte in flight without the session
ending (255, "the encrypted channel broke"). The node's key is pinned
at the first session (trust on first use) in `~/.3pm/known_nodes`,
one line `<node_id> <fingerprint> <name> <date>`, keyed by the node id:
the first contact asks on `/dev/tty`, or pins silently under
`--accept-new`, and with neither (no terminal) is refused. A key that
**changed** is always refused, the offending line named, exit 255 —
ssh's `NODE IDENTIFICATION HAS CHANGED`; `known-nodes rm` and the next
session asks again. The file is never filled from the backend: it is
what the pin guards against.

The id alone would let the backend dodge the pin by answering the
hostname you typed with **another node id**: a key it shows there is
a first contact. So a hostname typed in full (case aside) that is
already pinned under another id **of the same cluster** is refused like
a changed key, before any question and under `--accept-new` too: the
banner names both nodes and says `3pm fleet known-nodes rm <pinned id>`
(by id: `rm` by name is case-sensitive) for a node that was really
re-linked. One file serves every bucket, so each line carries a fifth
column, the cluster id; a line of an older 3pm has none and is compared
in every cluster, and an older 3pm reads the new lines. The limit: only
a hostname typed **in full** is held to its pin — a prefix (`3pm fleet
ssh we`) names whichever node the bucket answers, and an id is checked
by its key alone. `fleet cp` pins the node as its paths name it, like
`ssh` and `attach`.

**Old agents, old command lines.** An agent of this version refuses
every session in clear, with no setting to relax it: a 3pm that
predates the channel is told "this node requires end-to-end
encryption: 3pm >= x.y" (the agent's code `e2e_required`). The other
way round, a node whose agent cannot encrypt is refused before
anything opens (`409 ERR_NODE_NO_E2E`, naming its agent's version):
update it, or pass `--no-e2e`, which exists for that case only — the
session goes in clear, as every session did before, after a warning on
stderr; an up-to-date node refuses it. Both refusals exit 255; the
command line tells them by **code** (the backend's `ERR_NODE_NO_E2E`,
the agent's `e2e_required` / `e2e_broken` in the call's `result.code`),
never by their words.

**Recording, by the node.** Off unless the caller passes `--record`;
then, on a bucket that records shells, the target's agent writes the asciicast **before** sealing — the same
file, by the same writer, as the backend's for a session in clear:
the screen only, never the keyboard, never a forward, 64 MiB at most,
the session never waits on it — and uploads it as the call's artifact
before it answers `exited {recording}`. An agent killed mid-session
leaves no recording: the backend's `ended` line says `recording:
lost`. A sealed session goes LAN direct recorded or not.

| A compromised backend can… | in clear (`--no-e2e`) | end to end |
|---|---|---|
| read a session live (keys, screen, `cp` files, `-L` bytes) | yes | **no** |
| inject keystrokes, change a file in flight | yes | **no** (AEAD: the session ends) |
| pose as the node | yes | **no**, except at the very first contact (TOFU) |
| read the stored recording and audit lines | yes | yes (by choice: readable) |
| see metadata, sizes, timing; refuse to relay | yes | yes |
| **open** a session on a node with its fleet token | yes | **no** (the node's `authorized_keys`) |
| change what an admitted session runs (argv, `--user`, `host:port`, `record`) | yes | **no** (bound to the handshake) |

The last two lines are the next section's: a node opens a session only
for a caller key in its own `authorized_keys`, and only for exactly the
request that caller placed.

What "readable" covers: the recording is what the node wrote on the
screen leg. A copy (`3pm fleet cp`, up or down) is never recorded —
a download's screen is the file — by the backend nor the agent: both
recognise its fixed scripts byte for byte
(`threepm_link::cp::is_cp_transfer`: `sh -c <script> 3pm-cp <paths…>`),
and an argv one byte off is an ordinary command, recorded. Nothing is
hidden by it: the scripts only move bytes between the legs and the
named paths, and the audit keeps the command line, paths included.
`-L` is never recorded either.

**How it works.** `Noise_XX_25519_ChaChaPoly_BLAKE2s` (crate `snow`,
pure-Rust primitives), prologue = a label (`3pm fleet ssh e2e v2`), the
call id, the target node id and the canonical encoding of the request
(next section), so a handshake cannot be replayed on another call, two
sessions spliced, nor the request changed on the way. Three messages before any session byte: `e` on the
keyboard leg, `e, ee, s, es` on the screen leg, then `s, se` once the
node's key is checked against `known_nodes`; the node starts the shell,
the command or the forward only after it. Then each leg carries records
`[u16 BE length][Noise message]` (at most 65 519 clear bytes each) in
place of its raw bytes; the legs are lossless and ordered, which gives
the nonces. Keys: `shell.key` (X25519, 0600) in the agent's state
directory, `cli.key` beside `known_nodes` in the caller's `$THREEPM_HOME`
(moved there from the agent's state directory on first use, so `3pm link
delete` no longer changes it), each created when first needed.
Wire (additive): `e2e` and `record` on `shell`/`exec`/`attach`/
`forward`/`run_as`, `shell_e2e` in the agent's `hello`, `cli_key` on
the `open` audit event, `recording` on `exited`.

**Compression, inside the channel.** `3pm fleet cp` compresses by
default (`--no-compress` to send the bytes as they are), and `3pm fleet
ssh -C -- <cmd>` asks for it when the command runs without a pty (`-T`,
or stdin not a terminal); `-v` says whether the session is compressed,
and why not. Both directions, zstd level 1, one frame per sealed record
(a flag byte, then the bytes raw or compressed, at most 65 518 clear
bytes): each frame stands alone, so nothing waits for more bytes, and
one that would not come out smaller goes raw — a random or already
compressed file costs one byte per 64 KiB. A frame never opens to more
than its 65 518 bytes (no decompression bomb). Measured on one core
(EPYC 7281): source text 3.0× at 210 MB/s, a system journal 10×.

Never a shell, a pty or `fleet attach`, `-C` or not: a keystroke would
wait on the codec, and a compressed stream that mixes what you type or
read (a password, a token on screen) with bytes someone else chooses
leaks it through its length (CRIME). Never `-L`/`-R`/`-W` either (`-C`
is refused beside `-W` and `-N`): their bytes are mostly TLS or already
compressed. `--no-e2e` has no channel, so no compression. The recording
of a recorded `-- <cmd>` is written from the clear bytes, before sealing
and compressing, so it does not change.

Negotiated in the handshake, never through the backend: the caller's
first message carries what it asks (`zstd`) as its payload, the node's
answer what it grants. Both payloads are hashed into the handshake, so a
backend that strips or adds the ask breaks it (`e2e_broken`), as any
other change does. An older agent ignores the ask and answers nothing,
an older 3pm never asks: either way, no compression and nothing else
changes.

**Where it comes from.** `modules/agent/src/e2e/` (the handshake,
records, keys, `compress.rs`), `shell/legs.rs` and `shell/host.rs` (the responder),
`shell/record.rs` (the node's recording, writer
`threepm_protocol::cast`), `fleet/ssh/e2e.rs` (the initiator),
`fleet/known_nodes.rs`, `fleet/ssh/refusal.rs` (refusals by code).

### `3pm fleet ssh` access by key: the node's `authorized_keys`

**What you see.**

```
$ 3pm fleet key                              # this host's caller key (no link needed)
SHA256:Qm9v…  (/Users/me/.3pm/cli.key)

On a node, to admit it:
  sudo 3pm link authorize SHA256:Qm9v…
$ 3pm fleet key new --out /Volumes/USB/rescue.key   # an offline rescue key
$ 3pm fleet key push web-1 SHA256:Zx… --comment rescue   # or --all
$ 3pm fleet key revoke --all SHA256:Zx…
# on the node itself:
$ sudo 3pm link authorize SHA256:Qm9v… --comment mac-unitech
$ sudo 3pm link deauthorize mac-unitech      # by fingerprint or comment
$ 3pm link status                            # the authorized keys, or why none opens
$ 3pm link 3pmc_… 3pmk_… --authorize SHA256:…   # at link time (or THREEPM_LINK_AUTHORIZE)
$ 3pm fleet ssh web-1 --record               # record it (off by default)
```

A node opens a `3pm fleet ssh` session — shell, `-- cmd`, `fleet cp`,
`fleet attach`, `-L`, `-W`, `-R`, `--user` — only when **(a)** the
caller's key is in the node's own `authorized_keys` and **(b)** what it
runs is exactly what that caller asked for. The fleet backend, a stolen
`3pmu_` token or another node can no longer open a session, nor change
the command of an admitted one. A refusal exits 255 for every mode
(`-L`/`-W`/`-R` stop at the first one), with the refused fingerprint and
the `sudo 3pm link authorize …` to type on the node; the backend's
`ended` audit line carries the same message.

**The file.** `<agent state dir>/authorized_keys` — `$THREEPM_HOME/agent/`,
or `THREEPM_AGENT_STATE_DIR`; `/root/.3pm/agent/` on a root node. One
`SHA256:<fingerprint> [comment]` per line, `#` comments; a line that does
not parse is skipped with a warning, never read as "everyone". Checked
like sshd's `StrictModes`: owned by the agent's user and not writable by
group or others (0644 is fine), else refused. **Absent or empty: every
session is refused** — a node is closed until someone authorizes a key.
Re-read for each session (at most 64 KiB): no restart. `link authorize`
writes it atomically under a lock, idempotently; `3pm link delete`
keeps it, and `shell.key`.

**The caller's key.** `cli.key` (X25519, 0600) beside `known_nodes` in
the caller's `$THREEPM_HOME`. `THREEPM_CLI_KEY=<file>` uses another
one as it is (a rescue key from `fleet key new`). This key is the fleet
access credential; it is not the release signing key of `just publish`
(`~/.3pm-release/`), which is another pair entirely.

**The request is bound.** The handshake's prologue holds the canonical
encoding of every field that decides what runs: the operation, the argv
and `pty`, the `--user` account, the attach target/mode/`read_only`, the
`host:port` of `-L`/`-W`, the listen and connection of an `-R` takeover,
the `bind:port` of `-R`, the shell's `term` and first size, and
`record`. The command line encodes what it asked for, the node what it
received; one byte apart and the handshake fails before anything
starts. An `-R` connection is taken over only by the key that opened
the listen.

**Recording is the caller's to ask for.** A session is recorded (by
the node, see above) only when the caller, the bucket and the node all
agree: the command line asks with `--record` — without it the body says
`record: false`, the default (`--no-record` is still accepted and says
the same; an older 3pm that says nothing leaves it to the bucket) — the
backend may only lower it (bucket `record_shells` off, `cp`, forwards) and returns
the effective value before the handshake, which binds it; the command
line says "this session is recorded (screen and command lines)" before
opening. A node with `[shell] record = "never"` in `link.toml` records
nothing whatever was asked, and says so in its answer. The command
lines go with the recording: a session that is not recorded (any of the
three said no) sends no `line` and no `screen` audit event — the node
does not even read them off the screen — only `open`, `exec` or
`attach`, and `close` with `lines: 0`.

**What it does not cover.** Remote actions of apps and modules
(`nodeAction`, `remote: true` by default), `3pm:profile`,
restart/stop, the first contact's TOFU, other unsigned files of the
download site (SDKs), an app running as root writing `authorized_keys`.

**Locked out?** With local access (real ssh, console, `sshpass` on a
tablet): `sudo 3pm link authorize SHA256:…`. From a machine that holds
the rescue key: `THREEPM_CLI_KEY=/Volumes/USB/rescue.key 3pm fleet key
push <node> SHA256:…`.

**Where it comes from.** `modules/agent/src/e2e/authorized.rs` (the
file, `StrictModes`, the lock), `e2e/bind.rs` (the canonical request),
`shell/host.rs` (the check, after the handshake and before the
session's permit: 16 handshake slots, 8 per caller `from` so one
stolen token cannot hold them all, 5 s for the first message),
`authorize_cmd.rs` (`link authorize|deauthorize`), `fleet/key.rs`;
wire `AgentErrCode::KeyNotAuthorized` (`key_not_authorized`). Proven
end to end against the real agent by the backend's
`tests/it/real_agent_access.rs` and `real_agent_exec.rs`.

### `3pm fleet ssh <node> -- <cmd>`: one command on a node

**What you see.**

```
$ echo x | 3pm fleet ssh web-1 -- wc -c
2
$ 3pm fleet ssh web-1 -- 'journalctl -u nginx | tail -3'
$ tar -c ./site | 3pm fleet ssh web-1 -- tar -x -C /srv
$ 3pm fleet ssh web-1 -- cat /var/log/big.bin > big.bin   # bytes, unchanged
$ 3pm fleet ssh -t web-1 -- htop                         # a pty, from anywhere
$ 3pm fleet ssh web-1 -- false; echo $?
1
```

The command runs through the target's login shell, as ssh's does
(`<shell> -c`, not a login shell: no profile runs). Several words are
each quoted for that shell, so `-- cat 'my file'` reads one file; one
word is the whole line, so `-- 'ls | wc -l'` is a pipeline there.
**A pty or pipes**: the command gets a pty when stdin and stdout are
both terminals, or with `-t`; `-T` never gives one. On a pty it is a
shell's session (raw terminal, `~.`, the size followed). On pipes it is
bytes: stdin is streamed and then closed (`echo x | …` ends), the
command's stdout goes to stdout and its stderr to stderr, unchanged —
binary included —, with no raw mode and no `~.`; a command that stops
reading its stdin (`head -c1`) has the rest dropped, not the session
cut; after its exit, a tail still on its way to a paused reader
(`… | less`) is waited for, a background job holding its stdout
(`cmd &`) for 5 s of silence. The exit code is the command's
(`128 + signal`), 141 and nothing said when stdout stopped being read
(`… | head -1`, as `SIGPIPE`), 255 with one line on stderr when the
session ended otherwise (a leg cut, the node refusing, `Ctrl-C` on the
command line). The same rules as a shell:
the `invoke` grant, `3pm:shell` in the target's `callers`, `node:shell`
in its `caps`; an agent that predates this answers "too old".
**Audit**: the backend's `open_requested` and the agent's `exec` line
both carry the line the shell is given (`cat '>' /etc/x` for
`-- cat '>' /etc/x`).
Never the command's stdin; on pipes, nothing off its output either
(with `-t`, the screen's lines as for a shell).

**How it works.** `nodeShell` with `command` (and `pty`) places an
`exec` call — its own op, not a field of `shell`: an agent too old to
know it refuses it, where one ignoring a field would start a shell and
take a script's stdin for keystrokes. Same two `pipe` legs; on pipes
two more `attach` kinds ride them, `eof` (`0x05`, the end of stdin)
and `stderr` (`0x1d`). Back-pressure holds both ways with no buffer
that grows: the target reads a pipe only once its last message is
queued for the screen leg, and the keyboard leg only once its last
input is in the command's stdin. After the command exits, what its
pipes still hold is delivered (a background job holding them is left
after 5 s with nothing moving), then its end. 20 MB through `cat` there
and back: about 0.1 s on loopback (Linux, debug builds).

**Where it comes from.** Command line: `modules/agent/src/fleet/ssh/command.rs`
(the pty or pipes decision, the body), `fleet/ssh/session.rs`
(`run_command`, `Mode`). Target: `modules/agent/src/shell/pipes.rs`
(the command between its legs), `shell/open.rs` (`command_line`,
`spawn_pipes`, `spawn_command`). Wire: `AgentRequest::Exec`,
`ShellEvent::Exec` in `threepm-link`; `ClientMsg::Eof`,
`DaemonMsg::Stderr` in `threepm_protocol::attach`.

### `3pm fleet ssh -l <user>`: as another account of the node

```console
$ 3pm fleet ssh -l deploy web-1             # or deploy@web-1
$ 3pm fleet ssh -l deploy web-1 -- id -un
deploy
$ 3pm fleet cp -l deploy ./site.tar web-1:  # into deploy's home
```

- **Where the agent is root** (a tablet, a server installed by the
  package), the session becomes that unix account: `getpwnam` +
  `getgrouplist`, its home, its shell, `HOME`/`USER`/`LOGNAME`/`SHELL`
  from its `passwd` entry; in the child, after the new session and its
  controlling terminal, `setgroups` → `setgid` → `setuid`, then a check
  that uid 0 is out of reach, then `PR_SET_NO_NEW_PRIVS` on Linux — the
  daemon's drop for hosted users, copied
  (`modules/agent/src/shell/user.rs`). The pty is chowned to the
  account (`0620`), so `tty` and `sudo` work in it. No new right: a root
  shell could already `su`.
- **Where the agent is not root**, only its own name is taken; any
  other is refused (`forbidden`, "the agent runs as `pi`, not root"),
  never run as the agent. An unknown account is refused (`not_found`),
  before any leg opens.
- **The name** is a plain login, `[a-z_][a-z0-9_-]{0,31}`, checked by
  the command, the backend (422) and the agent alike. `-l` and `user@`
  together are refused. Never with `fleet attach` (the app's terminal is
  the app's).
- **Old agents are safe**: the request is a new op, `run_as {user,
  req}`, wrapping the `shell` or `exec`. An agent that predates it
  answers "unknown command" (the command says the agent is too old)
  instead of ignoring a field and running the session as itself.
- **Audit**: the backend's `open_requested` / `refused` carry `user`
  (the account asked for), the agent's `open` line the account it ran
  as.

### `3pm fleet ssh -L`: a local port through a node

**What you see.**

```
$ 3pm fleet ssh -L 5432:db.lan:5432 web-1          # a shell, and the forward beside it
$ 3pm fleet ssh -N -L 8080:127.0.0.1:80 web-1      # no shell: forwards only, until Ctrl-C
3pm fleet ssh: -L 127.0.0.1:8080:127.0.0.1:80: listening on 127.0.0.1:8080
$ 3pm fleet ssh -N -L '*:9000:[::1]:9000' -L 6379:cache:6379 web-1
```

`-L [bind:]port:host:hostport`, as ssh's: a listener on this host
(127.0.0.1 unless a bind address is given, every interface for `*`; an
IPv6 address in brackets; port 0 lets the system pick, and `-N` prints
what it bound), and every connection it accepts is opened from the node
to `host:hostport` — any destination the node itself can reach (a name
is resolved there). Repeatable. With a shell the forwards live as long
as the session; with `-N` (no shell, no terminal needed) until
`SIGINT`/`SIGTERM`, exit 0. A port already taken fails before anything
is placed; a connection that fails (the node refusing, the destination
refusing — `connect db.lan:5432: Connection refused` —, a leg cut) is
one line on stderr and that connection closed, the listener and the
other connections carrying on. A half-close crosses both ways (a
client's `shutdown(WR)` is the destination's EOF and its answer still
comes back), bytes are lossless and back-pressured. `-L` with
`-- <cmd>` is refused for now. Who may: the `invoke` grant, the action
`3pm:forward` in the target's `callers` (not `3pm:shell`: a forward
is its own permission), `node:forward` in its `caps` — on by default;
take it out to refuse. A listener off the loopback (`*`, an address
of the LAN) prints a warning: whoever reaches it goes through the node
with your grant; on a shared machine any local user can reach
127.0.0.1 too, as with ssh's. **Limits**: forwards count against the
target's 16 sessions in flight, with its shells — the command line
carries at most 16 connections at once, the next ones waiting accepted
(a browser's six stay well under it, but each open forward is a shell
fewer); each connection is one call and two blobs, so the backend's
blob quotas (100 open, 200 created a minute per owner) bound a client
that opens a fresh connection per request to ~100 a minute (`429`,
said on stderr) — keep-alive clients are unaffected. A leg cut ends the
forward at once (no wait for the destination to close). **Audit**: the backend's
`open_requested` / `refused` with `forward = host:port`, then `ended`;
the agent's `close` line carries the bytes each way.

**How it works.** Each accepted connection is one `nodeForward` call
(`POST …/nodes/{node}/forward {host, port}`) placing `forward {host,
port, input, output, from}` on the target: two `pipe` legs as a
shell's, but carrying **raw bytes** in `live` messages (no `attach`
kind byte), a message flagged `END` closing its way. The agent
connects (5 s), pumps both ways, and replies `exited {code: 0}` once
both ways ended by an `END` (1 when a leg or the connection broke); a
refused connect is the call's `err`, the legs never opened (the
caller's `GET` answers 408 and the command line reads the call back for
the reason). Code: `modules/agent/src/shell/forward.rs` (target),
`modules/agent/src/fleet/ssh/forward.rs` and `fleet/forward_spec.rs`
(command line).

### `3pm fleet ssh -W host:port`: the real `ssh` through a node

**What you see.**

```
$ 3pm fleet ssh web-1 -W localhost:22 --accept-new < request > answer
$ ssh -o ProxyCommand='3pm fleet ssh web-1 -W localhost:%p' envision@web-1
```

With this block in `~/.ssh/config`, every `<node>.3pm` host goes
through the node — `ssh`, `scp`, `rsync -e ssh`, `git`, ansible, VS
Code Remote-SSH:

```
Host *.3pm
  ProxyCommand 3pm fleet ssh $(echo %h | sed 's/\.3pm$//') -W localhost:%p
```

```
$ ssh envision@envision-surface-7caf.3pm
$ rsync -a ./site/ deploy@web-1.3pm:/srv/site/
```

ssh runs the line with `$SHELL -c`, never an interactive shell: `3pm`
must be on the `PATH` (an alias does not count), or written as a full
path.

`-W host:port`, as ssh's: no shell, no listener — stdin goes to
`host:port` as the node reaches it, what comes back goes to stdout,
one connection, and the process exits once both ways are closed. It is
`-L`'s connection with stdin and stdout for the local peer: the same
`nodeForward` call, sealed end to end the same way, the same grant
(`invoke`, `3pm:forward` in the target's `callers`, `node:forward` in
its `caps`), the same audit; nothing new on the node or the backend.
So the real `ssh` (its own keys, `authorized_keys`, agent forwarding)
reaches a node's sshd with no route to it, no port opened, through the
encrypted relay. **stdout is the destination's bytes only**: every
message (the first-use question, which goes to `/dev/tty`, a refusal, a
caution about the bearer) is on stderr. A node met for the first time
asks on `/dev/tty`; give `--accept-new` where there is none.
**Exit**: 0 when both ways closed cleanly, 255 (as ssh) for anything
else — the node unknown, a refusal, the destination refusing
(`connect localhost:22: Connection refused`), a leg cut — with one line
on stderr. `-W` with a command, `-L`, `-N`, `-t` or `-T` is refused by
the parser. **An idle tunnel stays up** (`-W`, `-L` and `-R` alike): a
local peer silent for 20 s sends an empty message on the input leg, a
keepalive the node ignores, so the relay no longer gives the legs up at
its one-minute idle bound. The output leg is not pinged: while the
destination says nothing, a reverse proxy in front of the backend
still cuts it at its read timeout (`proxy_read_timeout` /
`client_body_timeout`, 1 h on fleet.edgecr4ft.net, 60 s for a default
nginx). Code: `proxy` and `carry` in
`modules/agent/src/fleet/ssh/forward.rs`; e2e
`3pm-backend/tests/it/real_agent_proxy.rs` (20 MB through stdin and
stdout, the same hash; the host's own sshd through `ProxyCommand`).

### `3pm fleet ssh -R [bind:]port:host:hostport`: a node's port, served from here

**What you see.**

```
$ 3pm fleet ssh web-1 -N -R 0:localhost:8000
3pm fleet ssh: Allocated port 41237 on web-1
$ 3pm fleet ssh web-1 -R 8080:localhost:8000      # with a shell on web-1
web-1$ curl localhost:8080                         # reaches this host's :8000
```

`-R`, as ssh's: the node listens on `bind:port` (`bind` left out: its
loopback; port 0: one it picks, printed as above), and every connection
it accepts comes back here, connected to `host:hostport` as this host
reaches it. Repeatable, beside `-L`, with a shell or alone with `-N`;
refused with `-W` or a command. **This side places every call**: one
sealed `nodeListen` per `-R` (its legs carry `bound`, one `accepted`
per connection, and this side's keepalive every 20 s), then one sealed
`nodeForward` naming each accepted connection — so the end-to-end
channel, the node's key on first use, the grant (`invoke`,
`3pm:forward` in the target's `callers`, `node:forward` in its `caps`)
and the audit are `-L`'s; the node reaches nothing of this host.

**Where the node listens is its own decision**, as ssh's `GatewayPorts
no`: the loopback, unless its `link.toml` has

```toml
[shell]
gateway_ports = true
```

— written by hand on the node, never by the backend, kept across a
re-link. Without it, `-R 0.0.0.0:…` (or `*:…`, or a LAN address) is
refused with that sentence.

A connection the command line does not take over within 10 s is closed
on the node; a local connect that fails is one line on stderr and the
node's connection is taken over and closed at once (its peer reads the
end now rather than after the 10 s). The listen ends — the port closed
and every connection still waiting with it, those already carried left
to their own end — on Ctrl-C or the shell's end (the command line sends
its `END`), a leg cut, a minute without a keepalive, or the agent's
stop. Code: `modules/agent/src/shell/listen.rs` (node),
`modules/agent/src/fleet/ssh/reverse.rs` (command line); e2e
`3pm-backend/tests/it/real_agent_reverse.rs` (3 connections opened on
the node, 20 MB each echoed by a server on the command line's side, the
same hash; the port closed within milliseconds of `SIGTERM`).

### `3pm fleet attach <node> <app>`: a `tty` app's terminal on a node

**What you see.**

```
$ 3pm fleet ssh web-1 -- 3pm start bash --tty --name work   # once
$ 3pm fleet attach web-1 work            # ~. after Enter: detached, the app runs on
3pm fleet attach: detached from work on web-1
$ 3pm fleet attach web-1 work --peek     # the screen as text, then out
$ 3pm fleet attach web-1 work --raw --read-only --instance 1
```

`3pm attach`, from another node: the terminal of an app started with
`tty = true` there — persistence for a remote session (a `bash` left
running in a `tty` app survives every detach). `--raw`, `--peek`,
`--read-only` and `--instance` mean what they mean for `3pm attach`;
`~.` after Enter detaches (in place of `Ctrl-Z` locally), and the app
runs on. The screen, its title, its bell and its alternate screen are
drawn as `3pm attach` draws them; the app exiting is a status line, not
the end — it may come back. Exit 0 when the session was detached (`~.`,
a peek served, the app deleted), 255 with one line on stderr otherwise
(the node refusing, a leg cut). Refusals are the daemon's: an unknown
app is `not_found`, one without a terminal `not_tty`. Same rules as
`fleet ssh`: the `invoke` grant, `3pm:shell` in the target's `callers`,
`node:shell` in its `caps` (what the app runs, a shell could run); an
agent that predates this answers "too old". **Audit**: the backend's
`open_requested` carries the app, the agent writes `open`, `attach`
(the app, its instance, the mode), each raw line typed (`line {alt:
true}`: the agent relays the screen without reading it, so a password
typed into the app is logged as typed) and `close`.

**How it works.** `nodeShell` with `attach` places an `attach` call
(its own op). The target's agent opens a **fresh** connection to its
daemon's `rpc.sock` (never one of its pooled ones: the connection
leaves NDJSON), sends the daemon's own `attach`, and once it is
answered `attached` relays the messages as they are between the legs
and that socket — the same `attach` codec the shell's legs carry. A leg
that ends, or an agent that stops, sends the daemon `detach`; the
daemon's `detached` is forwarded, then the leg's `END`; the call
settles `exited` with no status.

### `3pm fleet cp`: files to and from a node

**What you see.**

```
$ 3pm fleet cp ./build/app.tar.gz web-1:/srv/releases/
$ 3pm fleet cp web-1:/var/log/syslog .
$ 3pm fleet cp -r -p ./site web-1:/srv            # → /srv/site, modes and times kept
$ 3pm fleet cp -r web-1:~/data ./data-web-1       # a new directory here
$ 3pm fleet cp a.conf b.conf web-1:/etc/app/      # several sources: a directory
$ 3pm fleet cp web-1:/nope .
cat: /nope: No such file or directory
[3PM][ERROR] the copy of /nope failed on web-1 (exit 1)
```

As scp has it: exactly one side is `<node>:<path>` (an id, a hostname
or a unique prefix, as for `fleet ssh`; `<node>:` alone is its home,
`~/x` is under it); a `:` before any `/` makes a node, so a local name
with one is written `./a:b`. A destination that is a directory takes
the source under its own name, else the copy is that name. `-r` copies
directories (a file under `-r` is a file); without it a directory is
refused, `Is a directory (use -r)`. `-p` keeps each file's mode
(setuid bits included) and modification time, and a directory's;
without it the receiving side's umask applies to the source's mode and
the time is the copy's. **Never half a file under its name**: a file
goes up into `<dst>.3pm-tmp-<rand>` (owner-only, `umask 077`, until
its `chmod`: a secret is never readable by others mid-copy), then `chmod`, `touch -t` with
`-p`, `mv` over the destination; it comes down into a hidden temporary
file beside its target, renamed only once the node's command exited 0
(a replaced file keeps its mode). A local read error hangs the node's
command up instead of closing its stdin, so the temporary file is
removed there, never renamed. A file is never written over a
directory, either way. Inside a tree, symlinks stay links; a single
file is followed (`-p` too: `tar -h`), as `cat` follows it; on the way
down, a member that climbs out (`..`, or through a symlink) is never
written. A destination that is a link to a file is written through
(the link stays, as with scp). A tree lands as `tar` lands one: merged
into an existing directory of that name, not atomic, and the node's
`tar` gives the directories it unpacks (an existing one included) the
source's mode under the umask, even without `-p`. One session per source, one after
the other; a failing source does not stop the next (a session lost, a
refusal or a destination that is not a directory does). Exits 0 when
every source was copied, else 1: one line per failure, then `n of m
sources not copied`. The node's own stderr is shown as it comes; with
stderr a terminal, one progress line (bytes, rate) per source. Nothing
on stdout. **Audit**: each copy is a `fleet ssh -- <cmd>` session, and
its command line is what the audit shows — `sh -c '…' 3pm-cp
/etc/app/ a.conf 0644 '' 1 0 1f2e3d4c`: the node saw a file written
into `/etc/app/`. Same grants as `fleet ssh`.

**How it works.** No new op and no new route: each source is an
`exec` on pipes. What runs is one of four fixed POSIX scripts, `sh -c
'<script>' 3pm-cp <args>`, the paths only ever its arguments (a name
with a space, a quote or a `$` cannot become code, and a relative one
starting with `-` is made `./-…`, never an option; `sh` is named so a
fish or csh login shell does not read the script): a file up is `cat >`
the temporary file, a file down is `cat`, a tree up is `tar -x -o
-p|-m -f - -C <dst>`, a tree (or a file with `-p`) down is `tar -c` of
the directory's content (`./…`) or of the one file. Here the archive
is the `tar` crate, never the local `tar` binary: the same bytes from
macOS and Linux (bsdtar would add `._` members), no child process, and
an unpacking that checks every path; there, `tar` with only the flags
GNU tar, bsdtar and BusyBox agree on (`-o`: owners are not taken from
the archive). The bytes are the command's stdin up and its stdout
down, both lossless legs. 100 MB up and back on loopback (Linux, debug
builds): about 0.2 s up, 0.6 s down, the same sha256 at each end.

**Where it comes from.** `modules/agent/src/fleet/cp.rs` (the command
line, the scripts, one session per source), `fleet/cp/local.rs` (this
side: the stream in, the archive built and unpacked, temporary files),
`fleet/cp/progress.rs` (the progress line); the session is
`fleet/ssh/session.rs::run_command`. End to end:
`3pm-backend/tests/it/real_agent_cp.rs`.

### Between LAN neighbours: the file skips the backend

**What you see.** Nothing changes in an app: `io.call(node, action,
{file})` behaves exactly as before. When the two nodes sit on one LAN,
the file goes from the caller's agent to the target's directly — the
backend still places, audits and settles the call, and shows which way
the bytes went on it (`path: direct` or `relay`). A neighbour that
cannot be reached (a firewall, another network behind the same public
address) costs the reader at most 300 ms per address and the writer at
most one second, then the relay carries the file as it always did; that
pair is left alone for a minute. `direct = false` in the agent's
`link.toml` turns the listener off.

**Where it comes from.** `modules/agent/src/data/direct/`: `tls.rs`
(TLS 1.3, raw Ed25519 keys — `node.key`'s seed behind the fixed PKCS#8
header — the dialler pins the key its ticket named, the listener checks
the key per blob), `peers.rs` (who may read which blob here, whom to
dial for one, what the last attempt with each neighbour came to),
`meet.rs` (the rendezvous: reader first or writer first, one lock, one
copy), `listener.rs` (the LAN port: `GET /blobs/<blob>` from the named
key, `404` for anything else, `409` when late), `dial.rs` (the reader's
`GET`). The hooks: `data/handler.rs` (`put` → `Direct::put` before the
relay, `get` → a neighbour before the relay), the two points that mint
local tokens (`fleet_gate/place.rs` for the caller's input,
`commands.rs::open_blobs` for the target's), and `run_fleet.rs` (the
listener bound with the data socket, its port in `hello`, the reports
forwarded to the link). Only a call's input: its output is read after
the reply, which is after its writer finished, so nobody could meet it.
A session (`io.open`) between two nodes both set to `all` sends both of
its legs the same way — the leg's frames still go through the abandon
sink (`data/sink.rs`) first, and the writer reports every 5 s how far
it got (`blob_stat`): the call's `live` counters, and "direct · no
preview" on the dashboard, since the relay never sees a frame.

## A virtual Ethernet between nodes (`lan`)

**Installed by default, harmless alone.** `lan` is marked `auto`: every
3pm installs it by itself (docs/modules.md). Where it cannot work — not
Linux, not a root daemon, no `/dev/net/tun`, a `lan.toml` it cannot use,
a host not linked yet — it says why in one log line and stays up, idle,
`lan.status` answering `{"state":"idle","reason":…}`: never a crash
loop. Its peers never call it: an idle `lan` declares no `link` action, and
the discovery keeps only nodes whose `lan` declares it (read per node
with `fleet:read`, `3pm fleet inspect <node> lan --json`, kept per `lan`
process for 10 min: a steady discovery is 2 `3pm fleet` calls); a `lan`
of 0.50/0.51 that still refuses with `lan-idle` waits the longest
backoff. Every failed open backs off per peer, 5 s doubling to 10 min
with ±20 % jitter, reset only by a session that came up, and past 6
failed opens a minute all dialling pauses (`lan.status` `warnings`):
each call, even refused, holds backend blob quota for a minute. Where it can, it still makes **no device until a peer shows up**:
the TAP (and the namespace) is created when a discovery first finds
another `lan` node, and removed when none has been found for 10
minutes, so a server alone in its bucket gets no interface. Before
creating it, the module checks the network against the host's
addresses and main routes: a range already in use (a VPN's `10.0.0.0/8`)
is not shadowed — a default route split in `/1` halves (OpenVPN's
`def1`) or written `/0` is no range in use — one warning, no device, `lan.status` `reason`, looked
at again at each discovery. To turn it off on a host, or on a whole
fleet through provisioning, write in `<home>/config.toml`:

```toml
[modules]
off = ["lan"]
```

The next `3pm update` (or daemon the CLI spawns) uninstalls it — apps
deleted, copy removed — and no later one puts it back; take the name out
and it comes back. `3pm uninstall lan` is the per-host refusal
(`modules.off`), as for any auto module.

**What you see.** A fourth bundled module, `lan`: every node of a bucket
that runs it as root gets a TAP device `3pmlan0` with an address on
`10.77.0.0/16`, and reaches every other one on it as if on one switch —
ARP, broadcast, UDP, anything Ethernet — wherever they are. Nothing to
list: each node finds the bucket's other `lan` nodes by itself. Frames
ride the fleet's own stream sessions (`io.open`): agent to agent when
both nodes are LAN neighbours (the node's `direct` setting, `all` by
default; `files` keeps live legs on the relay), through the backend relay
otherwise. No VPN, no port to open: each node needs its `3pm link`
and two grants. Made for LAN games run under Wine on the tablets
(discovery by broadcast, then UDP between peers).

```console
$ 3pm module list                        # lan: installed by itself (auto)
$ ip -br addr show 3pmlan0               # 3pmlan0  UNKNOWN  10.77.143.12/16
$ 3pm trigger lan lan.status             # peers, sessions (dir, up, call, stats), MACs, warnings
$ ping 10.77.201.7                       # another node of the bucket
```

**Grants.** A node finds its peers with `3pm fleet nodes --json` and
`3pm fleet apps --json` (its own bearer, `getBucket`): it needs
**`fleet:list`**. It opens sessions with **`fleet:invoke`**. Both are
given by the bucket's owner with `setNodeAccess` (`POST
/api/buckets/{cluster}/nodes/{node}/access`, `{"scopes":["fleet:list",
"fleet:invoke"],"direct":"all"}` — the list replaces the old one). A node
without `fleet:list` logs the backend's refusal once, names both grants,
and keeps running with the peers it knew (none at first); `lan.status`
shows the error under `discovery`. A peer whose `callers` hold rules
must admit `{from:"*", from_app:"lan", app:"lan", action:"link"}`.

**Who talks to whom.** At start, 5 s later (the bucket lists this
node's own app only once it is up, and the others find it then), every
15 s while it found nobody, then every minute, and after a failed open
(at most every 10 s) the module reads the bucket: the nodes `online` with an `online` app `lan` of the
module `lan`, itself excluded. `allow` in `lan.toml` (formerly `peers`,
still read) narrows that to an allow-list (node ids or hostnames);
empty, the whole bucket. Identity is
the **node id**: the lower id dials (`io().open(peer, "lan", "link",
{from: <own id>, mac, address})`), the other serves the `link` stream
action. A session whose `from` the last discovery did not list starts
a discovery at once and waits for it (≤ 3 s) before it is judged again;
only a `from` still not a `lan` node of the bucket is refused, before
any metric series or log line of its own is made for it. If both dialled, each keeps the lower id's session. Hostnames
are only shown. A peer that left the listing is not dialled again, but a
live session to it stays: it is its own proof.

**Trust boundary: the bucket.** Only a member of the bucket holding
`fleet:invoke` can place a session (the backend checks), but the module
reads the caller's id from the session's params: until the daemon hands
actions a verified `caller` (protocol work to come), a bucket member can
claim another member's id. A bucket is one trust domain for `lan`.

**At most 14 peers dialled.** A daemon holds 16 sessions. Past 14 `lan`
nodes this side dials the 14 nearest (a symmetric distance of the two
ids' FNV hashes, stable whatever the order of the listing), serves the
others while it holds fewer than 15 sessions (the 16th is kept for a
renewal), warns in the log and in
`lan.status` (`warnings`), and dials anyway a peer that stayed 30 s
without a session — the nearest relation is not symmetric.

`lan.toml` (optional) is read from `/etc/3pm/lan.toml`, else
`$THREEPM_HOME/lan.toml`: `iface` (`3pmlan0`), `network` (`10.77.0.0/16`),
`address` (default: a host of `network` derived from the node id —
stable; two nodes that collide refuse each other's sessions, the
conflict is logged and listed once in `lan.status` `warnings`: set
`address` on one), `allow` (the allow-list; `peers` still read),
`broadcast_route` (default
`false`: adds `255.255.255.255/32 dev 3pmlan0`), `rate_kbit` (default
`4000`, what one peer may be sent, `0` = no limit), `expect_direct`
(default `false`), `netns` (absent: off) and `netns_address` (see
below). An unknown key is a warning. The MAC is locally
administered and derived from the node id too.

**A namespace for programs that will not use the lan.** Some bind to
the interface of the default route whatever address they are told (AoE3
under Wine: `0.0.0.0%wlp1s0:2299`), so a host never answers over the
bridge. With `netns = "3pmlan"` the module also makes a network
namespace holding only the lan, for such a program to run in:

| | |
|---|---|
| path | `/run/netns/<netns>` (`ip netns add`: what `ip netns exec <netns> …` or a launcher's `setns` enters) |
| interface | `3pmlan1`, a macvlan of `3pmlan0` in `bridge` mode, MTU 1500, up; `lo` up |
| address | `netns_address`, else derived from the node id with a salt (never the host's), on the lan's network, `broadcast +` |
| MAC | locally administered, derived from the node id (its own, not the TAP's) |
| routes | the lan's network on `3pmlan1`, `255.255.255.255/32 dev 3pmlan1`; **no default route** |

The host keeps `3pmlan0` and its address. The namespace's frames leave
through the TAP like the host's own, and the switch carries them the
same way; a frame for its MAC written into the TAP reaches it. A macvlan
in `bridge` mode does not talk to its lower device: the host's `3pmlan0`
and the namespace do not reach each other locally (each reaches every
peer). The namespace lives with the device: made only once a peer
shows up, deleted with it — 10 minutes after the last peer left, or
when the app stops (a program still inside keeps an empty one). One
left by a previous run is reused and its port remade, so a program
inside survives a `lan` restart only if it re-binds. `lan.status` shows `netns` (`name`, `path`,
`iface`, `address`, `mac` as read from the device). The expected use: a
tabloid frame launches its Wine game inside `/run/netns/3pmlan`. The
namespace's address is not part of the hello: a collision with a peer's
is not detected — set `netns_address` when in doubt.

**The MAC is read back.** udev (`MACAddressPolicy=persistent`) renamed
a fresh TAP's MAC just after boot on one tablet. The module waits for
udev (`udevadm settle -t 5`, when there is one), sets the MAC, reads it
back and sets it once more if it moved — the TAP and the namespace's
port alike; `lan.status` reports the MAC the device has, and the frames
a peer sends back with any of this node's own MACs are dropped.

**It needs a root daemon.** Module apps run as the daemon's account, and
a TAP is created by root (`CAP_NET_ADMIN`). Under a user daemon the app
stays idle and `3pm logs lan` says so once; on macOS it says Linux only.
A host not yet linked waits for its link and goes on by itself. The
device is not persistent: it goes with the app (and with the last peer,
10 minutes later), and every session goes with it — given up, its peer
announced down. A session from a peer while this node has no device
first waits (≤ 3 s) for a discovery that may make one; refused then, it
says why — as `lan-idle` when the device cannot be made here (the
network overlaps, the TAP will not open). A TAP held by another process
(`EBUSY`: the previous `lan`, still exiting) is not that: no `lan-idle`,
tried again at the next discovery. The mesh dials from the moment the
device is made and stops when it goes, whatever a discovery under way
concluded.

**Sessions.** Reopened with backoff (5 s doubling to 10 min, ±20 %,
back to 5 s once a session carried a frame) and renewed after 23 h, the renewal
respecting the backoff: the new session takes over at its first frame
back, then the old one is retired. The serving side does the same: a new
session from a peer whose served session works (up, a frame within the
last 50 s) waits beside it and takes over at its first frame — one
renewal at a time, a third replacing the second; one that never comes
up ends with no backoff, and an old one that ends first leaves the new
one in its place — so neither side announces the peer down (no
`lan.peer_down`/`lan.peer_up` pair) at a renewal. A served session
that is silent, or never came up, is replaced at once. A daemon reconnect does not end a
dialled session (the reply grace is set past a session's day). Frames
written before the other side's reader joined are lost, so the opener
sends a hello (12 bytes, `3pmlan/1` + its address) every 250 ms until a
frame comes back, and is not routed to before; the server answers every
hello; an address in a hello is checked, either direction, before the
session counts as up. Every 25 s, all sessions together, each one quiet
for 12.5 s gets its keepalive — a ping, echoed (a leg is cut after a
minute without a byte); an empty frame, an older peer's keepalive, is
still skipped. A session this side dialled that has been silent for 50 s, or
never came up within 5 s, gives way to a new one from the same peer (it
hard-reset). Anything shorter than an Ethernet header (14 bytes) is
control: skipped when unknown.

A learning switch sits between the TAP and the sessions: a frame from a
peer teaches it the source MAC (300 s ageing, 4096 entries) and goes to
the TAP only; a frame from the TAP goes to the peer its destination was
learned behind, or — broadcast, multicast, unknown — to every session
up, one buffer shared by all (`send_shared`). The mesh is full, so
nothing a peer sent is ever sent on: no loop.

**Direct or relay, and the rate limit.** Measured a9b4 ⇄ Mac on one LAN
(300 B × 1000/s): direct p50 ~11 ms, p99 ~100 ms, loss under 0.2 %;
through the relay p50 46 ms, p99 ~480 ms. Bulk through the relay
collapses (1400 B × 3000/s: 10 s RTT, 72 % loss), because the backlog
builds downstream of the SDK's sink where this side cannot see it. So
what goes to each peer passes a token bucket (`rate_kbit`, 4 Mbit/s by
default, a 100 ms burst of at least 16 KiB; no bucket at all for `0`):
the excess is dropped and counted (`lan.dropped`, and `limited` per
session in `lan.status`), never queued. The module cannot see which
path a session took: at each `peer_up` it warns, unless `expect_direct =
true`, naming the call id — the call's `path` says direct or relay. The
`lan.peer_up` event carries `call_id` too. Known gap: with
`expect_direct = true`, nothing reports a session that fell back to the
relay; look the call up (`getCall`, its `path`).

**The latency guard: a peer's round trip cuts its rate.** On the relay
path a slow uplink keeps frames *in flight*, where no socket option
reaches: between a tablet on a 5G hotspot and one on Wi-Fi (2026-10-05),
ping over the lan climbed 0.18 s → 0.9 s → 2.8 s → 34 s under an AoE3
game and some ssh, until the uplink stalled. So each session measures
its round trip and sends less while it is long:

| | |
|---|---|
| probe | a ping (`3pmp?` + an 8-byte stamp, 13 bytes) the peer echoes as a pong (`3pmp!`): measured on this side's clock alone, the two hosts' clocks never compared |
| when | every second while the session carried an Ethernet frame within the last 5 s; idle, the 25 s keepalive **is** the ping — an idle node still wakes once per keepalive, and every session, up or not, still gets a frame at least that often |
| round trip | the last pong's, or the age of the oldest ping still out when longer (a stalled uplink answers nothing); nothing until a first pong — a peer older than the guard skips both frames and is never throttled |
| over 1 s | the peer's `rate_kbit` is halved, at most once per round trip, down to 64 kbit/s; logged at each cut |
| under 400 ms | it grows back by a tenth of `rate_kbit` per ping (≈ 10 s from the floor) |
| `rate_kbit = 0` | measured and shown, never acted on |

`lan.status` shows per session `rtt_ms` (smoothed), `rtt_last_ms` and
`rate_kbit` (in force); the histogram `lan.rtt` (ms, label `peer`)
records every sample. An idle keepalive's ping measures but never cuts.
Below the module, the agent's own `live` legs keep their queues shallow
(see "Fleet RPC": a `live` leg's connection), so what the rate lets
through is not then held for seconds in a socket.

**Closing without breaking the reply.** A session given up (replaced,
renewed, a duplicate, a conflict) is ended by a reaper thread, read
until the other side's `END` by its own reader, and closed 2 s later: in
direct mode a `close` right after `end` makes the peer's reply fail
(502, "the reader went away").

Metrics `lan.frames` and `lan.bytes` (Ethernet only — hellos, pings
and pongs are not counted; labels `peer` = node id, `dir` = `in`/`out`),
`lan.dropped` (no session up, over the rate, a refused send, the TAP
refusing a write), `lan.peers` (gauge), `lan.rtt` (histogram); events `lan.peer_up` and
`lan.peer_down` (with the reason), once per peer whatever its sessions
do underneath.

**How it works.** `modules/lan/`, a standalone workspace on `threepm-io`
(streams, metrics, events, actions), `libc` for the one `ioctl`,
`rustix`, `serde` + `toml`: `src/main.rs` (the app: the two actions
declared at once, then the setup on its own thread, or idle with the
reason), `src/config.rs` (the file, the defaults, the
derived address and MAC), `src/discovery.rs` (the two `fleet` documents
joined, the `3pm` binary found: `$THREEPM_EXE`, the daemon's own
`/proc/<ppid>/exe`, the shipped layouts, `PATH`), `src/tap.rs`
(`/dev/net/tun`, `TUNSETIFF` with `IFF_TAP | IFF_NO_PI`; the only file
allowed `unsafe`), `src/mesh.rs` (one slot per peer:
current and pending session, who dials, cap, backoff, renewal,
duplicates, conflicts, idle peers, and the switch — pure, one lock),
`src/ip.rs` (the `ip` commands: the TAP's MAC read back, address,
broadcast route, the namespace and its macvlan), `src/switch.rs` (the
learning table, pure), `src/bucket.rs` (the token
bucket, pure), `src/guard.rs` (the latency guard, pure),
`src/wire.rs` (hello, ping and pong, an older peer's empty keepalive),
`src/link.rs` (one session: bucket, guard, series), `src/links.rs` (the
data path), `src/status.rs` (`lan.status`), `src/sync.rs` (locks that
survive a panic), `src/device.rs` (the device made with the first peer and removed 10 min after the last, the network checked against the host's routes first, the TAP reader polling so it can stop), `src/sched.rs` (a scheduler asleep until the next deadline
— an idle node wakes once per keepalive —, the reaper, the discovery). Unit tests per file on any OS, including every
interleaving of a crossed dial; `tests/e2e.rs` against a real daemon
(the refusal under a user daemon, the binary run by hand), and two
root tests (`#[ignore]`, run with `sudo unshare -mn`): the TAP carries a
frame, and a namespace's limited broadcast leaves on the TAP while a
frame written into the TAP for its MAC reaches a socket inside.

## Bench fleet

**What you see.** One command fills the table with a realistic mix so
every column has something to show: 4 node apps (an HTTP server, a CPU
burner, a memory ballooner, one that crashes every 15 s), 2 python apps
(an HTTP server, a JSON-ticking worker) and 2 shell loops (`ls -la /`,
a poor man's `watch`). HTTP apps get `PORT=0` so the port column shows
the OS-assigned port; the crasher has `min_uptime 1000` / `restart_delay
500` so its restart counter climbs without ever tripping the crash-loop
cutoff. Rerunning restarts the same apps in place; `--delete` (alias
`--clean`) removes them, tolerating a fleet that is already half gone.
A missing `node` or `python3` skips those apps with a warning instead of
failing the run.

**How to get it.**

```console
$ 3pm bench                       # writes ~/.3pm/bench/*.{js,py,sh}, starts 8 apps
$ 3pm bench --dir ./demo          # scripts somewhere else
$ 3pm logs bench-py-worker        # {"tick": 42, "pid": …}
$ 3pm bench --json                # same Affected contract as `start --json`
$ 3pm bench --delete
```

**Where it comes from.** `commands/bench/scripts.rs` is the single
source of truth (names, files, bodies); `bench/fleet.rs` turns it into
`StartSpec`s through `SpecPatch::apply` and `start::start_specs` runs
the same start loop as `3pm start`. Every app is in the `bench`
namespace. Covered by unit tests on the table/specs and
`tests/e2e_bench.rs` (real node/python/bash, restart of the crasher,
delete idempotence).

</document>

<document url="https://3pm.edgecr4ft.net/docs/logs.md" title="Logs">

# Logs Reference

## Cheatsheet

```console
# read
3pm logs                                 # all apps: last 30 lines, then follow
3pm logs api --lines 200 --rotated       # reach into api-out.log.1, .2.gz…
3pm logs api --err --timestamp           # stderr only, stamped on arrival
3pm logs api --timestamp='%H:%M:%S'      # custom strftime
3pm logs api --format                    # logfmt: ts= app= id= instance= type= message=
3pm logs api --json --lines 0            # wire events only (NDJSON), no history
3pm logs api --highlight ERROR           # bright-black background on matches
3pm logs api --raw --nostream            # bare lines, tail and exit
3pm logs --size                          # file sizes + rotated generations
3pm show api                             # paths, sizes, date format, rotation policy

# write-side setup
3pm start app.js --time                  # file lines prefixed "%Y-%m-%dT%H:%M:%S: "
3pm start app.js --log-date-format '%H:%M:%S'
3pm start app.js -l /var/log/app.log     # one merged file (stdout + stderr)
3pm start app.js --merge-logs            # merged, default path: logs/app.log
3pm start app.js -o /var/log/app.out -e NULL      # custom stdout, no stderr file
3pm start w.js -i 4 -o '/var/log/{name}-{instance}.out'   # {name} {instance} {namespace}
3pm start app.js --disable-logs          # no file, no stream, no crash tail

# operate
3pm flush                                # empty every log file + 3pm.log, in place
3pm flush api                            # one app
3pm rotate api                           # rotate now (→ .1), regardless of size
3pm reloadLogs                           # reopen by path after an external mv
kill -USR2 $(cat ~/.3pm/3pm.pid)         # same as reloadLogs (daemon log included)
3pm delete api --flush                   # delete the app AND its log files
3pm logrotate --print                    # logrotate(8) copytruncate drop-in
sudo 3pm logrotate --user $USER          # install it in /etc/logrotate.d
3pm internals                            # effective policy, open files, rotations, bytes

# rotation policy (per app > env > config.toml > defaults 20M / 5 / 14d / no gzip)
THREEPM_LOG_MAX_SIZE=50M THREEPM_LOG_MAX_FILES=10 THREEPM_LOG_COMPRESS=1 3pm ls   # env → daemon
THREEPM_LOG_MAX_SIZE=0 ...               # rotation off
printf '[logs]\nrotate_size = "50M"\nrotate_days = 7\n' > ~/.3pm/config.toml
```

```toml
# ecosystem.toml, per app
[[apps]]
name = "api"
script = "./server.js"
log_date_format = "%Y-%m-%d %H:%M:%S"
merge_logs = false                        # or log_file = "/var/log/api.log"
out_file = "/var/log/{name}-{instance}.out"
error_file = "NULL"
log_rotate_size = "50M"                  # "0" = off
log_rotate_files = 10
log_rotate_days = 7
log_rotate_compress = true
```

Daemon restart needed for env/config.toml changes (`3pm update`).

---

Everything about how 3pm captures, stores, rotates and serves application
output. Mechanism rationale lives in OV `3pm-docs-archived/architecture` (§6.4 log pumps,
§7 logs pipeline); this is the field-level reference. User-facing
walkthrough: `../FEATURES.md` ("Logs").

## 1. Capture

Each managed process is spawned with `stdin` null and `stdout`/`stderr`
piped. Two pumps per incarnation (`child_task::pump`) read the pipes:

| Property | Value | Why |
|---|---|---|
| Line reader | `read_until(b'\n')` capped at `MAX_LOG_LINE` = 64 KiB | a child streaming GBs without a newline must not grow the daemon |
| File content | the child's bytes, verbatim | pm2 behavior; no UTF-8 conversion on the persistence path |
| Bus content | UTF-8 lossy, trailing `\n`/`\r\n` stripped | readers get text |
| Cap hit mid-line | `\n` appended in the file | the file stays line-structured |
| Invalid UTF-8 / write error | never kills the pump | a wedged pump stops logs *and* eventually blocks the child on a full pipe |
| Write errors | logged once, not per line | a full disk must not flood `3pm.log` at the app's output rate |
| Crash tail | stderr pump feeds a 30-line / 4 KiB ring | attached to the `exit` event and to `ProcessRecord.stderr_tail` |

Under `disable_logs` the pumps still drain the pipes but write nothing,
emit nothing and keep no crash tail.

### An app on a terminal (`tty = true`)

A `tty` app runs on a pseudo-terminal: **one** output stream (the pty
master), so **one** pump (`child_task_tty.rs`) and no stderr. Each read
(≤ 16 KiB) feeds the daemon's screen of the app first (what `3pm attach`
shows), then the logs:

| Property | Value | Why |
|---|---|---|
| Destination | one file: `<app>.log` / `<app>-<i>.log` (the merged name), or the `log_file`/`out_file` given; `err_log` is `""` | a terminal has no second stream; `error_file` with `tty` is refused by `check_fields` (`apply`, `validate`, the daemon) |
| File, bus, crash tail content | the output **stripped of escape sequences** (`threepm-core::ansi::Stripper`, streaming: a sequence cut by a read is still recognised) | a TUI's bytes are cursor moves and colours; the log must read as text |
| Stripped | CSI, OSC/DCS/APC/PM/SOS strings, short escapes, C0 controls but `\n` and `\t`, DEL; `\r\n` → `\n`, a lone `\r` dropped | |
| Horizontal gaps | a move right on the line (`CSI n C`, `CSI n G`, ``CSI n ` ``) after text becomes one space | TUIs draw word gaps that way (`Accessing\e[12Gworkspace` logs `Accessing workspace`) |
| Line cap | `MAX_LOG_LINE`, as for pipes | |
| Crash tail | the last lines of the one stream (`3pm show`: "last output before exit") | there is no stderr |
| `disable_logs` | no file, no bus line, no crash tail — the screen is still fed | the exit for a TUI that redraws too much to be worth a log |
| Keystrokes | never logged (what the app echoes is output, like any app's) | |

Across `3pm update` the pump stops at a read boundary: the partial
stripped line is delivered first by the next pump, and what the read
ended inside of (an escape sequence, a UTF-8 character) is replayed to
the next screen and stripper, so neither is logged twice nor as text.

## 2. Destinations

Resolved by the pure `Paths::log_targets(spec, instance)`
(`threepm-core/paths.rs`) on every spawn; the daemon re-targets only when
the result changed (a re-spec takes effect on its restart, the old file is
left alone).

| Situation | stdout | stderr |
|---|---|---|
| default, instance 0 | `<home>/logs/<app>-out.log` | `<home>/logs/<app>-error.log` |
| default, instance i ≥ 1 | `<app>-<i>-out.log` | `<app>-<i>-error.log` |
| `merge_logs` | `<app>.log` / `<app>-<i>.log` (one file) | same file |
| `log_file = P` | P (implies `merge_logs`) | P |
| `out_file` / `error_file` | as given | as given |
| `"NULL"` / `"/dev/null"` | no file (still on the bus) | no file |
| `disable_logs` | nothing | nothing |
| `tty` | `<app>.log` / `<app>-<i>.log`, or `log_file`/`out_file` | none (one stream; `error_file` refused) |

Custom paths:

- Relative paths resolve against the app's `cwd` (ecosystem file: the
  config's directory); `~/` expands. The daemon creates parent directories.
- Placeholders: `{name}`, `{instance}`, `{namespace}` (`default` when the
  app has none).
- **Multi-instance guard.** A custom path on an app with `instances != 1`
  that does not contain `{instance}` is infixed `-<instance>` before its
  extension for instances ≥ 1 (`/var/log/app.log` → `/var/log/app-1.log`);
  instance 0 keeps the path as written. `3pm start` warns once. Property
  test: no two instances of one app ever resolve to the same file.
- Two *different* apps pointed at one file share it: one handle, writes
  serialized, one rotation (the first opener's policy wins; the daemon
  logs a warning when policies differ).

`ProcessRecord.out_log` / `err_log` carry the resolved paths (`""` = no
file; equal when merged). `3pm show` and `3pm logs --size` add on-disk
sizes.

### An app that runs as another account (`user`)

A root daemon that hosts users (`[users]` below) writes an app's files in
the **home of the account the app runs as** — files follow `spec.user`,
visibility follows `spec.owner`:

```
/var/lib/3pm/users/srt/logs/<app>[-<i>]-out.log[.N[.gz]]   root:srt 0640
/var/lib/3pm/users/srt/pids/<app>[-<i>].pid                root:srt 0640
```

The mode and group are set **explicitly** (not left to the umask), on the
live file and on every generation a rotation produces — `.N` and `.gz`
alike — and again after a `flush` truncates in place. So root keeps
writing and the account (through its private primary group, and nobody
else) keeps reading: `3pm logs` of that user tails its own files
directly, and the home itself is `root:<gid> 0750`, so it cannot replace
them.

An app a hosted user starts may **not** choose its log paths:
`out_file`, `error_file` and `log_file` are refused (`"NULL"` to switch a
stream off still works). A path of its choosing would have root open — and
`delete --flush` unlink — any file on the machine.

## 3. Line format

- **File**: raw bytes. With `log_date_format`, each line is prefixed
  `"<strftime>: "` in local time (pm2 shape). The format is validated at
  `3pm start`; unknown specifiers fail there. `--time` on the CLI is
  shorthand for `%Y-%m-%dT%H:%M:%S`. There is no CLI-only default.
- **Bus** (`pub.sock`, NDJSON): the bare line plus metadata. Events are
  additive — older peers ignore the extra fields.

```json
{"event":"log","id":1,"name":"api","stream":"out","line":"hello","instance":0,"ts":1787669351920}
{"event":"log","id":1,"name":"api","stream":"err","line":"boom","instance":0,"ts":1787669351933}
{"event":"log_rotated","id":1,"name":"api","instance":0,"path":"/home/u/.3pm/logs/api-out.log"}
{"event":"lagged","missed":12}
{"event":"daemon_shutdown","reason":"kill"}
```

`ts` is the daemon's receive time, ms since the Unix epoch, and it is
always there: a history line the CLI replays from a file carries the
**live** file's last write, the closest thing this side knows. Every
line of one tail therefore shares a stamp, and with `--rotated` the
lines read from older generations share it too — a rotated file's own
mtime is when it was rotated, not when its lines were written, so it
would be no truer.

## 4. Rotation

Built in, **hot**: the sink (`daemon/logsink.rs`) counts bytes as it
writes; the write that crosses `max_size` lands, then — still under the
sink's lock — the generations are shifted, the live file becomes `.1`
and a fresh live file is opened. Every writer shares the same handle, so
all of them follow. No restart, no lost or torn line.

Layout: `<file>` live, `<file>.1` freshest generation … `<file>.N`
oldest, `.gz` suffix when compressed. Planning is pure
(`threepm-core/logrotate.rs`):

- remove generations that would exceed `max_files`, shift `.N → .N+1`
  oldest-first (never clobbers), move live to `.1`;
- `max_files = 0`: discard everything, truncate the live file in place;
- `compress`: gzip `.2` and older (`.1` stays plain so it can be tailed).
  Runs on a blocking thread but under the sink lock — released, a
  concurrent rotation could rename `.N` mid-read and the wrong plain
  file would be deleted. Output is written to `.gz.tmp` and renamed;
- age pruning (`max_age_days`) piggybacks on each rotation and on an
  hourly housekeeping tick (`LOG_HOUSEKEEPING_INTERVAL`), so retention
  holds for files that rarely rotate.

### Policy

Most specific wins, field by field:

| Layer | Keys |
|---|---|
| per app (ecosystem / `SpecPatch`) | `log_rotate_size` (human: `"10M"`, `"0"` = off), `log_rotate_files`, `log_rotate_days`, `log_rotate_compress` |
| daemon environment | `THREEPM_LOG_MAX_SIZE`, `THREEPM_LOG_MAX_FILES`, `THREEPM_LOG_MAX_DAYS`, `THREEPM_LOG_COMPRESS` (`1/true/yes/on`) |
| `<home>/config.toml` | `[logs] rotate_size`, `rotate_files`, `rotate_days`, `rotate_compress` |
| defaults | `20M`, `5`, `14`, `false` |

The daemon resolves its policy once at startup (`logsink::daemon_policy`)
and logs it; invalid values are logged and skipped, never fatal. The
effective daemon policy is in `3pm internals` (`log_rotate_*`), a
per-app override in `3pm show` ("log rotation").

### Rotation vs. external logrotate(8)

Both work. `3pm logrotate --print` renders a `copytruncate` drop-in
(`sudo 3pm logrotate` installs it in `/etc/logrotate.d/3pm-<user>`).
For `create`/move mode, add `postrotate kill -USR2 $(cat <home>/3pm.pid)`
or run `3pm reloadLogs`: every open log (and the daemon's own `3pm.log`)
is reopened by path.

## 5. Commands

| Command | Effect |
|---|---|
| `3pm logs [target…]` | tail then follow (see §6); several targets are a union, a name matching nothing is a notice |
| `3pm logs --size [target]` | per-file sizes and rotated generations; `--json` = `[{id,name,instance,stream,path,bytes,generations:[{path,bytes,compressed}]}]` |
| `3pm flush [target]` | truncate live files in place; no target = every app **and** `3pm.log`. Generations untouched |
| `3pm rotate [target]` | rotate now regardless of size |
| `3pm reloadLogs` | reopen every log by path (SIGUSR2 equivalent) |
| `3pm delete --flush <target>` | remove the rows' live + rotated files, except a file a surviving row also writes to |
| `3pm logrotate [--print] [--user U]` | logrotate(8) drop-in |
| `3pm show <target>` | paths, sizes (+ generations), date format, rotation policy |
| `3pm internals` | `log_sinks`, `log_bytes_written_total`, `log_rotations_total`, effective policy |

`flush` / `rotate` / `reloadLogs` reply `Response::LogsOp { touched }`
(the paths acted upon); `--json` prints `{"touched":[…]}`. A target is a
name, id or namespace.

### `3pm start` flags

`-o/--output PATH`, `-e/--error PATH`, `-l/--log PATH` (conflicts with
`-o`/`-e`), `--merge-logs`, `--log-date-format FMT`, `--time`,
`--disable-logs`. Rotation fields are ecosystem-only.

## 6. Reading: `3pm logs`

Order: resolve targets → **subscribe** to `pub.sock` → tail files →
stream. Subscribe-first trades a possible duplicate line for the
guaranteed gap of tail-then-subscribe on a chatty app, and it is why a
follower crosses a rotation without noticing (realtime never comes from
file watching).

| Flag | Default | Effect |
|---|---|---|
| `--lines N` | 30 | history per file (`0` = none) |
| `--rotated` | off | when the live file holds fewer than N, continue into `.1`, `.2`, … (gzip decoded through a bounded deque), chronological |
| `--err` / `--out` | both | one stream. On a merged file the tail is printed once (as stdout); the filter applies to the stream only |
| `--raw` | off | no prefix, no color |
| `--timestamp[=FMT]` | off; `%Y-%m-%d %H:%M:%S` | stamp streamed lines from `ts`; history lines get blank padding of the same width |
| `--format` | off | logfmt: `ts=<rfc3339> app= id= instance= type=out\|err message=` (quoted/escaped as needed; `ts=` omitted on history) |
| `--highlight TEXT` | — | bright-black background on matches (color only) |
| `--nostream` | off | tail and exit |
| `-f/--follow` | — | accepted, hidden (following is the default) |
| `--json` | off | wire events, tail and stream in one NDJSON schema, `process` and `lagged` events included |

Text mode: stdout lines to stdout, stderr lines to stderr, in every mode.
Prefix `id|name | ` green/red (pm2 look). `[3PM] log rotated: …` and
`[3PM] log stream lagged, N lines missed` go to stderr as chrome.

## 7. Files under `<home>`

```
logs/<app>[-<i>]-out.log[.N[.gz]]     stdout (default naming)
logs/<app>[-<i>]-error.log[.N[.gz]]   stderr
logs/<app>[-<i>].log[.N[.gz]]         merged
3pm.log                               the daemon's own log (tracing; THREEPM_LOG=debug …)
config.toml                           optional; [logs] rotation policy, [users] hosted accounts,
                                      [hostmetrics] Extended Monitoring, top processes (hostagent.md),
                                      [modules] off = ["lan"]: auto modules switched off (modules.md)
hosted                                only in a hosted home: the root daemon owns this home
```

### `[users]` in `config.toml`

Read once, at daemon start (`3pm update` to apply a change), and only by a
**root** daemon — anywhere else the section is ignored with a warning:

```toml
[users]
allow = ["srt"]              # accounts this daemon hosts; each needs a private
                             # primary group (`useradd -m -U srt`), and uid 0 is refused
# dir = "/var/lib/3pm/users" # where their homes go; root-owned ancestors only
# max_apps = 32              # most instances (rows) one account may run
```

Every problem — a name nobody has, a shared primary group, a `dir` anyone
but root may write — is logged and skips that user (or the section). A
typo must not keep the root daemon, and every app on the machine, down.
Full walkthrough: `../FEATURES.md` → *Hosted users*.

Log files survive `3pm delete` (use `--flush`), `3pm kill` and daemon
restarts: sinks reopen in append mode and seed their byte counter from
the file's size, so rotation thresholds carry over.

## 8. Invariants (tested)

- Two instances of one app never share a file (proptest on
  `log_targets`).
- No byte is lost across a rotation chain; every generation is
  line-structured (component + e2e).
- A sink shared by several writers follows a rotation (all handles are
  the same `Arc`).
- A rotation never renames a file another rotation is compressing
  (gzip under the lock).
- `disable_logs` drains the pipe (200 KiB written by the mock, nothing
  persisted, nothing on the bus, empty crash tail).
- A `tty` app logs text: no ESC nor C0 control in the file, any split
  of the stream into reads strips like one read, stripping is idempotent
  (proptests in `ansi.rs`); two `tty` instances never share a file
  (`log_targets`); the e2e (`e2e_tty.rs`) reads the log of a real pty.
- Wire growth: `ts`, `instance`, `log_rotated`, `LogsOp`, `Delete.flush`
  are all `serde(default)` / lenient-decoded.

Where things live: `threepm-core/{spec,paths,logrotate,ansi}.rs` (pure),
`threepm-daemon/{logsink,child_task,child_task_tty}.rs` + `supervisor/`
(`do_spawn`, `FlushLogs`/`ReloadLogs`/`RotateLogs`, `LogHousekeeping`),
`threepm-cli/commands/{logs,logsops,logrotate,start}.rs`,
`threepm-cli/render/{logs,show}.rs`.

</document>

<document url="https://3pm.edgecr4ft.net/docs/resources.md" title="Resource limits">

# Resource Limits Reference

`max_memory_restart`, `max_cpu_percent`, `limit_scope`, `cgroup_enforce`,
`deny_gpu`. The first two are *soft* guards (observe a sample of the
app's whole process tree, restart gracefully); `limit_scope` narrows
what they judge to the spawned pid; `cgroup_enforce` turns the same two
numbers into *hard* kernel limits on Linux; `deny_gpu` hides
accelerators through environment variables.

## Cheatsheet

```console
# soft guards: the process tree sampled every second, restart when a sample crosses the line
3pm start api.js --max-memory-restart 300M          # RSS of the whole tree > 300 MiB → graceful restart
3pm start worker.py --max-cpu-percent 80            # CPU > 80 % of one core → restart
3pm start train.py --max-cpu-percent 250            # multi-core: 250 % = 2.5 cores
3pm start api.js --max-memory-restart 1G --max-cpu-percent 120   # both

# hard limits (Linux, cgroup v2): the kernel throttles / OOM-kills
3pm start api.js --max-cpu-percent 50 --cgroup-enforce           # cpu.max = 50000 100000
3pm start api.js --max-memory-restart 512M --cgroup-enforce      # memory.max = 536870912
3pm start api.js --cgroup-enforce                                # own cgroup, no caps (accounting only)

# GPUs
3pm start infer.py --deny-gpu                       # CUDA_VISIBLE_DEVICES="" NVIDIA_VISIBLE_DEVICES=none …

# inspect
3pm ls                                              # cpu / mem = the tree's totals, `+N` = its other processes
3pm ls --tree                                       # who weighs what under each app; ⚠ = a stop will not reach it
3pm show api                                        # restarts count, every process, "last error" if enforcement failed
3pm logs api --err                                  # the app's own view (OOM kill = SIGKILL, exit signal 9)
cat /sys/fs/cgroup/3pm/api-1-0-*/cpu.stat           # nr_throttled, throttled_usec (Linux; leaf = <name>-<id>-<instance>-<lineage>-g<gen>)
cat /sys/fs/cgroup/3pm/api-1-0-*/memory.events      # oom_kill counter (Linux)

# change limits: restart on the new spec
3pm start api.js --max-cpu-percent 60               # same name = update spec + restart
3pm restart api --update-env                        # re-read the ecosystem file it came from
```

Ecosystem file (`3pm start ecosystem.toml`):

```toml
[defaults]
max_cpu_percent = 200        # every app starts from this

[[apps]]
name = "api"
script = "./server.js"
max_memory_restart = "300M"  # "512", "100K", "200M", "1G"
max_cpu_percent = 80         # float, > 0; overrides [defaults]
limit_scope = "tree"         # what both judge: "tree" (default) or "leader" (the spawned pid alone)
cgroup_enforce = true        # Linux only
deny_gpu = true

[apps.profiles.dev]          # 3pm start ecosystem.toml --env dev
cgroup_enforce = false       # laptops have no delegated cgroups
```

## 1. Soft guards: `max_memory_restart`, `max_cpu_percent`

The metrics sampler reads the process table once per second
(`THREEPM_METRICS_INTERVAL_MS` shrinks it for tests) and attributes to
each instance its **process tree** — the spawned pid and every
descendant; the rules are in `docs/metrics.md` → *Process trees*. Each
sample is fed to the pure policy as `MemorySampled{gen, tree_rss,
leader_rss}` and `CpuSampled{gen, tree_cpu, leader_cpu}`; the policy —
not the daemon — picks the figure named by the spec's `limit_scope`
(`tree`, the default, or `leader`), compares it to the limit and, when
**strictly above**, stops the app gracefully — `kill_signal` (default
SIGINT) to the process group, SIGKILL after `kill_timeout` — then
restarts it as after a crash: `restart` event, `waiting restart` for
`restart_delay` / the exponential backoff, respawn, restart counter +1.

What that implies:

- **The whole tree is judged.** `max_memory_restart = "300M"` fires when
  the members' RSS *summed* crosses 300 MiB, even if no single process
  does — a browser, a build or a helper that grows recycles the app.
  That is what `3pm ls` shows (`+N`, and `--tree` for who weighs what).
  An app whose children are meant to grow keeps the old meaning with
  `limit_scope = "leader"` (per app, in the ecosystem file or the
  `@3pm/api` start options — no CLI flag: the spawned pid alone is
  judged, the sampler still measures the tree);
  `THREEPM_PROCESS_TREE=off` on the daemon does it for every app.
- **A detached member counts but is not restarted.** A process that
  left the group (`setsid`; `⚠` in `3pm ls --tree`) weighs in the sum,
  and the restart's signals go to the group: it survives the restart,
  and no later incarnation adopts it — its environment names the old
  incarnation, and after a cold restart of the daemon an older lineage
  too. Kill it by hand, or `limit_scope = "leader"`; under
  `cgroup_enforce` it dies with its leaf when the incarnation ends.
- **One sample is enough.** A one-second CPU spike above `max_cpu_percent`
  recycles the app. Size the limit for the app's steady state, not its
  peak, or leave the CPU guard off and use `cgroup_enforce` to throttle
  instead of restarting.
- **The first sample after a spawn reads 0 % CPU** (CPU% is a delta
  between refreshes). A guard therefore fires no sooner than ~2 s after
  start.
- **Limits only fire on `online` rows.** A row that is `launching`,
  `stopping`, `stopped`, `errored`, or being shut down with the daemon
  ignores samples. A stale sample for a superseded incarnation (gen
  mismatch, pid reuse) is dropped.
- **Restart budget applies.** A guard restart is judged like a crash,
  on the incarnation's uptime **at the trip** (the graceful stop does
  not count), counted from the moment it went `online` — its spawn,
  or under `wait_ready` its ready signal or `listen_timeout` — against
  `min_uptime` + 10 s (`GUARD_SETTLE_MS`: the sampler needs a few
  periods to see a limit, the CPU guard two samples), or + 3 periods
  when the sampler is slow enough (`THREEPM_METRICS_INTERVAL_MS` above
  3.33 s) that this is longer: a trip before that is unstable and
  counts toward `max_restarts`, a later one starts the count over.
  `max_restarts` unstable trips in a row end in `errored`, without a
  respawn — with the defaults (16, 1 s, a 1 s sampler) an app over its
  limit from the moment it is `online` is `errored` after 16 trips, one
  that runs more than ~11 s between trips never is. Known limit: under
  `limit_scope = "tree"` the uptime is the leader's, so a wrapper whose
  busy (or growing) child appears late — past that line minus the two
  periods the guard needs to see it — trips as a healthy run, and such
  an app may cycle without ever reaching `errored`. A user `restart` / `stop` / `delete` during
  the guard's stop, a watch or cron restart start a new series
  (unstable count and backoff at 0, immediate respawn); `3pm reset`
  clears the counters. A guard's stop in progress, or its pending
  respawn, crosses an in-place `3pm update`. `max_restarts = 0` means
  "never respawn on its own", as for a crash: the first trip ends
  `errored`, stable or not.
- **`restart_policy` and `stop_exit_codes` do not apply.** They judge
  the exits an app makes on its own; the exit after a trip is its
  answer to the guard's `kill_signal`. A guard restart is the daemon's
  own recycle, like a watch or cron restart — and like pm2, whose
  `max_memory_restart` reloads the app whatever `autorestart` says — so
  an app under `restart_policy = "never"` (or `autorestart = false`)
  that crosses its limit is restarted. To have it stop instead, leave
  the guard off; `cgroup_enforce` then caps it without a restart.
- **Units.** `max_memory_restart` is RSS in bytes (`K`/`M`/`G` are
  1024-based), summed over the tree — an upper bound when members share
  pages. `max_cpu_percent` is percent of one core, summed over the tree
  — `100` is one core saturated, `400` is four; it may exceed 100.

Validation: `max_cpu_percent` must be a finite number > 0. `3pm start
--max-cpu-percent 0` and `max_cpu_percent = -1` in a file are rejected
before anything reaches the daemon; the daemon re-validates decoded specs
so a hand-edited `dump.json` cannot smuggle one in.

## 2. Hard limits: `cgroup_enforce` (Linux, cgroup v2)

With `cgroup_enforce = true` every incarnation runs, from before its
`exec`, in a leaf of its own:

```
/sys/fs/cgroup/3pm/<name>-<id>-<instance>-<lineage>-g<gen>/
  cpu.max        "<quota> 100000"   quota = max_cpu_percent % of 100 ms   (only if set)
  memory.max     "<bytes>"          = max_memory_restart                   (only if set)
  cgroup.procs   the child (written by itself before exec), then all it forks
```

`<lineage>` is the daemon's `THREEPM_BOOT` (16 hex digits, drawn at
every cold start, kept by an in-place `3pm update`) and `<gen>` the
incarnation's: a leaf is never shared, neither with the next
incarnation nor with a daemon started cold after a crash, which numbers
its rows from 0 again. `<name>` is sanitized.

The kernel then throttles the cgroup at the CPU quota and OOM-kills it
above the memory cap. The leaf's `cgroup.procs` is also the exact
membership of the app's process tree for the sampler: a `setsid`
escapee stays in the cgroup — a member, marked `detached` since the
group's signals will not reach it. The soft guards keep running on
top: with both, memory usually ends in an OOM kill (exit signal 9, seen
in `3pm show`) before the sampler sees the RSS cross the line, and the
restart policy handles it like any crash — except for a tree that
shares many pages (Electron): the kernel counts them once, the
sampler's RSS sum counts them per member, so the soft guard may fire
first. `limit_scope = "leader"` leaves the kernel in charge of the tree:
the guard then judges the spawned pid alone while `memory.max` bounds
the whole cgroup.

Order of operations (`supervisor::lifecycle::spawn_child`, `cgroup.rs`,
`spawner_cgroup.rs`):

0. Open the log files (every spawn, enforce or not): one that cannot be
   opened fails the spawn, `last error` naming it, before any process
   or leaf exists.
1. Resolve the hierarchy **before** spawning. Not Linux, or no
   `cgroup.controllers` at `/sys/fs/cgroup` → the spawn fails with
   `cgroup_enforce is only supported on Linux` /
   `needs cgroup v2 mounted at /sys/fs/cgroup`. No process was started;
   the row backs off and retries per its restart policy and, once the
   budget is spent, sits in `errored` with the reason under `last error`
   in `3pm show`.
2. Prepare the leaf (`CgroupFs::prepare`): create `/sys/fs/cgroup/3pm`,
   enable the `cpu` / `memory` controllers for its children as needed,
   create the leaf — one that already exists is refused: it is somebody
   else's — and write the limits. Any failure fails the spawn, nothing
   started; a leaf whose limits cannot be written is removed at once.
3. Spawn. The daemon opens the leaf's `cgroup.procs`; the child writes
   `0` (itself) into it between `fork` and `exec`, so the program and
   everything it forks run inside from their first instruction. A join
   that fails in the child fails the spawn with `cgroup_enforce: cannot
   join <leaf>/cgroup.procs: …` — never blamed on the program — and
   counts as a spawn failure. An enforce spec never runs unenforced.
4. A spawn that failed after step 2 removes its leaf (one `rmdir`,
   `CgroupFs::discard`).
5. When the incarnation is over — exit, `delete`, death during the
   handoff window — `CgroupFs::cleanup` runs on a blocking thread:
   everything left in the leaf is killed (`cgroup.kill`, Linux ≥ 5.14;
   otherwise SIGKILL pid by pid from `cgroup.procs`, nested cgroups
   included, never pid 0, 1 or the daemon — with the leaf frozen by
   `cgroup.freeze`, Linux ≥ 5.2, from before the read to after the last
   kill, so that no member can exit and its pid go to another process in
   between; a frozen task still dies of SIGKILL; thawed after, waited
   for at most ≈ 45 ms to report `frozen 1`), the cgroups the app made
   below its leaf are removed deepest first (directories only, never
   through a symlink, 8 levels and 1 024 of them at most), then the leaf
   (up to 10 attempts 20 ms apart, ≈ 0.2 s; one still busy is left to
   a later cleanup), then the **empty** leaves of the same instance and
   lineage with a **lower** gen go too. A newer leaf (the next
   incarnation's, made while this runs), another instance's, another
   lineage's or a leaf named before lineages is never swept.

Known gap: the empty leaves a crashed daemon left (another lineage) are
not cleaned; `rmdir` them by hand.

`cgroup_enforce` also changes what the end of an incarnation reaches:
without it, the process group is SIGKILLed at the exit and a member
that left the group (`setsid`) survives; with it, the whole leaf is
killed, those members included.

### Permissions

Writing under `/sys/fs/cgroup` needs either root or a delegated subtree.
The error message says which file was refused:

```
cgroup_enforce: /sys/fs/cgroup/3pm: permission denied — run the daemon as root or delegate /sys/fs/cgroup/3pm to this user
```

Options, from simplest to cleanest:

```console
# a) daemon as root (3pm startup systemd installs it as the given user — use root)
sudo 3pm start ecosystem.toml

# b) one-off delegation to the daemon's user (lost at reboot)
sudo mkdir -p /sys/fs/cgroup/3pm
sudo chown -R "$USER" /sys/fs/cgroup/3pm
echo '+cpu +memory' | sudo tee /sys/fs/cgroup/cgroup.subtree_control

# c) systemd: run the daemon in a delegated slice (survives reboot)
#    in the unit written by `3pm startup systemd`, add:
#    [Service]
#    Delegate=cpu memory
```

Under (c) the daemon's own cgroup is delegated, not `/sys/fs/cgroup`; 3pm
still uses the fixed `/sys/fs/cgroup/3pm` path, so pair it with (b)'s
`chown` on that directory or run as root. Containers: the container's
cgroup namespace must expose a writable `/sys/fs/cgroup`
(`--cgroupns=host --privileged`, or a delegated `--cgroup-parent`).

### Checking it works

```console
$ cat /proc/$(3pm show api --json | jq .pid)/cgroup
0::/3pm/api-1-0-3f9c2a17e4b80d51-g1
$ cat /sys/fs/cgroup/3pm/api-1-0-3f9c2a17e4b80d51-g1/cpu.max
50000 100000
$ cat /sys/fs/cgroup/3pm/api-1-0-3f9c2a17e4b80d51-g1/cpu.stat | grep throttled
nr_throttled 1432
throttled_usec 71283910
$ cat /sys/fs/cgroup/3pm/api-1-0-3f9c2a17e4b80d51-g1/memory.events
oom_kill 2
```

`3pm doctor`-style prerequisite checks do not exist yet: the spawn error
is the diagnostic.

## 3. `deny_gpu`

Sets, after the app's own `env` (so the deny wins over a user value):

| Variable | Value | Honoured by |
|---|---|---|
| `CUDA_VISIBLE_DEVICES` | `""` | CUDA runtime, PyTorch, TensorFlow, JAX |
| `NVIDIA_VISIBLE_DEVICES` | `none` | NVIDIA container toolkit |
| `HIP_VISIBLE_DEVICES` | `""` | ROCm / HIP |
| `ROCR_VISIBLE_DEVICES` | `""` | ROCm runtime |

Best effort by construction: a process that clears these variables, or
opens `/dev/nvidia*` / `/dev/kfd` directly, still reaches the hardware. A
real wall is a device cgroup (`devices.deny` / BPF device programs), which
3pm does not manage. Works on every platform since it is only environment.

## 4. Choosing between soft and hard

| Goal | Use |
|---|---|
| Recycle a leaking service before it swaps | `max_memory_restart` |
| Recycle an app stuck in a busy loop | `max_cpu_percent` (steady-state sized) |
| Keep a batch job from starving neighbours, no restart | `max_cpu_percent` + `cgroup_enforce` — set the soft guard above the cap: the kernel throttles at the cap, so the sampler never sees it crossed |
| Guarantee an app can never exceed N bytes | `max_memory_restart` + `cgroup_enforce` |
| Let the children grow (a build, a browser) without recycling the app | `limit_scope = "leader"` — the guards judge the spawned pid alone |
| Share a GPU box with CPU-only workers | `deny_gpu` on the workers |
| Same file on Linux servers and macOS laptops | `cgroup_enforce = true` in the base, `false` in a `[apps.profiles.dev]` overlay |

Note the throttle case: `cpu.max` caps usage *at* `max_cpu_percent`, and
the guard fires only *above* it, so with `cgroup_enforce` a CPU-bound app
is throttled rather than restarted. The memory pair does not have that
property — an OOM kill is a crash, and a leak that hits the cap restarts
either way.

## 5. Where it lives

- Spec fields, validation, TOML/CLI mapping: `threepm-core/src/spec.rs`
  (`max_cpu_percent`, `limit_scope`, `cgroup_enforce`, `deny_gpu`),
  overlay rule in `profile.rs` (scalar: the profile's value wins).
- Decision: `threepm-core/src/policy/guards.rs` — `MemorySampled` /
  `CpuSampled` carry both figures, `scoped` picks by `limit_scope`,
  `on_resource_sampled` is shared by the two guards, `after_guard_stop`
  respawns through `respawn_or_errored` (`policy.rs`, the crash path),
  `settle_margin_ms` is the stability margin (`GUARD_SETTLE_MS` or
  `GUARD_SETTLE_PERIODS` sampler periods, whichever is longer; the
  period rides each sample, `sample_period_ms`).
- Membership and totals: `threepm-daemon/src/metrics/tree.rs` (pure),
  `metrics/table.rs` (the sweep), `supervisor/trees.rs` (the record).
- Enforcement: `threepm-daemon/src/cgroup.rs` (`CgroupFs::prepare` /
  `discard` / `cleanup`), `spawner_cgroup.rs` (the child's join before
  `exec`), hooked from `supervisor/lifecycle.rs::spawn_child`, released
  (`release_cgroup`, on a blocking thread) on `ChildExited` (`actor.rs`)
  and `remove_process`.
- GPU env: `threepm-daemon/src/spawner.rs::GPU_DENY_ENV`.
- Tests: fake-hierarchy unit tests in `cgroup_tests.rs` (a leaf per
  incarnation, the older-leaves sweep, a leaf whose limit cannot be
  written removed) and `cgroup_kill_tests.rs` (`cgroup.kill`, the
  frozen SIGKILL fallback, the cgroups below a leaf, symlinks and the
  depth bound); the join before `exec` in `spawner_tests.rs`
  (`a_cgroup_leaf_is_joined_before_the_program_runs`,
  `a_join_that_fails_in_the_child_is_named_as_the_cgroups`); policy
  tables in `policy/guard_tests.rs`
  (`limit_scope_picks_the_tree_sum_or_the_leader_alone`,
  `a_guard_tripping_on_every_incarnation_ends_errored_with_the_defaults`,
  `a_trip_is_judged_on_its_uptime_at_the_trip`,
  `deliberate_restarts_never_count`,
  `no_restart_budget_errors_on_the_first_trip_as_on_the_first_crash`,
  `a_guard_restarts_whatever_the_restart_policy_and_the_stop_exit_codes`);
  component tests `cpu_exceeded_sample_triggers_restart`,
  `memory_guard_judges_the_tree_sum_unless_the_scope_is_the_leader`,
  `cgroup_enforce_off_linux_fails_the_spawn_before_it_happens`,
  `a_log_file_that_cannot_be_opened_fails_the_spawn_before_anything_starts`
  (`supervisor/tests/spawn.rs`); e2e
  `max_cpu_percent_cycles_the_process`, `deny_gpu_hides_devices_via_env`,
  `cgroup_enforce_off_linux_errors_the_row_with_a_reason`.

</document>

<document url="https://3pm.edgecr4ft.net/docs/RUNTIME.md" title="Containers: 3pm runtime">

# `3pm runtime` cheatsheet — containers, PID 1, auto-exit

One foreground process owns the apps, streams their output to the
container's stdout/stderr, drains on SIGTERM, and **exits when there is
nothing left to supervise** so the orchestrator's restart policy is the
outer loop. Same idea as `pm2-runtime`; unlike `oxmgr runtime` the apps
live in a real daemon, so `3pm ls | logs | events | restart` work from
`docker exec` while it runs. Long-form rationale: `FEATURES.md` →
*Containers*.

## Invocations

```console
$ 3pm runtime ecosystem.toml                     # every app in the file
$ 3pm runtime ecosystem.toml --env production    # profile, same as `start`
$ 3pm runtime ecosystem.toml --only api,worker   # subset
$ 3pm runtime server.js --name api -i 4          # single script, any `start` flag
$ 3pm runtime server.js -- --port 3000           # app args after `--`

$ 3pm runtime app.py --no-auto-exit              # stay up after every app is down
$ 3pm runtime app.py --delay 5                   # wait 5 s before starting
$ 3pm runtime app.py --log-files                 # also write ~/.3pm/logs files
$ 3pm runtime app.py --raw                       # bare lines (pm2-runtime default)
$ 3pm runtime app.py --format                    # logfmt: ts= app= id= type= message=
$ 3pm runtime app.py --timestamp                 # "2026-08-28 10:27:39 0|app | line"
$ 3pm runtime app.py --timestamp=%H:%M:%S        # custom strftime
$ 3pm runtime app.py --json                      # NDJSON wire events (log_*, process, daemon_shutdown)
```

Every `3pm start` flag is accepted (`--name`, `-i`, `--cwd`, `--env`,
`--max-memory-restart`, `--restart-policy`, `--watch`, …); `3pm runtime
--help` is the authoritative list.

## Exit codes

| Code | When | What the orchestrator should do |
|---|---|---|
| `0` | SIGTERM / SIGINT → every app drained through its policy | normal stop |
| `2` | nothing left to supervise: zero apps selected, or every app is `stopped` / `errored` — a crash loop that exhausted `max_restarts` counts | restart the container (`restart: on-failure`) |
| `1` | the daemon died without announcing, or a start failed | restart the container; read the log |
| `130` | a second SIGTERM/SIGINT forced the daemon down (128 + SIGINT) | — |

`--no-auto-exit` turns `2` off: the daemon stays up with nothing running,
`docker exec … 3pm start` can add apps later.

## What you see

```
[3PM] 3pm runtime: daemon pid 7, home /root/.3pm
[3PM] App [api] launched
0|api | listening on :3000              ← stdout lines, prefixed id|name
0|api | warn: slow query                ← stderr lines go to stderr, red on a tty
[3PM] App [api] exit code=1 signal=SIGSEGV
[3PM] App [api] errored
[3PM] nothing left to supervise: stopping the daemon (exit 2)
[3PM] daemon shutting down (kill)
```

Daemon `tracing` lines (`INFO threepm_daemon::…`) share stderr;
`THREEPM_LOG=warn` quiets them. `[3PM]` chrome goes to stdout in text
mode, stderr in `--json` mode (stdout stays pure NDJSON).

## Dockerfile shape

```dockerfile
FROM node:22-slim
COPY --from=build /src/target/release/3pm /usr/local/bin/3pm   # see examples/Dockerfile
WORKDIR /app
COPY . .
ENV THREEPM_LOG=warn
CMD ["3pm", "runtime", "ecosystem.toml", "--env", "production"]
```

```console
$ docker run --rm -p 3000:3000 img                # streams
$ docker exec ctr 3pm ls                          # live table
$ docker exec ctr 3pm logs api --lines 50         # history needs --log-files
$ docker exec ctr 3pm events -f 'process:*'       # lifecycle stream
$ docker stop ctr; echo $?                        # SIGTERM → 0
```

```yaml
# compose / k8s
restart: on-failure          # exit 2 = crash-looped app → let the platform restart the pod
stop_grace_period: 20s       # ≥ the longest kill_timeout in the file
```

## Rules of thumb

- **Exec form only** (`CMD ["3pm", "runtime", …]`): a shell in front
  would be PID 1 and swallow SIGTERM.
- Log **files** are off by default (`out_file`/`error_file` = `NULL`) —
  stdout/stderr is what `docker logs` collects. `--log-files`, or an
  explicit `-o`/`-e`/`-l` (or the same fields in the file), turns them on
  for that stream. Avoid `--disable-logs` here: it also drops the bus
  stream, so nothing reaches stdout.
- `runtime` **must own the daemon**: if one already runs for
  `THREEPM_HOME` it exits `1` with "already owns this home". Use a
  separate `THREEPM_HOME` or `3pm kill` first.
- A **hosted home** is never one of its own: pointed at a home a root
  daemon hosts for a user (`[users] allow`, marked by a `hosted` file),
  `runtime` exits `1` with "is hosted by the root 3pm daemon: `3pm
  runtime` cannot own it" — that daemon owns the home, the apps and the
  exit code. Give the container its own `THREEPM_HOME`.
- `errored` is down on purpose: 3pm already gave the app `max_restarts`
  tries; the container restart is the next escalation.
- `WaitingRestart`, `Launching`, `Stopping` are *not* down — a crash
  that still has retries left never triggers exit `2`.
- Signals: the first SIGTERM/SIGINT is a graceful `3pm kill` (each app
  gets its `kill_signal`, then SIGKILL after `kill_timeout`); a second
  one SIGKILLs the daemon (`130`).
- Orphans re-parented to PID 1 are reaped (`waitpid(-1)`), so a
  double-forking app never leaves zombies.
- No `save`/`resurrect` in this mode; the ecosystem file *is* the state.
- An app with **`tty = true`** needs a pseudo-terminal: `/dev/ptmx` and
  devpts on `/dev/pts` (Docker and Kubernetes mount them by default; a
  minimal sandbox may not). Without them the spawn fails with `tty:
  cannot open a terminal for the app (is devpts mounted on /dev/pts?)`
  in `last_error` — the app never falls back to pipes. Its stripped
  output reaches stdout like any app's; `docker exec -it ctr 3pm attach
  <app>` connects to it (`Ctrl-Z` detaches).
- A `tty` app **dies with a daemon that crashes**: the kernel hangs up
  its terminal (`SIGHUP`) when the last master closes. Here that is the
  container's end anyway; on a host, `3pm update` keeps it running, a
  crash does not.

## Where it comes from

`crates/threepm-cli/src/commands/runtime.rs` (+ `runtime/reaper.rs`),
`threepm_protocol::spawn_foreground` / `SpawnMode::Foreground`,
`3pm __daemon --foreground` (`threepm-daemon/src/bootstrap.rs`).
E2E scenarios: `crates/threepm-cli/tests/e2e.rs`, `runtime_*`.

</document>

<document url="https://3pm.edgecr4ft.net/docs/SERVE.md" title="Static server: 3pm serve">

# `3pm serve` cheatsheet

pm2's `serve`, as a bundled module: a directory (or one HTML page) served
over HTTP by a 3pm-managed app. Nothing to install — `3pm serve` works
out of the box (`3pm module list` shows it). Full contract of modules:
[modules.md](modules.md).

## Invocations

| Command | What happens |
|---|---|
| `3pm serve` | Serves `./` on 8080 (or the next free port above it), prints the links and a QR code, returns |
| `3pm serve ./dist` | Same, for a directory |
| `3pm serve ./dist 9000` | Explicit port; busy → error, nothing started |
| `3pm serve ./dist 0` | Kernel-chosen port (shown in the links) |
| `3pm serve page.html` | Serves the page's directory with the page as homepage, opens it in the browser, **stays attached: Ctrl-C stops the server** |
| `3pm serve page.html --detach` | Same, but returns and leaves the server running |
| `3pm serve ./dist --attach` | Directory with the attached behavior (Ctrl-C stops it) |
| `3pm serve ./dist` (again) | Restarts the existing site with the new options (same name or same port) |
| `3pm --json serve ./dist 0` | `{"affected":[ProcessRecord…]}` on stdout, no links, no QR |

The app is named `static-page-server-<port>` (`--name` overrides) and is
**online only once the socket is bound** (`wait_ready`).

## Flags

| Flag | Default | Effect |
|---|---|---|
| `[PATH]` | `.` | Directory, or a single `.html`/`.htm` file |
| `[PORT]` / `--port N` | 8080 or next free | `0` = kernel-chosen |
| `--host ADDR` | `0.0.0.0` | Bind address; a loopback host prints the local link only |
| `--spa` | off | Unknown paths → homepage (pm2's two-step fallback: same basename at the root, then the homepage) |
| `--ftp` | off | Directory listing (`index.html` wins, dirs first, dotfiles hidden, `/dir` → `/dir/`) |
| `--homepage FILE` | `/index.html` | Page served for `/` (not allowed with a file argument: the file is the homepage) |
| `--basic-auth-username U --basic-auth-password P` | none | HTTP Basic auth on every request (401 + `WWW-Authenticate` otherwise) |
| `--name NAME` | `static-page-server-<port>` | App name |
| `--namespace NS` | none | App namespace |
| `--attach` / `--detach` | file: attach, directory: detach | Hold the terminal until Ctrl-C (deletes the app) / return at once |
| `--no-qr` | — | Links only, no QR code (a single page never prints one) |
| `--json` | — | Machine output (also `THREEPM_JSON=1`, set by `3pm --json serve`) |

## What you get on the wire

| Situation | Response |
|---|---|
| File | `200`, `Content-Type` from the extension (`text/*; charset=utf-8`), `Content-Length`, `Access-Control-Allow-Origin: *`, body streamed |
| `HEAD` | Same headers, empty body |
| Other methods | `405`, `Allow: GET, HEAD` |
| Missing | `404` — the site's own `404.html` when it exists |
| `..` anywhere in the path | `403` |
| Symlink resolving outside the served directory (file, directory, or anything reached through it) | `403`; never listed by `--ftp`; skipped when it would only be a fallback (`404.html`, SPA homepage). A symlink staying inside the directory is served normally. Enforced at open time: every file is resolved component by component from a descriptor of the served directory (`openat` + `O_NOFOLLOW`, links read with `readlinkat`), so a link swapped between the check and the read is still refused |
| Malformed percent-escape (`/%`, `/%e0%a4`), NUL byte | `400` (GHSA-9prm-75wm-xpg6: never a crash) |
| Directory without slash in `--ftp` mode | `301` to `…/` |

## Managing a served site

```console
$ 3pm ls                                  # under "Modules", below the apps started by hand
$ 3pm stop static-page-server-8080        # name it: `stop|restart|delete all` skip module apps
$ 3pm logs static-page-server-8080        # "Exposing <dir> directory on <host>:<port>" at start
$ 3pm delete static-page-server-8080
$ 3pm uninstall serve                     # removes every served site with the module
```

A served site survives `3pm update` like any app; for a reboot, `3pm
save` then `3pm startup`, as for any app. Either way the module's
binary must still exist (a site of the bundled module is dropped from
`resurrect` once the bundle is gone).

## pm2 ecosystem files

Files written for pm2 keep working with the module binary as the script
and `PM2_SERVE_*` env (`THREEPM_SERVE_*` wins when both are set):

```toml
[[apps]]
name = "legacy-site"
script = "/usr/lib/3pm/modules/serve/bin/3pm-serve"   # or ~/.3pm/bin/modules/serve/bin/3pm-serve
args = ["__run"]
cwd = "/srv/www"
wait_ready = true

[apps.env]
PM2_SERVE_PATH = "/srv/www"
PM2_SERVE_PORT = "8080"
PM2_SERVE_SPA = "true"
```

| Variable | Default | Also accepted |
|---|---|---|
| `THREEPM_SERVE_PATH` | `.` | `PM2_SERVE_PATH` |
| `THREEPM_SERVE_PORT` | `8080` | `PM2_SERVE_PORT` |
| `THREEPM_SERVE_HOST` | `0.0.0.0` | `PM2_SERVE_HOST` |
| `THREEPM_SERVE_SPA` / `_FTP` | `false` | `PM2_SERVE_SPA` / `PM2_SERVE_FTP` (`true`, `1`, `yes`, `on`) |
| `THREEPM_SERVE_HOMEPAGE` | `/index.html` | `PM2_SERVE_HOMEPAGE` |
| `THREEPM_SERVE_BASIC_AUTH_USERNAME` / `_PASSWORD` | none (both or neither) | `PM2_SERVE_BASIC_AUTH_*` |

Not carried over from pm2: `--monitor` (the PM2.io browser snippet).

## Rules of thumb

- Sharing with a phone: bind on every interface (the default) and scan
  the QR code of the `Network:` link; `--host 127.0.0.1` keeps it local.
- Looking at one generated report: `3pm serve report.html`, Ctrl-C when
  done — nothing is left behind.
- Long-lived site on a server: `3pm serve /srv/www 8080 --name www`,
  then `3pm save` and `3pm startup` so it comes back after a reboot.

</document>

<document url="https://3pm.edgecr4ft.net/docs/modules.md" title="Modules">

# 3pm modules

Optional features ship as **modules**: directories with a `module.toml`
and executables, bundled with the 3pm install (tarball, `install.sh`,
deb/rpm) and enabled per machine with `3pm install <name>`. A bundled
module is a copy of what the release already brought; a module 3pm does
not bundle (`automonit`) is downloaded from the download site, the same
one `autoupdate` reads. Nothing about a module lives in the 3pm binary:
its dependencies stay in its own `Cargo.lock`.

The mental model is pm2's `pm2 install <module>`, minus npm.

## Using modules

```console
$ 3pm module list                 # what is installed / bundled with this 3pm
$ 3pm serve ./dist 8080 --spa     # a bundled command runs without install
$ 3pm install serve               # copy it into ~/.3pm/modules (pins a version)
$ 3pm install ./my-module         # any directory holding a module.toml
$ 3pm install automonit           # not bundled: from the download site
$ 3pm install automonit@0.42.0    # a pinned version (always the site)
$ 3pm uninstall serve
```

- `3pm <module> …` runs the module's `command` — the installed copy if
  there is one, else the bundled one. When the bundle is newer than the
  installed copy, a one-line hint says to reinstall.
- `3pm install <name>` copies the module into `~/.3pm/modules/<name>/`
  (a staged copy swapped in atomically), starts its `[[apps]]` and
  **saves the process list**, so they come back after a reboot without
  a separate `3pm save`. Bare `3pm install` lists. On a module already
  installed the files are swapped and the module's rows are restarted
  **in place** — same row, same name, a fresh `spec.module.version` —
  never deleted and recreated: the manifest apps on their new spec
  (one the release added is started, one it removed is deleted), the
  apps the module started itself (`3pm serve` sites) and the apps
  started by hand from its files as they are.
- `3pm update` does the same after the daemon, for every installed
  module whose bundled version changed **and** for the apps running
  straight from the bundle without an install (the agent after
  `3pm link`): a row whose stamp differs from the `module.toml` next to
  its script is restarted on the new files and re-stamped; one already
  at that version is left alone, so a second `update` touches nothing.
  The check is per row (an installed copy whose files are current but
  whose rows are behind them, after a refresh that failed midway, is
  refreshed too), never a bare comparison of versions.
  `--skip-modules` skips the step; a module app whose directory vanished
  is left as is, with a notice.
- `3pm uninstall <name>` deletes the module's apps — and any app whose
  script runs from the module's files, such as sites started with
  `3pm serve` — then removes the directory and saves. On a module the
  release marks `auto` it is also the **refusal**: the name goes into
  `<home>/modules.off` and nothing puts it back until `3pm install
  <name>` asks for it again.
- **A module the release marks `auto = true` installs itself.** It is
  part of 3pm the way a built-in command is (`autoupdate`, so that any
  node can be updated from the dashboard), so 3pm does what `3pm
  install` would have done — copy, start, save — without being asked,
  at the two moments it has a reason to look: a daemon this CLI just
  spawned (the first command on a fresh machine, and every boot without
  a startup service) and after a `3pm update` (how a host already
  running learns that a release added one). Only the **bundle**'s
  manifest is read for it, only a module this host has neither
  installed nor refused is touched, and a module that will not install
  is a warning, never a failed command. A 3pm that predates the key
  ignores it.
- **The configuration can switch an `auto` module off**, the same on
  every host it is provisioned to: `[modules] off = ["lan"]` in
  `<home>/config.toml`. A module named there is never auto-installed,
  and one already installed is uninstalled at the next look (a daemon
  this CLI spawns, `3pm update`): its apps stopped and deleted, its copy
  removed, the list saved — as `3pm uninstall` does, but nothing is
  written to `modules.off`: take the name out of the file and the next
  look installs it again. `3pm install <name>` still works, until that
  next look. The daemon checks the section with the rest of the file
  (an unknown key is named in its log).
- Module apps appear in a second **Modules** table in `3pm ls` and are
  ordinary targets by name (`3pm stop serve`, `3pm logs <module>` by
  namespace), but `stop|restart|delete all`, `delete --flush all`,
  `apply --prune-all` and `restart --update-env` leave them alone:
  modules are managed by install/uninstall. A dumped module app whose
  directory is gone is dropped on `resurrect`.
- Unknown subcommands get a did-you-mean over built-ins and modules.

### Modules from the download site

`3pm install <arg>` resolves, first hit wins: a path holding a
`module.toml`; `<name>@<version>` or `<name>@latest`, which always
means the site; a bundled name; an installed name (its apps restart);
then the site. The site is `$THREEPM_CDN_URL`, `https://3pm.edgecr4ft.net`
by default, laid out by `distribution/scripts/publish-automonit.sh`:

```text
<site>/modules/<name>/latest/version.json          the release to take
<site>/modules/<name>/v<version>/version.json      a pinned one
<site>/modules/<name>/v<version>/<name>-<os>-<arch>.tar.gz
```

3pm reads the index, takes the asset of its platform (`os` `linux` or
`macos`, `arch` `x86_64` or `arm64`), downloads it with `curl` (else
`wget`), checks its sha256, unpacks it into
`<home>/modules/<name>.site.tmp/` and installs that directory exactly as
`3pm install ./dir` would: copy, start, save. The installed copy
carries `.3pm-site.json` (`site`, `version`, `pinned`); that marker is
how `3pm update` knows the module came from the site. On each update an
unpinned site module is checked against `latest/version.json` and, when
a newer release is out, fetched and reinstalled in place like a module
whose bundle moved on; a site out of reach is a warning, never a failed
update. `<name>@<version>` pins (the update leaves it there),
`<name>@latest` unpins, and installing from a directory or the bundle
drops the marker.

What is refused, before anything is installed:

- a site other than `https://` (a `file://` mirror and plain `http` on
  the loopback are accepted, for tests and local mirrors);
- a name outside the module-name grammar, a version outside
  `[0-9][A-Za-z0-9.+-]*` (no `/`, no `..`, no leading `-`);
- an index for another module or another version than asked;
- an archive URL other than `<site>/modules/<name>/v<version>/<file>`:
  only the file name of the index's `url` is used;
- an archive larger than its announced `size` (plus 64 KiB), slower
  than 5 minutes, or whose sha256 differs — the hash is checked before
  the archive is opened;
- an entry with an absolute path or a `..`, outside `<name>/`, a link
  leading out of it, a device or a fifo, more than 10 000 entries or
  2 GiB unpacked; a `module.toml` missing or naming another module.

An unknown name says what was looked for and where, with the local
did-you-mean: `no module named `automonitt` (did you mean `automonit`?)
— available: …; not on https://3pm.edgecr4ft.net either (cannot read
…/modules/automonitt/latest/version.json: … 404 …)`. No build for this
platform names the published ones. The shell installer
(`<site>/modules/<name>/install.sh`) stays for a 3pm up to 0.42.0.

Where the bundle is looked up, first hit wins: `$THREEPM_MODULES_DIR`,
`<dir of the 3pm binary>/modules` (tarball / `install.sh` → `~/.3pm/bin/modules`),
`<dir>/../lib/3pm/modules` (deb/rpm → `/usr/lib/3pm/modules`).

## Writing a module

```
my-module/
├── module.toml
└── bin/
    ├── my-module        # `command`: what `3pm my-module …` runs
    └── worker           # scripts of [[apps]]
```

```toml
[module]
name = "my-module"            # letters, digits, '-', '_'; not a built-in command
version = "1.0.0"
description = "one line for `3pm module list`"
command = "bin/my-module"     # optional, relative, inside the module, executable
auto = false                  # optional: true installs it without being asked,
                              # until `3pm uninstall` refuses it (bundle only)

[defaults]                    # optional, ecosystem syntax
max_restarts = 5

[[apps]]                      # optional: started by `3pm install`
name = "worker"
script = "bin/worker"         # relative to the module directory
```

Rules the loader enforces: `[[apps]]` use the ecosystem vocabulary
(`docs/ECOSYSTEM.md`) with paths relative to the module directory;
scripts without an interpreter must be executable; apps are named
`<module>:<app>` (a single app named like the module keeps the bare
name — pm2's rule), get `namespace = <module>` unless set, and carry
`spec.module = {name, version}`; `spec.source` points at the installed
`module.toml`. Unknown fields warn with a did-you-mean.

A `command` receives the caller's arguments after the module name and,
on top of the environment:

| Variable | Value |
|---|---|
| `THREEPM_HOME` | the 3pm home (the daemon is already running when the command starts) |
| `THREEPM_EXE` | path of the `3pm` binary (e.g. to print the table with `3pm ls`) |
| `THREEPM_MODULE_DIR` | the module's directory (installed or bundled) |
| `THREEPM_JSON` | `1` when `3pm --json <module> …` was called (a `--json` *after* the name is the module's own) |
| `THREEPM_PARENT_FD` | the read end of a pipe whose write end only the dispatching `3pm` holds: EOF there means `3pm` died (even by SIGKILL) |

Its exit code is 3pm's, and `SIGINT`, `SIGTERM`, `SIGHUP` and
`SIGQUIT` are forwarded to it (3pm waits for it to finish; a signal 3pm
was started ignoring stays ignored). A `3pm` killed outright forwards
nothing: a module that must not outlive it watches `THREEPM_PARENT_FD`
and stops on its EOF (a Rust module calls
`threepm_protocol::module::parent::watch()` first thing in `main`,
which raises `SIGTERM` at it — its normal way out; `agent` and `serve` do);
on Linux the kernel also sends it `SIGTERM` (`PR_SET_PDEATHSIG`). Any language works; a Rust module depends on
`threepm-protocol` (`client::connect`, `Request::Start` with a
`SpecPatch`-built spec) and, for apps that want `wait_ready`, speaks the
control channel (`hello` then `ready` on `$THREEPM_CTL_SOCK`, see
OV `3pm-docs-archived/architecture` §4.3b). `threepm_protocol::module` is that contract
as code; `serve` and `agent` are built on it, while `autoupdate` and `lan` speak
the control channel through the Rust SDK (`threepm-io`, `io().serve()`)
instead. The toolkit: `is_run_role` (the
`__run` split), `Out` (`--json`/`THREEPM_JSON`, the `[3PM]` chrome —
colored like the CLI's, routed to stderr in JSON mode so stdout stays
one document), `exit_code` (`[3PM][ERROR] …`), `ctl::{Identity,
announce_ready}` (the handshake, plus a callback on `shutdown`),
`affected`/`unexpected` (a daemon reply as the rows, or as the one
sentence every module fails with) and `shutdown_signal` (SIGINT or
SIGTERM, handlers installed at the call). A
module prints only through the pipe-safe `oprintln!`/`oeprintln!`
(`clippy::print_stdout`/`print_stderr` denied, as in the CLI: `3pm serve
… | head` must not panic). Apps a command starts should carry
`spec.module = {name, version}` too (as `3pm serve` does): they then
list under **Modules** in `3pm ls`, below the apps started by hand, and
get the same treatment as installed apps (`all` targets skip them,
uninstall removes them).

In-tree modules are standalone Cargo workspaces under `modules/<name>/`
(a `[workspace]` table, path-deps on `../../crates/threepm-core` and
`threepm-protocol`, own lockfile), built by `just modules` / `just
bundle` and by the release matrix, and packaged as
`modules/<name>/{module.toml,bin/…}` next to the binary. Their e2e
tests use `threepm-testsupport` with `THREEPM_BIN` pointing at a 3pm
binary and the module staged as a bundle (`modules/serve/tests/e2e.rs`).

## Catalogue

| Module | Command | Apps | Notes |
|---|---|---|---|
| `serve` | `3pm serve [dir\|page.html] [port]` ([cheatsheet](SERVE.md)) | none | pm2 `serve`: `--spa`, `--ftp`, basic auth, `--host`, `--homepage`, custom `404.html`; free port picked, local/network links + QR code; a single `.html` file is served as homepage, opened in the browser (no QR code) and held until Ctrl-C (`--detach`/`--attach`); `PM2_SERVE_*` env honored |
| `agent` | `3pm agent link <cluster-key> <token> [server]` (default `wss://fleet.edgecr4ft.net`, `THREEPM_LINK_SERVER` wins) / `link` alone (check, exit 0 if established) / `link stop` / `link start` / `link delete` / `unlink` / `status` (`3pm link …` / `3pm unlink` are built-in sugar over them) | one (`agent`, supervised, `wait_ready`) | Fleet link to a `3pm-backend`: enrolls this host (Ed25519 node key in `<home>/agent`, owner-only), then streams the process table and host metrics over one WebSocket. `THREEPM_LINK_*` env overrides the pasted line; clear text off loopback needs `--insecure`; `--cap`, `--no-env-keys`, `--force`; contract, files and caveats in [backend-agent/](backend-agent/README.md) |
| `autoupdate` | none — the actions are the surface | one (`autoupdate`, supervised, **`auto`: installed without being asked**) | Updates 3pm itself: `3pm trigger autoupdate update [{"version":"0.33.0"}]` downloads `install.sh` from `$THREEPM_CDN_URL` (default `https://3pm.edgecr4ft.net`) and runs it — the daemon then hands off — and `check` compares `latest/VERSION` with the version these files shipped with. The call answers as soon as the updater is started, because the update restarts this very app: the updater `setsid`s out of its process group, holds `<home>/autoupdate.lock` and logs to `<home>/autoupdate.log`. **Every hour** (`THREEPM_AUTOUPDATE_EVERY`: `30m`, `90s`, `off`) the app checks the site by itself and updates when it is behind — once per published version (`<home>/autoupdate.last`), never in a loop |
| `lan` | none — `3pm trigger lan lan.status` | one (`lan`, supervised, **`auto`: installed without being asked**; works on a root daemon on Linux, idle anywhere else) | A virtual Ethernet between the nodes of a bucket, made only once a peer shows up (removed 10 min after the last one left; never over a network already in use on the host); off with `[modules] off = ["lan"]` in `config.toml`: a TAP device (`3pmlan0`, `10.77.0.0/16`, address and MAC derived from the node id) whose frames ride fleet stream sessions (`io.open`) to every other node of the bucket running `lan`, found by itself (`fleet:list`; sessions need `fleet:invoke`). Optional `/etc/3pm/lan.toml` (else `$THREEPM_HOME/lan.toml`): `iface`, `network`, `address`, `allow` (an allow-list; `peers` still read), `broadcast_route`, `rate_kbit` (per peer, default 4000, cut by a latency guard while the peer's round trip is over 1 s), `expect_direct`, `netns` (a namespace `/run/netns/<name>` holding only the lan: macvlan `3pmlan1`, own address and MAC, no default route — for a program that binds to the default route's interface), `netns_address`. The bucket is the trust boundary. Metrics `lan.frames`/`lan.bytes`/`lan.dropped`/`lan.peers`/`lan.rtt`, events `lan.peer_up`/`lan.peer_down`; see FEATURES.md, *A virtual Ethernet between nodes* |

## Not yet

- A config store (`3pm set <module>:<key> <value>` / `3pm get`, pm2's
  `module_conf.json`) — the seam is `SpecPatch` as a partial update.
- A catalogue of the site's modules (`3pm module list` shows what is
  installed and bundled only; the did-you-mean does not know the site).
- `script = "serve"` shorthand in ecosystem files: use the module binary
  as the script with `args = ["__run"]` and `PM2_SERVE_*` env.

</document>

<document url="https://3pm.edgecr4ft.net/docs/metrics.md" title="Metrics reference">

# Metrics Reference

Every metric 3pm collects, where it comes from, and how retrieval works.
Design rationale lives in OV `3pm-docs-archived/architecture` (§8); this is the field-level
reference. The research trail behind the host-metrics design (OTel
`hostmetricsreceiver` survey, Datadog gap analysis) is
`doc/3pm-multi-host-metrics-retrieval.md` in the archive
`~/3pm-rd-doc-2026-09-09.tgz` — it is no longer in any tree.

Three commands expose three scopes:

| Command | Scope | Wire type | Collected |
|---|---|---|---|
| `3pm ls` / `3pm show` | Per managed instance — its whole process tree | `ProcessRecord` | Continuously, on the metrics tick |
| `3pm ls --tree` / `3pm show` | Every process of the tree, one line each | `AppTree` | Same tick, served on request (`process_trees`) |
| `3pm internals` | The 3pm daemon itself | `DaemonInternals` | On request (self-sample rides the tick) |
| `3pm hostmetrics` | The whole host, and 3pm's own footprint on it (`3pmDaemon`) | `HostMetrics` | By the host agent process in cost tiers — cheap gauges every 1s, walks/forks every 10s, the top processes every 5s, NTP every 5min, 3pm's own processes + the daemon's counters (pulled over `rpc.sock`) every 10s — served from its cache (`collected_at_epoch_ms`, `sampled_at_epoch_ms`) |

All three support `--json` (the raw wire struct, one document on stdout).

## Retrieval model

- **The metrics tick** (`threepm-daemon/src/metrics/`): a daemon task
  reads the process table — cpu and memory in one pass — through one
  reused table (Linux: `/proc/<pid>/stat` read directly, one file a
  process, `metrics/procs_linux.rs`; elsewhere a `sysinfo::System`),
  every `THREEPM_METRICS_INTERVAL_MS`
  (default 1000ms, min 50), off the runtime (`spawn_blocking`), and
  attributes every process to the instance it belongs to (**Process
  trees** below); the daemon's own pid rides the same pass. The whole
  table is swept on the first tick and every thirtieth (`metrics/table.rs`,
  `FULL_SWEEP_EVERY`); in between a tick lists the pids alive and
  refreshes only those born or gone since, the leaders, the members
  known and 3pm's own — each pid at most once a tick (a second refresh
  would reset its cpu delta), no stat file held open. The full sweep
  only resyncs a pid that died and was reborn between two listings
  (the table keeps the dead process's start and parent): a member
  reborn there joins its tree at once through its process group when
  the dead process was younger than its leader, otherwise at the next
  full sweep (≤ 30 s) — a reborn double-fork daemon, in its own
  session, always waits, since the environment's answer is remembered
  by pid and start. Every other membership fact (births, deaths,
  groups, environments, a member's name and parent) is live on every
  tick; a stranger's stale parent pid, since reused by a member, never
  pulls it in (a row older than its parent is not its child, and
  nothing is claimed below a row the tick did not read — a member
  reborn within the tick keeps the dead process's start until the next
  one reads it). CPU% is a
  delta between refreshes, so the first sample after a spawn — and a
  member's first tick — legitimately reads 0%; so does a process
  adopted on a partial tick until its last two refreshes are one tick
  apart (its first spans every tick it was left alone, and would read
  as that many ticks of work done in one). Samples are keyed by OS
  pid and gated against the *current* pid in the supervisor, so a
  dead/reused pid is never attributed to a stale row. The same tick
  lists the TCP LISTEN sockets (Linux: netlink `sock_diag`, `/proc/net/tcp*`
  if refused; elsewhere the `listeners` crate) and attributes each port
  to the member that holds it, walking a member's fd table only when it
  is new, every 10 ticks, or when a LISTEN socket appears that no member
  is known to hold — that last walk at most once every 5 ticks, so a
  stranger opening a listener every second costs one walk in five, and
  a member's own new listener shows within 5 s (at once on a quiet
  host). A daemon with nothing running does not
  sweep the table (its own pid is still sampled); `3pm internals` prices
  the last sweep (`last metrics tick`).
- **Reads that can hang, and a sweep that stalls.** Two reads take the
  target's memory-map lock or its exec lock in the kernel and block for
  as long as the target holds it (`D` state: a hung NFS mount, an
  `exec` or a page fault that does not finish; heavy memory pressure):
  Linux's `/proc/<pid>/environ` (the membership probe) and the `io` /
  `smaps_rollup` of Extended Monitoring; macOS's `KERN_PROCARGS2` goes
  the same way. The rest of the tick takes neither lock: the refresh
  reads cpu, memory, parent and name from `stat` and the uid once from
  `status` (Linux; macOS asks sysinfo for cpu, memory and the uid only —
  not its default `io`, `exe` and tasks), a member read this tick is
  named from that `stat` (as its `comm` file shows it), one that was
  not from `comm`, ports from the fd tables; `process_start_time`, called on the
  supervisor, reads `stat` alone (macOS: `proc_pidinfo`
  `PROC_PIDTBSDINFO`). One read is left unguarded: on macOS, sysinfo
  reads every newly seen pid's `KERN_PROCARGS2` on the sweep's thread,
  whatever the refresh asks — there the stall report below is the
  backstop. The policy (`metrics/risky.rs`):
  each risky read runs on a reader thread under a 500 ms deadline
  (`READ_DEADLINE`, `metrics/guarded.rs`); a tick spends at most 1 s
  on them (`TICK_BUDGET`), after which the reads left wait for a later
  tick, nothing held against them — a tick pays at most 1.5 s however
  many members are slow; a read that missed (or panicked) is not tried
  again on that process before a wait doubling from 1 to 60 ticks, and
  an answer clears it; while the reader abandoned on a process is still
  in its read, that process is not read at all (quarantine, daemon-wide:
  a table rebuilt after a sweep panic keeps it), until the read returns
  or the process goes. A read not made is no answer: the process is not
  adopted through its environment this tick and is asked again on the
  next (the environment memo backs off only on an environment actually
  read unreadable), a member keeps its last extended counters (then
  possibly older than one read cycle: the wire carries no age for
  them). With four readers stuck at once, daemon-wide (`MAX_STUCK`,
  `guarded::STUCK`), both
  reads are off until one returns (one `error` in the daemon log, one
  `info` when they resume): a host where everything hangs does not grow
  the daemon a thread per tick. Cpu, RSS, the trees and the listeners
  go on, and so do `max_memory_restart` and `max_cpu_percent`. One
  sweep runs at a time; one that runs past `max(5 × period, 5 s)`
  anyway is reported: an `error` in the daemon log, `3pm internals`
  shows `metrics sweep stalled for …` (`DaemonInternals.metrics_stalled_ms`,
  counted from when that sweep began) — no sample, so no resource
  guard, for that long — and the loop keeps waiting for that same sweep
  (a second one would queue behind the same read); its return is
  logged, the line goes, sampling resumes.
- **On-request snapshot** (`internals`): assembled when asked.
- **The host agent** (`hostmetrics`): a separate process
  (`3pm __host-agent`, supervised by the daemon) collects in three cost
  tiers on independent tickers — fast `THREEPM_HOSTMETRICS_INTERVAL_MS`
  (default 1s: cpu, memory, load, PSI, counters; stateful, the tick is
  the CPU-% window), medium `THREEPM_HOSTMETRICS_MEDIUM_MS` (10s:
  process/socket walks, sensors, GPU tools, `statvfs`, batteries), slow
  `THREEPM_HOSTMETRICS_SLOW_MS` (5min: NTP, identity) — and caches the
  assembled result; the daemon proxies `host_metrics` to that cache and
  reports `err{unavailable}` when the agent is down, never collecting
  inline (OV 3pm-docs-archived/architecture §8.2, `hostagent.md`, `hostagent-tiers.md`
  for the per-field tier).
- **Cumulative counters, not rates.** Where the OS provides monotonic
  counters (disk I/O, network bytes, TCP segments, page faults), 3pm
  reports them as-is. Rates (`r/s`, `%util`, `await`) require two samples
  over time — that derivation belongs to whatever polls the daemon
  periodically (a future fleet layer), not a one-shot CLI.
- **Absent, never faked.** A metric the platform can't provide is `None`
  or empty — never zero. Notable case: recent macOS zeroes `netstat`'s
  TCP counters for unprivileged callers; 3pm detects the all-zero read
  and reports the whole section absent.
- **Wire growth contract.** Every metrics struct is fully
  `#[serde(default)]`: fields can be added one at a time without breaking
  a version-skewed CLI/daemon pair, in either direction.
- **No elevation.** Everything below works as the unprivileged daemon
  user on stock systems.

Env knobs: `THREEPM_METRICS_INTERVAL_MS` (tick; floor 50 ms; a longer
tick widens the resource guards' settle margin to `max(10 s, 3 ×
period)`, see `docs/resources.md`), `THREEPM_PROCESS_TREE`
(`off`, `0` or `false`: sample the spawned pid alone, as before process
trees), `THREEPM_NTP_SERVER` (clock-offset probe server; empty string
disables the probe).

## Per-process metrics (`ProcessRecord` — `3pm ls`, `3pm show`)

| Field | Meaning | Source |
|---|---|---|
| `cpu_percent` | CPU% summed over the instance's process tree, 100 = one core (the spawned pid's own share is the first line of its tree) | the table sweep (Linux `stat`, computed as sysinfo computed it; sysinfo elsewhere), metrics tick |
| `mem_bytes` | RSS summed over the tree — an upper bound when members share pages | same |
| `listen_ports` | TCP LISTEN ports of every member of the tree (cluster workers roll up) | `listeners::get_all()` sweep on the tick; reported only while online, cleared on respawn |
| `pss_bytes` | Extended Monitoring (Linux): the members' PSS summed — shared pages split between their sharers, so not inflated the way the RSS sum is. Present only when every member has one (read or carried), never a partial sum: absent when any member's PSS is unreadable (another account's process under a user daemon) or not yet read (a newcomer over the tick's read budget), and on macOS (its `footprint_bytes` stays per member) and with `THREEPM_PROCESS_TREE=off`; refreshed every 10 s | `smaps_rollup`, see "Extended counters" below |
| `net` | The tree's network traffic over the last 5 s window and since the incarnation was first measured: `rx_bytes_per_s`, `tx_bytes_per_s`, `rx_bytes`, `tx_bytes`, `partial` (a lower bound: notifications lost). Each socket once per tree. Absent before the meter's second window, on a platform with no source, and with `THREEPM_PROCESS_TREE=off`. Every `TreeProc` has its own (every socket it holds, whole) | `metrics/net.rs`, see "Network traffic per app" below |
| `proc_count` | Live processes in the tree, the spawned pid included: `1` alone, `0` when not running (or from a daemon predating trees) | tree membership, metrics tick |
| `tree_rev` | Bumped when the listed membership changes (a pid enters or leaves the first 256 lines, `detached` flips, a member takes another name, the incarnation exits), never on cpu/mem; members beyond the cap do not move it; the next value of one counter per daemon — never back within one image, from 0 again on a daemon started cold; an in-place `3pm update` carries the counter but not the rows' revisions (an adopted row reports 0 until its next tick, and `boot_id` changes) | supervisor |
| `restarts` / `unstable_restarts` | Lifetime restarts / consecutive quick-crash count | policy state machine |
| `uptime_ms` | Current incarnation wall-clock uptime | daemon runtime state |
| `last_error` | Cause of the most recent spawn failure (e.g. `cwd '…' does not exist`) | spawner ENOENT diagnosis; cleared by a successful spawn |
| `status`, `pid`, `exit_code`, `instance`, `created_at_epoch_ms` | Lifecycle bookkeeping | supervisor |

## Process trees (`AppTree` — `3pm ls --tree`, `3pm show`)

An instance's figures are those of its **process tree**: the spawned pid
and every process the daemon attributes to it. `3pm ls` writes `saw +13`
when an app has other processes, `3pm ls --tree` (`-t`) draws them under
the row, `3pm show` lists them all, `--json` carries the tree under a
`tree` key (`Request::ProcessTrees`, `docs/protocol.md`). A daemon
predating trees answers `unknown_request`: the CLI renders without and
says `run \`3pm update\``. On a linked host the tree reaches the fleet
backend too — the shape when it moves, the figures every second — and
each member becomes a series there (`docs/backend-agent/link-protocol.md`
§6c).

### Membership

Each tick, for every running instance, in this order — a process
belongs to the first instance that claims it (`metrics/tree.rs`, pure
and table-tested; the OS lookups are injected):

1. **The cgroup** (Linux, `cgroup_enforce`): the leaf's `cgroup.procs`,
   nested cgroups included — exact, and it replaces the rules below.
2. **Descendants** by parent pid, from the spawned pid.
3. **The leader's process group** (`getpgid`): the child whose parent
   died and was reparented (to pid 1, or to the nearest subreaper —
   `systemd --user`, `tini`). Compared to the leader's *actual* group,
   so a leader that `setsid`s keeps its own, and only when that group
   is the leader's own (led by itself, by a member, or by a pid that is
   gone) — never a live stranger's.
4. **The environment** of a process no earlier rule placed: the
   spawner injects `THREEPM_ID`, `THREEPM_GEN`, `THREEPM_CTL_SOCK` and
   the daemon's lineage `THREEPM_BOOT`, inherited by every descendant,
   so the classic daemon (double fork + `setsid` before the first tick)
   is still tied to its app. The socket path must be this daemon's, so
   two daemons of one user (two homes) never adopt each other's
   children; and the lineage the incarnation's — drawn at every cold
   start, kept by an in-place `3pm update` — so a survivor of an
   earlier daemon of the same home, which carries the same numbers, is
   never adopted either. A malformed `THREEPM_BOOT` makes the process a
   stranger; an incarnation adopted from an image that predates
   lineages is recognised without one until its next spawn. A readable
   answer holds for the process's life (an `exec` keeps the
   environment); an unreadable or empty one is asked again 1, 2, 4… up
   to 60 ticks later (`metrics/probe.rs`).
5. **Memory**: a member stays one while it lives, by `(pid, start
   time)`, under the instance's `(id, gen)` — a new leader that reuses a
   dead one's pid inherits nothing, a restart starts from nothing.
   Looked up before the environment: it only ever confirms an earlier
   tick's claim.

Then every descendant, by parent pid, of a member rules 3–5 placed (the
shell a daemonized worker runs a job through, whatever its own
environment): membership is closed under parent pid. And no rule ever
claims a process that started before the instance's leader: a real
descendant cannot predate its ancestor, whatever a stale parent pid, a
leftover in a cgroup leaf or an inherited environment says. The same
holds link by link: a row older than its parent is not its child, so a
stranger whose parent died, and whose pid a member now reuses, is not
claimed through that stale parent pid (a subreaper member adopts only
its own orphaned descendants, all younger than it). And nothing is
claimed below a row the tick did not read (a stranger's, kept between
full sweeps): it may be a dead process's whose pid a member was just
reborn on, with the dead one's start — old enough for the dead one's
orphans to pass as the member's children. A stranger adopted from a
kept row (rules 3–4) is read the next tick, as a member, and brings its
children then, one level per tick.

Never a thread (Linux tasks are not processes), never a zombie, never a
row without a start time (macOS: another user's process — no parent, no
start, no uid), never 3pm's own daemon or host agent. A member outside the leader's process group is
**detached**, whichever rule placed it: `3pm stop`/`restart` signal the
group and will not reach it. `⚠` in `--tree`, `⚠ detached` in `show`,
`detached: true` in JSON — never on the spawned pid.

What the rules cannot see:

- **Another user's process.** Linux: cpu and RSS are readable, the
  environment is not — ppid and group still tie it. macOS: the row is
  ignored altogether (rule 3 would otherwise attach a `sudo` child with
  0 bytes and a 56-year uptime).
- **macOS platform binaries.** The OS withholds the environment of
  Apple's own binaries (`/bin/sh`, `/bin/bash`, `sleep`, `perl`…) even
  from their owner: a daemonized `bash` is not adopted by rule 4 —
  until it `exec`s a program whose environment reads fine, the
  environment being asked again —, a daemonized `node`, `python` or
  Homebrew binary is.
- **A nested daemon.** An app that runs `3pm start` against a home
  with no daemon spawns one as its own descendant, inside the app's
  process group (the launcher neither `setsid`s nor sets a group): that
  daemon and its apps join the tree and count toward the limits.
- **After an in-place `3pm update`** the memory (rule 5) is rebuilt on
  the first tick from rules 1–4; only a detached member whose
  environment cannot be read is lost — another user's, or on macOS an
  Apple platform binary's. After a cold start (a crash, `3pm update
  --cold`) what an earlier daemon left running is never a member: its
  lineage is not the new daemon's.

### Figures

| Field | Meaning |
|---|---|
| `AppTree.id`, `gen`, `rev` | The instance, its incarnation, and the record's `tree_rev` at the tick |
| `AppTree.cpu_percent`, `mem_bytes` | The totals of the same tick as the lines, omitted members included — a `list` answered from another tick may differ, so a reader sums nothing itself |
| `AppTree.procs[]` | `procs[0]` is the spawned pid; a parent precedes its children, siblings by pid; members whose parent is not a member follow, by pid; empty when the instance has at most one process or is not running |
| `AppTree.omitted` | Members cut by the caps (elided at 0); their figures are still in the totals |
| `TreeProc.pid`, `ppid`, `name` | The kernel's ppid (not necessarily a member); `name` is the one the OS gives the process now, read again every tick (a new name moves `tree_rev`): on Linux its `comm`, 15 bytes at most — the executable's file name after an `exec`, or what the program set since with `prctl(PR_SET_NAME)`, which is how Node's `process.title` shows —; on macOS the executable's file name (it follows an `exec`, never a `process.title`); elsewhere the name first seen, 64 bytes at most, controls and bidi marks blanked, private-use glyphs kept, never the arguments — text the app controls (`process.title`, `PR_SET_NAME`) |
| `TreeProc.cpu_percent`, `mem_bytes` | This member's share: percent of one core (0 on its first tick), RSS |
| `TreeProc.started_at_epoch_ms` | Second resolution; uptime in the table, identity against pid reuse |
| `TreeProc.detached`, `listen_ports` | Elided when false / empty; 4 ports at most, the record's `listen_ports` is the uncapped union |
| `TreeProc.ext` | Extended Monitoring's counters, below; elided when the mode is off, and shed from the reply's tail past its byte budget (`tree::budget_extras`) |

Caps: 256 lines per tree, 2 000 per reply — `min(256, 2000 / n)` per
instance, `n` the instances answered that list members, at least one
line each, so a reply never drops an instance whole and an app alone or
stopped never shrinks another's share — 4 ports per line, 64-byte names. Memory is the **sum of the
members' RSS**, an upper bound when they share pages (Electron helpers
share the framework). With Extended Monitoring on, `pss_bytes` (Linux)
is the honest sum; the cgroup's `memory.current` is not read.

### Extended counters (`TreeProc.ext`)

`[hostmetrics] extended` (on by default, `THREEPM_HOSTMETRICS_EXTENDED`)
has the daemon read, for the **members of its trees only** — never the
rest of the host —, the kernel's own counters (`metrics/ext.rs`). A
member is read on the first tick that finds it, then once every ten
ticks (10 s at the default interval); in between its last value is
carried, present and unchanged. A tick reads at most 256 processes
(`EXT_READS_PER_TICK`), first sights included: a member over the budget
keeps its carried value (none if never read) and is read first on the
next tick — the never-read, then the longest unread. A member that leaves, and a pid reused (another
start time), start over. Every field is absent where the platform or
the permissions do not give it, never zero: no uid is checked, each read
that fails (`EACCES`, `hidepid`, a process gone) leaves its fields out and
the next read tries again — a user daemon reads another account's member
(`[users]`, a `setuid` child) without `io`, fds or PSS. A read slower
than its 500 ms deadline, or over the tick's 1 s budget for such reads,
leaves the member its last values, asked again later with a backoff
(**Retrieval model**, `metrics/risky.rs`).

| Field | Linux (`/proc/<pid>/…`) | macOS (`libc`) |
|---|---|---|
| `minflt` / `majflt` | `stat` | — / `ri_pageins` (pages read in) |
| `ctx_voluntary` / `ctx_involuntary` | `status` | — (the kernel's one combined count is not split) |
| `read_bytes` / `write_bytes` | `io` (storage) | `ri_diskio_bytesread` / `…written` |
| `rchar`, `wchar`, `syscr`, `syscw` | `io` | — |
| `threads` | `stat` | `PROC_PIDTASKINFO` |
| `fds` / `fd_limit` | fd count / `limits` soft `RLIMIT_NOFILE` (absent when unlimited) | `PROC_PIDLISTFDS` / — |
| `pss_bytes` | `smaps_rollup` | — |
| `footprint_bytes` | — | `ri_phys_footprint` |
| `swap_bytes` | `status` `VmSwap` | — |

Cost, measured on `edgecr4ft.net` (release, non-root): 161 µs per
sleeping child, 471 µs per real process (a larger address space makes
`smaps_rollup` longer). The budget bounds a tick's reads at 256, about
120 ms at 471 µs each, on the tick's blocking thread. `THREEPM_PROCESS_TREE=off`
reads nothing.

### Rendering

`3pm ls --tree` compacts (`threepm_protocol::tree::compact`, pure, in
the protocol crate so the fleet backend serves the same picture on
`GET …/nodes/{node}/trees`, `docs/backend-agent/link-protocol.md` §6c): a
chain of single children is one line
(`npm › bash › node › Electron`: figures summed, pid and uptime of the
last link, ports in union), siblings with the same label are one line
(`Electron Helper ×4`: figures summed, no pid, uptime of the oldest),
a detached link cuts its chain and is never grouped, so `⚠` is never
hidden; `--all` unfolds everything. Tree rows are dimmed, take the
table's columns (`id`, `↺`, `status`, `watching` empty) and widen
`name` up to 32 columns of label plus the guides; `… N more` closes a
capped tree. With `--watch` the period defaults to 1 s (the daemon's
tick) and a tree reply that overruns the frame's budget is left out of
that frame, with the host line and the orphan notice (the connection is
reopened for the next one). `3pm show` says `cpu 62.2% (14 processes)`,
`mem 3.0gb (14 processes, RSS sum)` and lists every process,
uncompacted, under ` Processes`. With Extended Monitoring, the `mem` row
reads `3.0gb RSS · 2.2gb PSS (14 processes, sums)` when the record has
`pss_bytes`, and the table gains `pss` (Linux) or `footprint` (macOS),
`threads` and `fds` — each column drawn only when a member has it, `-`
for a member without; the other `ext` counters are `--json` only
(`render::tree::EXT_COLS`, `render::show::rows::mem_text`).

### Switch and cost

`THREEPM_PROCESS_TREE=off` on the daemon restores the pre-tree sampler:
the spawned pid alone is refreshed, `proc_count = 1`, empty trees, the
limits judge it alone, ports attributed by process group. Otherwise one
refresh of the table per tick under `spawn_blocking` — of the whole table
every thirtieth tick, about 12 ms for a thousand processes on the
development Mac, listener scan included; in between, of the pids born
or gone, the leaders, the members and 3pm's own only —; `metrics_tick_ms` (`3pm internals`, `last
metrics tick`) prices it, and the daemon logs `metrics tick slower than
its period` when a tick first overruns, once per streak (the ticker then
delays, cpu deltas stay right). Bench: `bench_members_and_build_on_a_thousand_rows`
(`--release`, `#[ignore]`).

## Application metrics (`Request::AppMetrics` — `3pm show`)

What the app's own processes report on the control channel
(`docs/protocol.md`, "Control channel"), pm2's custom metrics without
the Node-only IPC: any process of the app connects to `ctl.sock`, says
`hello` with its `source` (`cv-engine`, `renderer`…) and sends its full
set of metrics every second, plus events as they happen. Nothing is
injected into the child; the app embeds an SDK (`packages/`) or writes
the NDJSON itself (`examples/metrics.py`). The SDKs — `@3pm/api` (Node,
Electron), `threepm-io` (Python), `threepm_io.hpp` (C++17) — and their
common surface: `docs/app-metrics.md`.

| Kind | Semantics | Stored as |
|---|---|---|
| `gauge` | last value | one series |
| `counter` | cumulative since process start (rates at read time) | one series |
| `histogram` | base-2 exponential buckets (`exphist`), cumulative `count`/`sum`/zero/buckets, quantiles at read time | 2 + one per non-empty bucket (+ zero) |

The daemon keeps the **last value** of every series `(source, name,
labels)` on the instance's runtime — no time series, nothing on disk —
and answers `app_metrics` with **the whole state**, always
(`def`/`val`/`hist` by daemon-assigned `sid`). It cuts no diff: a
reader that wants one — the fleet agent, for the link — holds what it
last sent and compares (`threepm_link::node_state`). The request is `id`
alone, and the answer says nothing about diffing: protocol v2 took out
the five fields that pretended otherwise. What the daemon does keep
is a **revision**, so a reader knows whether asking is worth it: only a
batch that defines or changes something moves `app_metrics_rev` — an
idle registry flushed every second leaves it alone —, and so does a
control connection attaching with a `source` or closing (the sources
are part of what a reader holds). Cost of a whole answer at the series
cap: 0.28 ms and 120 KB for 950 series (release, M-series), 5.5 ms a
second for the twenty instances a node manages at most. `ProcessRecord`
carries only `app_series` (live series, buckets included),
`app_metrics_rev`, `app_metrics_rejected` and `app_events_dropped`; the
series never ride the record (`3pm ls` stays small, the 1 MiB codec cap
stays far).

Bounds, per instance incarnation (constants in `app_metrics/limits.rs`):

| Bound | Value |
|---|---|
| live series, buckets included | 1 000 |
| distinct names | 200 |
| label sets per name; the excess folds into `overflow="true"` (counters and histograms summed, gauges last) — an SDK that folds its own excess sends `overflow="true"` itself, the one reserved label it may use | 50 |
| creations per hour (anti-churn): series / names | 1 000 / 200 |
| series not refreshed for | 120 s → dropped by a sweep every 10 s, whether or not the sampler runs (a source that disconnects drops its series at once) |
| entries per `metrics` line / non-empty buckets per histogram (an SDK lowers its scale before reaching it) | 256 / 128 |
| events per second: per instance (burst) / per fingerprint (burst) | 10 (50) / 1 (10) |
| event attributes / exception type / message / stacktrace | 16 keys of ≤ 256 chars, ≤ 4 KiB / 128 / 1 KiB / 8 KiB, truncated |

Refused entries count in `app_metrics_rejected`, suppressed events in
`app_events_dropped` (both on the record, shown by `3pm show`); a
suppressed event surfaces as `repeat` on the next one of its fingerprint
that gets through. Every incarnation starts empty: the sid space, the
revision and the counters restart with the process.

Events go out on `pub.sock` as `Event::App` (OpenTelemetry log-record
shape: `event_name`, `severity_number`, flat `attrs`, `exc.{type,
message,stacktrace}`, `fingerprint`, `repeat`); the daemon stamps the
time and replaces the instance's `cwd` and `$HOME` in the trace by `.`
and `~`. `3pm events -f app` follows them.

## Daemon self-monitoring (`DaemonInternals` — `3pm internals`)

| Field | Source |
|---|---|
| `mem_bytes`, `cpu_percent` | The daemon's own pid rides the metrics tick (real sampled values) |
| `metrics_tick_ms` | Duration of the last metrics tick — the process-table sweep behind the trees; `0` before the first tick |
| `metrics_stalled_ms` | How long the sweep in flight has been running, counted from when it began, once past `max(5 × period, 5 s)` (**Retrieval model**); absent otherwise. `3pm internals`: `metrics sweep  stalled for …`, red |
| `uptime_ms`, `pid`, `version`, `home` | Supervisor state |
| `open_fds` | Count of `/dev/fd` entries (macOS + Linux) |
| `nofile_soft/hard` | `getrlimit(RLIMIT_NOFILE)`; `RLIM_INFINITY` normalized to `u64::MAX` = "unlimited" |
| `load_avg` | `sysinfo::System::load_average()` |
| per-status process counts, restart totals | Supervisor process table |
| `bus_subscribers` | Live `pub.sock` subscribers (e.g. `3pm logs` followers) |
| `metrics_interval_ms`, `daemon_log_bytes`, `dump_count`, `dump_saved_at_epoch_ms` | Config/env + file metadata |

Reading it: daemon RSS trending up across days = daemon leak (`3pm update`
restarts it cleanly); `open_fds` climbing toward the soft limit = fd leak;
stale dump age = `resurrect` would restore old state.

## Host metrics (`HostMetrics` — `3pm hostmetrics`)

Grouped by section. **Platform**: ✔ = both, L = Linux-only, M = macOS-only.

### Identity & uptime — ✔

`hostname`, `os_name`, `os_version`, `kernel_version`, `arch`,
`boot_time_epoch_s`, `uptime_s` — sysinfo statics.

### CPU

| Field | Pf | Source |
|---|---|---|
| `cpu_brand`, `physical_cores`, `logical_cpus`, `cpu_frequency_mhz` | ✔ | sysinfo (frequency hidden in the human view when implausible — Apple Silicon reports junk) |
| `global_cpu_percent`, `per_core_percent` | ✔ | sysinfo, two refreshes over the warmup window |
| `per_core_freq_mhz` | ✔ | sysinfo, every fast tick (1 s), one per core in `per_core_percent`'s order (Linux: each core's `scaling_cur_freq`); `cpu_frequency_mhz` is its first |
| `cpu_states` (user/nice/system/interrupt/idle) | ✔ | Linux: own two-sample `/proc/stat` delta; macOS: `systemstat` (`host_processor_info`) |
| `cpu_states.iowait/softirq/steal/guest` | L | `/proc/stat` delta — `steal` is the hypervisor noisy-neighbor signal (red > 5%) |
| `kernel` (context switches, forks since boot, procs running/blocked, interrupts) | L | `/proc/stat` (`ctxt`, `processes`, `procs_*`, `intr`) |
| `load_avg` (1/5/15m) | ✔ | sysinfo |
| `rate_window_ms` (top level) | ✔ | Agent: length of the window every `*_per_s` rate below was derived over (≈ the fast tier period); absent on the first sample |
| `sampled_at_epoch_ms.{fast,medium,slow,extended,top}` (top level) | ✔ | Agent: when each sampling tier last landed (Unix ms); a tier not run yet is absent, the whole object absent before tiered sampling |
| `pressure` (PSI: cpu/mem/io, some+full avg10/60/300) | L | `/proc/pressure/*` (kernel ≥ 4.20; absent when CONFIG_PSI is off). The modern stall-based saturation signal — prefer it over load average |

### Memory

| Field | Pf | Source |
|---|---|---|
| `mem_total/used/available/free`, `swap_total/used` | ✔ | sysinfo |
| `mem_detail` (cached, buffers, slab, shmem, dirty, writeback, committed_as, commit_limit, anon_pages, swap_cached) | L | `/proc/meminfo` |
| `paging` (pgfault, pgmajfault, pswpin/out, pgpgin/out, **oom_kill**, allocstall, compact_stall) | L | `/proc/vmstat` — `oom_kill` is OOM detection without a logs pipeline (red when > 0) |
| `mem_compressor` (compressions, decompressions, compressor size, pageins/outs, swapins/outs) | M | `vm_stat` subprocess — the macOS memory-pressure signal |
| `cgroup` (mem limit/free/rss, cpu quota in cores, pids current/max) | L | `sysinfo::cgroup_limits()` + `/sys/fs/cgroup/{cpu.max,pids.*}` — the container's real allowance (host totals lie inside Docker/k8s) |

### Processes — ✔

`processes_count` and `process_states` (runnable/sleeping/zombie/… counts)
from a walk of `/proc/<pid>/stat` on Linux (the sysinfo process table
elsewhere; the state names are sysinfo's, lowercased). Userland threads
(Linux tasks) are excluded; kernel threads are counted — `ps ax`
semantics. With the top-processes table on, Linux takes the count from
that tier's own walk (every 5 s) while it is younger than the medium
period: a medium tick then reports figures up to one top period older
than its own walk would have been.

### Network

| Field | Pf | Source |
|---|---|---|
| `networks[]` (per interface: `ips` as addr/prefix, mac, rx/tx bytes + packets + errors, cumulative) | ✔ | Linux: counters from `/proc/net/dev` every fast tick, mac + ips from sysinfo `Networks` + `ip_networks()` (link-local filtered) every 10 s and when an interface appears; elsewhere sysinfo for all. Human view hides interfaces with no traffic and no address ("active only"); `--json` keeps all |
| `networks[].rx_bytes_per_s/tx_bytes_per_s/errors_per_s` | ✔ | Agent-derived over `rate_window_ms` (absent on the first sample / after a counter reset); feeds the `ls` host line |
| `tcp_states` (connection counts by state: established, listen, time_wait, …) | ✔ | Linux: `/proc/net/tcp{,6}` — every socket, other users' and ownerless TIME_WAIT included. macOS: `listeners::get_all()` — only sockets attributable to a visible process |
| `tcp_counters` (retrans/in/out segs, opens, failures, resets; + listen_drops, **listen_overflows**, syn_retrans, memory_pressures) | L (partial M) | Linux: `/proc/net/snmp` + `/proc/net/netstat` TcpExt. macOS: `netstat -s -p tcp` subprocess — absent when the OS zeroes the counters for unprivileged callers. Listen drops/overflows = the kernel silently refusing connections while the app looks healthy (red > 0) |
| `sockets` (in use, tcp orphan/tw/alloc/mem, udp) | L | `/proc/net/sockstat` |
| `conntrack` (count/max) | L | `/proc/sys/net/netfilter/nf_conntrack_*` (absent when the module isn't loaded; red > 90% full) |

### Storage

| Field | Pf | Source |
|---|---|---|
| `disks[]` (mount, fs, kind, used/total, cumulative read/written bytes) | ✔ | sysinfo `Disks` |
| `disks[].read_bytes_per_s/write_bytes_per_s` | ✔ | Agent-derived over `rate_window_ms`; the `ls` host line sums them per distinct device |
| `disks[].inodes_total/free` | ✔ | `statvfs` per mount — inode exhaustion looks like a "70% free" disk that can't create files |
| `block_devices[]` (read/write IOPS, merges, sectors, in-flight, io-time, queue-time) | L | `systemstat` `/proc/diskstats` decode (macOS would need hand-rolled IOKit — deliberately not done) |
| `raid[]` (array, member map, degraded flag) | L | `/proc/mdstat` — degraded = disk-failure early warning (red) |

### GPU

One `gpus[]` row per device; per-vendor blanks stay absent. Sources by
platform: `nvidia-smi` CSV (any box with the NVIDIA driver — chosen over
NVML FFI for the same no-unsafe rationale as the IOKit refusal, 2s
subprocess timeout), `/sys/class/drm` sysfs (L), `ioreg` IOAccelerator (M).
An NVIDIA card on a box without `nvidia-smi` still appears as an
identity-only row from sysfs (vendor/pci id/driver, gauges absent).

| Field | Pf | Source |
|---|---|---|
| `vendor`, `model`, `pci_id`, `driver`, `driver_version` | ✔ | nvidia-smi / sysfs `vendor`+`device`+`uevent` / ioreg `model`+`vendor-id`+`IOSourceVersion` |
| `utilization_percent` (instantaneous gauge) | ✔ | nvidia-smi `utilization.gpu` / amdgpu `gpu_busy_percent` / ioreg `Device Utilization %` |
| `vram_used/total_bytes` (dedicated VRAM) | L | nvidia-smi `memory.*` / amdgpu `mem_info_vram_*` |
| `unified_mem_used/alloc_bytes` | M | ioreg `In use/Alloc system memory` — driver-held **shared** memory, deliberately not called VRAM |
| `core_count` | M | ioreg `gpu-core-count` |
| `temperature_c`, `power_w`, `power_limit_w`, `fan_percent` | L | nvidia-smi / amdgpu `hwmon` (µW→W, m°C→°C) |
| `recovery_count` | M | ioreg `recoveryCount` — driver-recovered GPU hangs (red when > 0: the GPU is faulting) |

### Power

One `batteries[]` row per pack, plus the host-wide `on_ac`. Linux only:
`/sys/class/power_supply`, world-readable attribute files — no root, no
FFI, no daemon. macOS would need IOKit and stays absent rather than faked.
"No battery" has two shapes — the directory absent and the directory
present but empty — and neither is an error.

The kind of a supply comes from its `type` (`BAT0`, `BAT1`, `ACAD`,
`axp20x-battery`, `macsmc-battery` all exist), never from its name; a
`scope=Device` supply is a mouse or a headset and is excluded, while an
**absent** `scope` means System. **Every reading is optional and an
absent one stays absent**: an amp-hour gauge has no `energy_now` at all
and most packs have no `temp`, while a full pack legitimately draws
`0 W` — `null` and `0` are different answers and the renderers show
`N/A`, never a zero.

| Field | Pf | Source |
|---|---|---|
| `on_ac` | L | Any non-battery supply's `online` (Mains, USB, USB_PD, Wireless, UPS). Absent when the host exposes no such supply at all — not the same as running on battery |
| `batteries[].name`, `state`, `present` | L | The supply's directory name, `status` verbatim (`Charging`/`Discharging`/`Full`/`Not charging`/`Unknown` — the direction of the flow comes from here, never from the sign of a reading), `present` (`0` = an empty bay: the bay is reported, its stale numbers are not) |
| `batteries[].capacity_percent` | L | `capacity`, else derived from `energy_now`/`energy_full` |
| `batteries[].capacity_level` | L | `capacity_level` (`Critical`…`Full`) — the coarse last resort, carried only when no numeric capacity exists |
| `batteries[].energy_wh`, `full_wh`, `design_wh` | L | `energy_*` (µWh) directly, or `charge_*` (µAh) × volts on an amp-hour gauge; absent when such a gauge reports no usable voltage either |
| `batteries[].power_w` | L | `power_now` (µW), else `current_now` (µA) × volts. **Unsigned**: gauges disagree on the sign, so the magnitude is carried |
| `batteries[].voltage_v` | L | `voltage_now`, else `voltage_min_design`; a `0` reading is a driver exposing the file without the measurement, not a battery at zero volts |
| `batteries[].cycle_count`, `technology`, `model` | L | `cycle_count`, `technology`, `model_name` |
| `batteries[].temperature_c` | L | `temp`, tenths of °C — usually absent, and then absent; legitimately negative on a cold pack |

Health is `full_wh / design_wh` and is left to the consumer, like every
other derivative here. It can legitimately exceed 100% on a fresh pack:
nothing clamps it.

### System health

| Field | Pf | Source |
|---|---|---|
| `file_handles` (allocated/max, host-wide) | L | `/proc/sys/fs/file-nr` |
| `vfs_cache` (dentries, inodes) | L | `/proc/sys/fs/{dentry-state,inode-nr}` |
| `entropy_avail` | L | `/proc/sys/kernel/random/entropy_avail` |
| `sensors[]` (label, temp/max/critical) | ✔ (thin off Linux/Apple-Silicon) | sysinfo `Components` (hwmon / Apple PMU) |
| `ntp_offset_ms` | ✔ | `rsntp` SNTP probe, 500ms timeout, best-effort — clock drift breaks TLS/auth/log ordering and nothing else detects it (red past ±100ms) |

### 3pm itself (`3pmDaemon`) — ✔

What 3pm costs the box, measured by the agent (not self-reported), on
the medium period; the whole object is absent from a daemon predating
it and until the tier has landed once (`hostagent.md` → *3pm's own
metrics*).

| Field | Pf | Source |
|---|---|---|
| `3pmDaemon.at_epoch_ms` | ✔ | Agent: when this part landed (Unix ms) |
| `3pmDaemon.processes[]` — `role` (`daemon`, `host-agent`), `pid`, `cpu_percent`, `mem_bytes`, `uptime_ms` | ✔ | Agent: `sysinfo` refresh of the two pids (its parent and itself); cpu % is a delta over the tier's window (0 on the first sample); a pid the kernel no longer knows has no row |
| `3pmDaemon.processes[].open_fds` | ✔ agent row; daemon row: L from the OS, ✔ via the daemon | `/dev/fd` for the agent itself, `/proc/<pid>/fd` for the daemon on Linux; elsewhere the daemon row takes `Internals.open_fds` (the daemon's own count) |
| `3pmDaemon.processes[].threads` | L | `Threads:` of `/proc/<pid>/status`; absent elsewhere |
| `3pmDaemon.runtime` — `version`, `boot_id`, `uptime_ms`, `apps_total`, `online`/`launching`/`stopping`/`stopped`/`waiting_restart`/`errored`, `restarts_total`, `unstable_restarts_total`, `timers_armed`, `bus_subscribers`, `log_sinks`, `log_rotations_total`, `log_bytes_written_total`, `daemon_log_bytes`, `nofile_soft`/`nofile_hard`, `handoffs`, `host_agent_restarts` | ✔ | Agent: `Request::Internals` over `rpc.sock` (one connection per tick, 2 s bound), reduced to this health subset of `DaemonInternals` — no `home`, no pid, no rotation policy. Absent for the tick when the daemon did not answer (down, wedged, handing off, skewed) |

### Extended Monitoring (`extended`) — L

The kernel's own counters, family by family (`HostExtended`: `families`
→ `kind` + rows → fields, and `unavailable` → reason); absent with the
mode off, present and empty on macOS. The human view is `3pm hostmetrics
--extended` (`hostagent.md` → *Extended Monitoring*); the default view
never shows them.

### Top processes (`top_processes`) — L, macOS

The heaviest processes of the whole host, one table of at most 30 rows
sorted by pid: the ten largest by CPU, by memory (the footprint on
macOS, the RSS on Linux) and by network rate (in + out), merged. Absent with `[hostmetrics] top_processes` off,
on the first tick (it primes), and on a platform with no reader;
`sampled_at_epoch_ms.top` is when the table last landed, which is how a
reader tells a stale one.

| Field | Source |
|---|---|
| `pid`, `name` | Linux: the kernel's process name. macOS: `ps` `ucomm` (16 bytes), completed from the executable's file name (`proc_pidpath`) when it may be cut. Never a path or arguments; cleaned and cut like a tree name |
| `user` | the account of the process's uid; the number when it has no name; absent when the platform does not say |
| `cpu_percent` | cumulative CPU time differenced over the real window between two readings, 100 = one core. A pid seen for the first time (or reused) has no row that window |
| `mem_bytes` | RSS, on both platforms. Not the PSS / footprint a managed process has in its tree |
| `footprint_bytes` | macOS only: `phys_footprint`, what Activity Monitor calls Memory, from one `top -l 0 -s <period> -stats pid,mem` kept running by the host agent (~1.8 % of a core at 5 s after a 0.78 s first sample, measured 2026-10-01; a `top -l 1` per tick would cost 15 %). A sample is used while at most two periods old. Absent on Linux, for a pid `top` did not list, and while `top` is down. When a table has it, the memory top was ranked on it — a row without one has no memory rank, never its RSS; without it, on the RSS |
| `net_rx_bytes_per_s`, `net_tx_bytes_per_s` | cumulative per-pid counters differenced over the window. macOS: `nettop -P -n`. Linux: `sock_diag` (both families in one dump, `tcp_info`) and the fd tables, as `ss -tinpH` reported them (`ss` itself where `sock_diag` is refused), TCP only, `bytes_received` and `bytes_sent` (`bytes_acked` before kernel 4.19) summed per holder — a shared socket counts whole for each; other users' sockets only under root. Absent, never zero, when there is no source, the process is not visible to it, or a counter went backwards |

A failed process read publishes an empty table. No series is stored for
these rows by the backend (pids churn); they are live state.

## Known gaps (deliberate)

- **Rates/derivatives** (`await`, `%util`, per-second anything) — needs
  periodic polling; belongs to the consumer (see Retrieval model).
- **Exact memory of a tree** — the RSS sum double-counts shared pages.
  Extended Monitoring reads PSS (`/proc/<pid>/smaps_rollup`, summed into
  `pss_bytes`) and `phys_footprint` (macOS, per member only); with the
  mode off, and on macOS for the app total, the RSS sum is all there is.
  The cgroup's `memory.current` is not read.
- **Detached members and `3pm stop`** — a member that left the process
  group is counted and marked, not signalled: the stop leaves it alive.
- **macOS**: disk IOPS (IOKit) and page-fault counters (raw mach
  `host_statistics64`) — no maintained crates; hand-rolled unsafe not
  worth it.
- **Different-league sources**: per-IRQ rates, NUMA, buddyinfo, schedstat,
  SMART, systemd unit states — need root, heavy deps, or a daemon of
  their own. (Wi-Fi link quality, softirqs, `vmstat`, `/proc/net/*` and
  the other kernel counter files are Extended Monitoring families, Linux.) (Batteries were listed here until the Power section: sysfs
  `power_supply` needs none of the three.)
- **GPU processes / clocks / ECC detail** — the `gpus[]` section covers
  inventory + core gauges; per-process GPU memory, clock speeds and ECC
  counters would need NVML/vendor FFI, declined for the same no-unsafe
  reason as IOKit.

## Network traffic per app

`ProcessRecord.net` and `TreeProc.net`, measured by the daemon on its
metrics loop, one window every 5 ticks (`metrics::net::WINDOW_TICKS`; 5 s
at the default interval). The first window primes; figures are carried
between windows; the meter starts with the first app and is dropped by
the first tick with none (no `nettop`, no notification socket, no
thread on an idle daemon), and a handoff rebuilds it, so the counters
restart then.

**The app's figure is its traffic with the outside of the app.** A
connection whose two ends are processes of the same app (Electron ⇄
its `node`, `npm` ⇄ its server, on loopback or the host's own address)
is counted for each process (`TreeProc.net`, the `net ↓/↑` column of
`3pm show`'s ` Processes` table) and not for the app
(`ProcessRecord.net`, `3pm ls`, the backend's `process.net.*`): each of
its bytes would come in twice, once per end. The members' figures can
therefore add up to more than the app's. Traffic with another app or
with a local database still counts. One opened and closed between two
windows on Linux is seen on its accepted side alone and counts once.

| Platform | Source | Counts | Misses |
|---|---|---|---|
| Linux | `sock_diag` dumps of our TCP sockets with `INET_DIAG_INFO` (both families in one, `TCPDIAG_GETSOCK`: one walk of the kernel's table, not one per family) (`tcpi_bytes_received`, `tcpi_bytes_acked`), filtered in the kernel on our local ports (`INET_DIAG_REQ_BYTECODE`); the members' fd tables (`socket:[inode]`); the `SKNLGRP_INET{,6}_TCP_DESTROY` notifications (final counters by cookie) behind a BPF filter on the same ports — all unprivileged | each socket's delta between dumps; a socket closed since the last dump: its tail, by cookie; an accepted connection, alive or closed, through its listener (whole the first time) | an outbound connection opened and closed within a walk's period (10 s, 30 s when the members hold 2 000 sockets or more); a listener shared by several instances (counted as `unattributed`); UDP |
| macOS | one `/usr/bin/nettop -n -d -s 1 -L 0 -x -J interface,bytes_in,bytes_out` (per socket: each process row followed by its sockets) kept running on a pty (on a pipe it buffers ten seconds of output), under a sentinel `sh` that kills it when the daemon's pipe closes (`nettop` outlives its terminal), its stdin on that pipe (on `/dev/null` it spins a core and a half), signal mask cleared, `LC_ALL=C` | per-process deltas, closed flows included, loopback (`lo0`) too; the `lo0` sockets that moved bytes, to pair the two ends of a connection inside an app; the first block (cumulative) skipped. Measured 2026-10-02: 0.32 s of `nettop` CPU per 10 s (0.07 with `-P`), 12 KB a second to read | UDP pairs; a dead or silent (10 s) `nettop` is replaced at most every 30 s, figures absent meanwhile |
| other | — | — | everything (absent) |

**What it costs is the apps', not the host's** (Linux). The dump returns
only sockets bound to our ports (the members' listeners and the ports of
the sockets they hold, at most 1 024 — beyond, an unfiltered dump keeping
ours, and the notifications on the listeners alone); another process's
connection closing is dropped by the kernel before it wakes the reader.
With no port of ours there is no dump and no subscription. A member's fd
table is read when the member is new, every member's every 2 windows
while they hold fewer than 2 000 socket inodes (30 s / 6 windows above),
and when a filtered dump shows a socket on our ports nobody is known to
hold (at most every 2 windows). A walk that found a TCP socket no dump
has shown is followed by one whole dump, to learn its port; inodes no
whole dump shows (unix, UDP) are remembered so they never ask again.
Measured 2026-10-02 (edgecr4ft, release): 40 000 host sockets churning
at thousands of connections a second from processes 3pm does not manage
cost the notification thread nothing; an app holding 10 000 connections
(20 000 fds) costs the daemon ~1.2 % of a core and ~18 MB over the same
daemon without it measuring (2.5 % and 30 MB in all).

The accounting is socket by socket (`metrics/net/sockets.rs`): a socket
alive at the dump counts `now − before`, or `now` when the previous dump
did not have it; a counter that went backwards is another socket; a
socket the previous dump had and this one does not is kept one window
for a late notification. Shared sockets: whole for each member holding
one, once per instance. An accepted connection credited by its listener
goes to the instance, and to a member only when one pid alone holds the
listener. The two ends of a connection are paired by address (the
peer's local address is this one's remote): when an instance owns
both, neither counts for it; the tail of such a socket, once closed,
stays out of it too. A notification the reader's buffer (50 000) or the kernel's
queue (`ENOBUFS`) could not hold is counted in
`DaemonInternals.net_meter.events_lost` and flags the window `partial`.
`3pm internals` shows the source (`netlink`, `netlink (no destroy
notifications: …)`, `nettop`, `unavailable: …`), the unattributed bytes
and the lost notifications.

</document>

<document url="https://3pm.edgecr4ft.net/docs/hostmetrics.md" title="Host metrics agent">

# Host metrics agent — `3pm hostmetrics` from a separate, cached process

`3pm hostmetrics` reports the whole box (CPU incl. steal, PSI, memory,
disks, RAID, network, TCP health, sensors, GPUs, batteries, NTP offset — field list
in `metrics.md`). Since the host agent landed, **none of that is
collected by the daemon**: a dedicated process samples it on timers —
one per cost tier, see *Sampling tiers* below and `hostagent-tiers.md` —
and the daemon only relays the latest assembled snapshot. Long-form
rationale: `FEATURES.md` → *Host metrics agent*; internals:
OV `3pm-docs-archived/architecture` §8.2.

## Why a separate process

Host collection is the code most exposed to the outside world: `/proc`
parsers, `sysinfo`/`systemstat`, a UDP NTP probe, `nvidia-smi` and
`ioreg` subprocesses, per-vendor GPU quirks. A panic, a memory leak or a
hang there must never reach the supervisor that owns your apps. The
agent is that blast wall: it can die, leak or wedge, and the worst
outcome is `3pm hostmetrics` reporting `unavailable` until the daemon has
respawned it.

Two side effects you get for free:

- **Instant answers.** The command reads a cache: ~10 ms instead of the
  200–700 ms a live collection needs (CPU% is a delta over a warmup
  window, plus the NTP round trip). The daemon forwards the agent's
  reply line as it came (`hostagent/proxy.rs`): it looks at the tag the
  line opens on, never decodes the ~50 KB snapshot, and does not encode
  it again — save for a hosted user's copy, which is decoded to be
  scrubbed (`Caller::scrub_host`).
- **A steady sampling cadence**, which is what a fleet/monitoring layer
  wants to pull anyway.

## What it looks like

```
3pm (CLI) ──rpc.sock──▶ 3pm __daemon ──host.sock──▶ 3pm __host-agent
                          │ spawn / respawn / kill        │ fast 1 s / medium 10 s / slow 5 min → cache
                          └── Request::HostMetrics = proxy └── serves ping + host_metrics
```

```console
$ 3pm ls
┌────┬──────┬─────┬────────┬───┬────────┬──────────┬─────┬─────┬───────┐
│ id ┆ name ┆ pid ┆ uptime ┆ ↺ ┆ status ┆ watching ┆ cpu ┆ mem ┆ ports │
…
host metrics | cpu: 38.8% 62.8º | load: 7.42 | ram: 63.5% | gpu: 22% 1199/11538mb | en0: ⇓ 1.2mb/s ⇑ 300kb/s | disk: ⇓ 512kb/s ⇑ 2.0mb/s

$ 3pm hostmetrics | head -4
 box — Linux 6.8 (x86_64)
┌─────────┬───────────────────────────────────┐
│ sampled ┆ 2026-08-28T15:02:48Z (3s ago)     │
│ uptime  ┆ 48D                               │

$ 3pm hostmetrics --json | jq .collected_at_epoch_ms
1787490168000

$ 3pm internals | grep 'host agent'
│ host agent  ┆ pid 50632, 0 restarts │

$ 3pm internals | tail -6
 internal processes
┌────────────┬──────────────────┬───────┬────────┬────────┬──────┬────────┬───┐
│ role       ┆ command          ┆ pid   ┆ status ┆ uptime ┆ cpu  ┆ mem    ┆ ↺ │
│ daemon     ┆ 3pm __daemon     ┆ 50631 ┆ online ┆ 2D     ┆ 0.4% ┆ 26.7mb ┆ 0 │
│ host-agent ┆ 3pm __host-agent ┆ 50632 ┆ online ┆ 2D     ┆ 0.2% ┆ 35.9mb ┆ 0 │
└────────────┴──────────────────┴───────┴────────┴────────┴──────┴────────┴───┘

$ ps -o pid,ppid,command | grep 3pm
50631     1 3pm __daemon
50632 50631 3pm __host-agent
```

The `host metrics` line under `3pm ls` (pm2 parity; also printed by
`start`/`stop`/`restart`/`delete`/`scale`, which reprint the table) is
the same cache read under a 500 ms budget: cpu (+ hottest CPU sensor),
load 1m, ram %, gpu (utilization, memory, temperature when a GPU
reports), one entry per physical NIC carrying traffic, disk throughput
plus any mount over 60 % full. Colors follow pm2's thresholds. Absence
of the line is never an error (agent down, no sample yet, wedged);
`ls --json` stays the bare records array. Throughput segments need two
fast samples — they show up from the second tick on (~2 s).

The `sampled` row (and `collected_at_epoch_ms` in `--json`) tells you
how old the cheap gauges are; `sampled (medium)` / `sampled (slow)`
rows appear only when a slower tier lags it by more than 2 s
(`sampled_at_epoch_ms.{fast,medium,slow}` in `--json`). `host agent` in
`3pm internals` tells you the agent's pid and how many times the daemon
had to respawn it since it started; its `internal processes` table adds
the agent's uptime, cpu and RSS next to the daemon's (`internal_processes`
in `--json`) — the place to watch a long-lived agent for a leak. The
agent's cpu/mem ride the daemon's 1 s metrics tick and are attributed
only while the sampled pid is the current one, so a fresh incarnation
starts at 0 instead of inheriting its predecessor's numbers; between
incarnations the row reads `waiting restart` with no pid.

## Sampling tiers

Collectors are grouped by cost, each on its own ticker and blocking
thread (`hostagent/sampler.rs`), so a hung `nvidia-smi` or a stalled
network mount never delays CPU/RAM (`docs/hostagent-tiers.md` has the
audit behind the split):

| Tier | Period (knob) | What |
|---|---|---|
| fast | 1 s (`THREEPM_HOSTMETRICS_INTERVAL_MS`) | cpu %/per-core/states, kernel, load, uptime, memory/swap, paging, meminfo, PSI, file handles, sockets, TCP counters, conntrack, cgroup, VFS cache, entropy, NIC counters + rates, block devices, per-mount I/O + rates |
| medium | 10 s (`THREEPM_HOSTMETRICS_MEDIUM_MS`) | process table, TCP states (socket table), sensors, GPUs, mount space + inodes (`statvfs`), RAID, batteries (`/sys/class/power_supply`, Linux), macOS `vm_stat`/`netstat`, logged-in accounts (`sessions`, one utmp walk) |
| slow | 5 min (`THREEPM_HOSTMETRICS_SLOW_MS`) | NTP offset; host identity re-read (hostname, OS, kernel, CPU model/cores, the account the daemon runs as) |
| 3pm | = medium | 3pm's own footprint (`3pmDaemon`, below): the daemon and the agent read from the OS, the daemon's counters pulled over `rpc.sock` |

The fast collector is stateful (keeps its `sysinfo` handles and the
previous CPU ticks): the tick is the CPU-% window, so only the very
first sample pays a 200 ms warmup. A snapshot is served once identity,
fast and medium have each landed once; the slow tier (up to 500 ms of
NTP) is not waited for.

## 3pm's own metrics (`3pmDaemon`)

The snapshot also reports what 3pm itself costs the box, collected by
the agent like everything else — **not** self-reported by the daemon —
on a tier of its own (`hostagent/selfmetrics.rs`, medium period) from
two sources kept apart so either can fail alone:

- **The OS.** One row per 3pm process — the daemon (the agent's parent
  pid, the one `watchdog.rs` already polls) and the agent itself — cpu %
  over the tier's window, RSS, uptime, open descriptors and threads
  (`hostmetrics/threepm.rs`). A process the kernel no longer knows is
  left out, never reported at 0. Descriptors and threads of a *foreign*
  pid are Linux-only (`/proc/<pid>/fd`, `/proc/<pid>/status`); on macOS
  the daemon row's `open_fds` comes from the daemon's own count instead
  and `threads` is absent.
- **The daemon.** `Request::Internals` on `rpc.sock` — the counters
  that exist nowhere else (apps by status, restarts, timers, bus
  subscribers, log sinks/rotations/bytes, `RLIMIT_NOFILE`, handoffs,
  agent respawns), reduced to the health subset `ThreepmRuntime`
  (nothing about paths or configuration: the snapshot rides the fleet
  link). One connection per tick — `connect_at` (ping + protocol check,
  never a spawn) then the request — bounded to 2 s in total. A dead,
  wedged, handing-off (`3pm update`) or version-skewed daemon blanks
  `runtime` for that tick while the rows keep coming; the previous part
  is served meanwhile, and the command never fails for it.

`3pm hostmetrics` prints the two tables after the sensors; `--json`
carries `3pmDaemon: {at_epoch_ms, processes[], runtime?}`. The section
is optional on the wire and never gates the first snapshot (identity +
fast + medium still do), so a slow daemon cannot delay `hostmetrics`.
The daemon's own 1 s self-sample (`3pm internals` → `internal
processes`) is unchanged: `internals` does not depend on the agent.

```console
$ 3pm hostmetrics | tail -22
 3pm processes
┌────────────┬───────┬────────┬──────┬────────┬─────┬─────────┐
│ role       ┆ pid   ┆ uptime ┆ cpu  ┆ mem    ┆ fds ┆ threads │
│ daemon     ┆ 50631 ┆ 2D     ┆ 0.4% ┆ 26.7mb ┆ 41  ┆ 9       │
│ host-agent ┆ 50632 ┆ 2D     ┆ 0.2% ┆ 35.9mb ┆ 18  ┆ 7       │
└────────────┴───────┴────────┴──────┴────────┴─────┴─────────┘

 3pm runtime
┌────────────┬─────────────────────────────────────────────────────┐
│ daemon     ┆ v0.9.3, up 2D, boot 0x1f3a, 1 handoff               │
│ apps       ┆ 12 (11 online, 1 errored)                           │
│ restarts   ┆ 37 (2 unstable)                                     │
│ timers/bus ┆ 4 armed, 1 subscriber                               │
│ logs       ┆ 12 sinks, 3 rotations, 1.1gb written; 3pm.log 4.0kb │
│ nofile     ┆ 1024 / 1048576                                      │
│ host agent ┆ 0 restarts                                          │
│ sampled    ┆ 2026-08-28T15:02:40Z (11s ago)                      │
└────────────┴─────────────────────────────────────────────────────┘
$ 3pm hostmetrics --json | jq '."3pmDaemon".runtime.apps_total'
12
```

## Extended Monitoring (`3pm hostmetrics --extended`)

With `[hostmetrics] extended` on (the default, below), the snapshot's
`extended` map carries the kernel's own counters. `3pm hostmetrics`
leaves them out — a Linux box has ~1 200 — and `--extended` prints them
**instead of** the overview (the typed view is one command away and
would otherwise scroll out of reach): the host banner, a count, one table
per family and last the sources that could not be read.

```console
$ 3pm hostmetrics --extended | grep ' — '
 labomatic — Ubuntu Linux (Ubuntu 20.04) (5.4.0-153-generic, x86_64)
 extended monitoring — 23 families, 1228 values
 cpu — counters since boot, 32 rows
 softirqs — counters since boot
 meminfo — gauges
 vmstat — counters since boot
 vmstat_gauge — gauges
 …
 cpuidle — counters since boot, 96 rows
 cpufreq — gauges, 31 rows
 thermal — gauges, 32 rows
$ 3pm hostmetrics --extended | tail -6     # the agent runs as ubuntu, not root
 unavailable
┌────────────┬──────────┐
│ family     │ reason   │
├────────────┼──────────┤
│ shortlived │ not_root │
└────────────┴──────────┘
```

The same host with a root agent has no `unavailable` table and three
more families, `shortlived`, `shortlived_by_comm` and
`shortlived_by_comm_max`, last (the table does not name them).

- **Order**: the collection table's (`hostmetrics/extended/table.rs`,
  `EXTENDED` then `SLOW`, exported as
  `threepm_daemon::host_extended_family_order`), each `<family>_gauge`
  right after its family, then any family the table does not name
  (a newer daemon, eBPF's `shortlived*`) alphabetically. Rows sort
  naturally (`2` before `10`, `0/state2` before `0/state10`).
- **Values**: `meminfo` in bytes (its `HugePages_*` lines are page
  counts and stay numbers), whole numbers plainly, fractions to three
  decimals; every other unit is the kernel's own (µs, kHz, m°C, USER_HZ
  ticks). Nothing is summarized, nothing hidden: all-zero rows print.
- **`unavailable`**: a closing `family | reason` table (`permission`,
  `error`, and eBPF's `not_root` / `bpf_disabled` / `load_failed`); a
  family present but only partly read says `(partial: <reason>)` in its
  title.
- **Mode off** (`extended` absent once the slow tier has landed, since
  with the mode on that tier always carries the map): one line,
  `extended monitoring is off`. **Not yet** (`extended` absent and no
  `sampled_at_epoch_ms.slow`: the agent just started or respawned, or
  the daemon predates Extended Monitoring): `extended metrics not
  collected yet`. **macOS** (present, empty): `no extended source` on
  this platform.
- `--json` is the whole snapshot either way, `extended` included.
- **Short-lived processes** (Linux, the agent running as root): the
  eBPF families `shortlived` (every exec and exit since the agent
  started, and how many lived under a second), `shortlived_by_comm`
  (exits and lifetime per process name, the window's 10 busiest names
  and `other`; processes whose exec was seen, so a fork-only worker is
  in `shortlived` alone) and `shortlived_by_comm_max` (the window's longest life
  per name). An agent without the capabilities (`CAP_SYS_ADMIN`, or
  `CAP_BPF` + `CAP_PERFMON`, read from `CapEff` before any load) lists
  `shortlived | not_root` under `unavailable`; macOS says nothing. Those
  capabilities are necessary, not sufficient: attaching also reads
  tracefs (`/sys/kernel/tracing/events/…/id`, often `0700` root: uid 0
  or `CAP_DAC_READ_SEARCH`), Ubuntu's `perf_event_paranoid = 3` may
  refuse `perf_event_open` without `CAP_SYS_ADMIN`, and before Linux
  5.11 the maps count against `RLIMIT_MEMLOCK` — any of these is
  `not_root` too, found by trying. How and why:
  `docs/hostagent-tiers.md` §3.7.

## Lifecycle

| Moment | What happens |
|---|---|
| Daemon boot | The daemon re-execs itself as `3pm __host-agent` (stdin null, stdout/stderr → `~/.3pm/hostagent.log`), waits for its readiness byte. |
| Every 1 s / 10 s / 5 min | Each tier collects on a blocking thread, stamps its part, and the cache re-assembles the snapshot (`collected_at_epoch_ms` = the fast stamp). A collector panic is logged, that tier's previous part stays served and the collector is rebuilt. |
| Agent dies (crash, `kill -9`) | The daemon respawns it after an exponential backoff: 1 s, 2 s, 4 s … capped at 30 s, reset once an incarnation has lived 60 s. `restarts` increments. |
| Agent alive but not answering (stopped, deadlocked, no longer listening) | Health check: from one period after readiness, the daemon pings `host.sock` every 5 s (`THREEPM_HOSTAGENT_HEALTH_MS`), each ping bounded by 3 s (never above the period). A success resets the count; 6 misses in a row (~30 s) and the daemon logs `host agent is unhealthy; replacing it` with the reason `wedged`, SIGTERMs it (SIGKILL after 2 s), and respawns it through the same backoff. `restarts` increments. A short `SIGSTOP` or a loaded box stays far below the threshold. |
| Agent answers with another protocol | It is alive, so it is left alone: one warning in `3pm.log` (`` host agent runs a different protocol — run `3pm update` ``), `hostmetrics` says `unavailable`, `restarts` does not move. A respawn would run the same binary and be skewed again. On Linux a respawn runs the daemon's own image (below), so this only happens on macOS after a new binary was installed and before `3pm update`. |
| Binary replaced on disk (`install.sh` `mv`s the new one over the path) | A respawn execs the binary's path while it is usable (the process is then named `3pm`). On Linux, once `current_exe()` reads `<path> (deleted)` (which nothing can exec), it execs `/proc/self/exe` instead, the daemon's running image even unlinked: daemon and agent stay the same version until `3pm update` replaces both; that incarnation's process name (`comm`, what `ps`/`top`/`pgrep` show) is `exe`. The handoff guardian's fallback (re-run the previous binary when the new one cannot be exec'd) follows the same rule. macOS: the path, as before. |
| `3pm kill` / SIGTERM / `3pm update` | The daemon sends the agent SIGTERM, waits 2 s, SIGKILLs, and only then removes its own sockets. `3pm update` therefore restarts both binaries together (the daemon checks the agent's protocol version on every request). |
| Daemon SIGKILLed | The agent notices its parent pid changed (1 s poll) and exits on its own, unlinking `host.sock`. |
| `THREEPM_HOME` deleted | Same watchdog: the agent exits, like the daemon does. |

The agent holds `flock(hostagent.lock)` for its lifetime (a second copy
started during a teardown race exits 0 and the respawn loop tries
again) and writes `hostagent.pid`.

## Memory

The agent runs **two** tokio workers (`WORKER_THREADS`; every collector
is on `spawn_blocking`, whose pool is sized apart): the default of one
per core gave 37 threads on a 32-core host, each worker keeping its own
glibc arena at its high-water mark.

On Linux with glibc, the agent's first act is `mallopt(M_MMAP_THRESHOLD,
128 KiB)`. Left dynamic, glibc raises the mmap threshold (and the trim
threshold with it) to the size of every multi-megabyte block freed — the
eBPF loader's parse of the kernel BTF frees 9 MB ones, a process table
filled afresh is megabytes more — and from then on such blocks come from
the arenas, each blocking thread's arena keeping its peak (~10 MB
apiece). Measured on a 32-core host: 80 MB of RSS left dynamic, 18.6 MB
pinned; with the eBPF gate and the single BTF parse, anonymous memory
after 15 min went 17.2 → 6.6 MB (Extended Monitoring on, user agent) and
17.1 → 8.3 MB (root agent). The call overrides `MALLOC_MMAP_THRESHOLD_`
and `GLIBC_TUNABLES=glibc.malloc.mmap_threshold=…` for the agent (glibc
reads those at exec, the `mallopt` comes after); the daemon and the apps
keep whatever their environment says. musl and macOS: nothing is set.

## Failure semantics — no silent fallback

The daemon **never collects inline**. If the agent cannot answer, the
command fails with a retryable error naming the log:

```console
$ 3pm hostmetrics
[3PM][ERROR] host metrics agent is not reachable (…) — see /home/u/.3pm/hostagent.log
$ echo $?
1
$ 3pm hostmetrics --json
{"error":{"code":"unavailable","message":"…"}}
```

Bounds on the daemon side, so the CLI can never hang on a broken agent:

| Situation | Behaviour |
|---|---|
| Agent mid-respawn | Connect retried every 100 ms for up to 2 s, then `unavailable`. |
| Agent just started, no sample yet | The agent waits up to 2 s for its first collection before answering `unavailable`. |
| Agent accepts but never answers (stopped, deadlocked) | Ping and request each bounded by 3 s → `unavailable`. |
| Agent binary from another 3pm version | Immediate `unavailable` (protocol skew); the agent is left running (a respawn would be skewed too); run `3pm update`. |

The first `3pm hostmetrics` right after a daemon spawn can therefore take
a few hundred milliseconds (it waits for the first sample rather than
failing); every later call is a cache read.

## Knobs and files

| Knob | Default | Meaning |
|---|---|---|
| `THREEPM_HOSTMETRICS_INTERVAL_MS` | `1000` (min `200`) | Fast tier period (cheap gauges). Read by the agent at start; restart the daemon to apply. |
| `THREEPM_HOSTMETRICS_MEDIUM_MS` | `10000` (min = fast) | Medium tier period (walks, forks, `statvfs`). A value below its floor is raised, with a warning in `hostagent.log`. |
| `THREEPM_HOSTMETRICS_SLOW_MS` | `300000` (min = medium) | Slow tier period (NTP probe, identity re-read). Same clamp rule. |
| `THREEPM_HOSTMETRICS_EXTENDED_MS` | `10000` (min = fast) | Extended Monitoring tier period (the kernel's own counters). Same clamp rule. |
| `THREEPM_HOSTMETRICS_EXTENDED` | unset | `1`/`true`/`yes`/`on` or `0`/`false`/`no`/`off` (any case): Extended Monitoring on or off, above `[hostmetrics] extended` (below); any other value is logged and ignored. Read from the daemon's environment at start: a handoff keeps the old environment, so `THREEPM_HOSTMETRICS_EXTENDED=0 3pm update` changes nothing — restart the daemon (`3pm kill`, then any command). |
| `THREEPM_HOSTMETRICS_TOP_MS` | `5000` (min = fast) | Top-processes tier period (`top_processes`). Same clamp rule. |
| `THREEPM_HOSTMETRICS_TOP_PROCESSES` | unset | Same values and rules as `THREEPM_HOSTMETRICS_EXTENDED`: the top-processes table on or off, above `[hostmetrics] top_processes`. |
| `THREEPM_NTP_SERVER` | `pool.ntp.org` | NTP probe server used by each collection; empty disables the probe. |
| `THREEPM_HOSTAGENT_HEALTH_MS` | `5000` (min `100`) | Daemon side: the agent's health-probe period. 6 missed probes in a row put the agent down; each probe is bounded by `min(3 s, period)`. Read by the daemon at start. |
| `THREEPM_SPAWN_TIMEOUT_MS` | `3000` | Readiness wait for spawned processes (daemon and agent alike). |
| `THREEPM_LOG` | `info` | Tracing filter, honoured by the agent too. |

| File under `~/.3pm` | Purpose |
|---|---|
| `host.sock` | Agent RPC socket (0600, same-uid peer check). Internal: the daemon is its only client. |
| `hostagent.lock` | The agent's singleton flock. |
| `hostagent.pid` | Its pid (informational; liveness is the socket ping). |
| `hostagent.log` | Its stdout/stderr — the first place to look when `hostmetrics` says `unavailable`. |

### `[hostmetrics]` in `config.toml`

```toml
[hostmetrics]
extended = true        # the default; the whole section may be absent
top_processes = true   # the default
```

Extended Monitoring: resolved as `THREEPM_HOSTMETRICS_EXTENDED` > this key >
`true`, once per daemon image (`hostagent::daemon_extended`); the daemon's
metrics sampler (per-process fields) uses that answer and hands it to every
host agent incarnation (host families) in `THREEPM_HOSTMETRICS_EXTENDED`,
so the two always agree. A change to this key takes `3pm update` (the new
image re-reads the file); an agent crash-restart alone does not. A value
that is not a boolean, or an unknown key, is named in `3pm.log` at daemon
boot and leaves the default; never fatal. `extended = false`: no extra
kernel file is read and nothing extra goes on the wire.

`top_processes` is resolved the same way and as independently
(`THREEPM_HOSTMETRICS_TOP_PROCESSES` > this key > `true`, once per daemon
image, `hostagent::daemon_top`). `top_processes = false`: no tier, no
`ps`/`nettop`/`ss` fork, no process walk, and neither `top_processes` nor
`sampled_at_epoch_ms.top` on the wire. What the table holds and how it is
read: `hostagent-tiers.md` §3.8.

## Rates

The OS only exposes cumulative counters. The agent keeps its previous
fast part and derives, over the window between two fast samples
(`rate_window_ms`, ≈ the fast period): `HostNic.rx_bytes_per_s` / `tx_bytes_per_s` /
`errors_per_s` and `HostDisk.read_bytes_per_s` / `write_bytes_per_s`.
A rate is absent on the first sample, for a NIC/mount that was not in
the previous sample, and after a counter reset (hot-plug, driver
reload) — never faked as a huge negative-wrapped number.

## Wire

Additive within protocol v1 (see `protocol.md`): `HostMetrics.collected_at_epoch_ms`
(optional; absent from a daemon predating the agent), the `*_per_s`
rates and `rate_window_ms` above, `HostMetrics.sampled_at_epoch_ms`
(optional `{fast, medium, slow}` stamps; absent before tiered sampling),
`DaemonInternals.host_agent_pid` / `host_agent_restarts`,
`HostMetrics.3pmDaemon` (optional; absent from a daemon predating
self-metrics), and `ErrCode::unavailable`. `host.sock` speaks the same `Request`/`Response`
NDJSON as `rpc.sock` but only serves `ping` and `host_metrics`; anything
else is `err{unknown_request}`.

## Troubleshooting

- **`unavailable` on every call** → `tail ~/.3pm/hostagent.log`; check
  `3pm internals` for a climbing `restarts` count (a crash loop) and
  whether `host.sock` exists.
- **`host agent is unhealthy; replacing it` in `3pm.log`** → the
  agent stopped answering its ping for ~30 s (reason `wedged`). The
  previous incarnation's last lines are in `hostagent.log`.
- **`host agent runs a different protocol` in `3pm.log`** → the agent
  binary is not the daemon's version (macOS, a new binary installed
  under a running daemon): run `3pm update`.
- **Stale `sampled` age (minutes)** → the agent is alive but its
  collector is stuck (typically a hung `nvidia-smi`/`ioreg`, though each
  subprocess has a 2 s deadline). `kill <agent pid>`: the daemon respawns
  it, the cache refreshes on the next tick.
- **Want fresher data** → lower `THREEPM_HOSTMETRICS_INTERVAL_MS`
  (cheap gauges) or `THREEPM_HOSTMETRICS_MEDIUM_MS` (process table,
  GPUs, sensors, mount space) and restart the daemon (`3pm update`).
  The fast tier is a few ms of reads; the medium one forks and walks,
  so keep it ≥ a few seconds on busy boxes.

</document>

<document url="https://3pm.edgecr4ft.net/docs/app-metrics.md" title="Application metrics: all SDKs">

# Application metrics and events: the SDKs

What a process **started by 3pm** reports about itself — pm2's
`io.metric` / `io.counter` / `io.histogram` / `io.notifyError`, for any
language and any number of processes per app. Four SDKs with the same
vocabulary and the same behavior, in `packages/`:

| Language | Package | Install |
|---|---|---|
| Node ≥ 18 / Electron | `@3pm/api` (`io`, plus `@3pm/api/renderer`) | `npm install https://3pm.edgecr4ft.net/sdk/node/3pm-api-0.64.0.tgz` — `packages/node` |
| Python ≥ 3.9 | `threepm-io` (`threepm_io`, standard library only) | `pip install --index-url https://3pm.edgecr4ft.net/simple/ threepm-io` or `pip install -e packages/python` |
| C++17 | `threepm_io.hpp` (header-only, POSIX, no JSON library) | `FetchContent` on `https://3pm.edgecr4ft.net/sdk/cpp/threepm-io-cpp-0.64.0.tar.gz`, `add_subdirectory(packages/cpp)` → `threepm::io`, or copy `include/` |
| Rust ≥ 1.88 | `threepm-io` (`serde` + `serde_json`, std threads) | `cargo add threepm-io --registry threepm`, the registry being `sparse+https://3pm.edgecr4ft.net/sdk/cargo/` in `.cargo/config.toml` — `packages/rust` |

The four are published on `https://3pm.edgecr4ft.net` only (not on npm,
PyPI or crates.io) by `just publish-sdk`; every version, its files and
their sha256: `https://3pm.edgecr4ft.net/sdk/sdk.json`.

Read them: `3pm show <app>` (section "Custom metrics", approximate
quantiles for histograms), `3pm show <app> --json` (`app_metrics`),
`3pm events -f app`. On a fleet: the agent and the backend
(`docs/backend-agent/`). Storage model, bounds and the wire:
`docs/metrics.md` "Application metrics" and `docs/protocol.md` "Control
channel".

## How it works

Nothing is injected into your process. The SDK reads three variables the
daemon puts in every child's environment — `THREEPM_CTL_SOCK`,
`THREEPM_ID`, `THREEPM_GEN` — connects to the control socket in the
background, says `hello` with your `source`, then sends its **whole
registry once per second** (`flush_ms`) and every event as it happens.
Counters are cumulative and histograms are cumulative buckets, so a lost
batch loses nothing: the next one carries the same totals.

Without those variables (a plain `node app.js`, a unit test) the SDK is
**disabled**: every call is a no-op, no socket, no thread, no timer.

A `source` names one process within the app (`^[a-z0-9][a-z0-9_-]{0,31}$`;
`electron-main`, `cv-engine`, `plugin-host`…) and becomes the `src`
label. It is unique per instance: a process that restarts reconnects
under the same source and replaces its predecessor; its counters restart
from zero (a reset, which rates absorb). The same goes for two live
processes that say hello under one source: they evict each other, so
give each process its own. The Node SDK picks a distinct default when
you name none: `main` for the process 3pm spawned, `worker-<pid>-<threadId>`
in a worker thread (thread ids restart in every process), `pid-<pid>` in a process forked with an IPC channel
(`child_process.fork`, a cluster worker) or started by the process that
holds `main` (it leaves `THREEPM_IO_MAIN=<id>.<gen>.<pid>` in the
environment its children inherit; a child started before that call, by
a non-Node parent, cannot be told apart — name it). Two such processes
that both claim `main` settle it: the one whose hello is refused falls
back to `pid-<pid>` once, and that refusal never counts toward the SDK
giving up.

## The surface

| Call | Semantics |
|---|---|
| `init({source, flush_ms = 1000, on_shutdown})` | Connect in the background; idempotent. `on_shutdown` runs when the daemon asks for a graceful stop (`shutdown_with_message`) |
| `gauge(name, {unit, description, labels})` → `.set(v)` | Last value |
| `counter(name, …)` → `.inc(n = 1)` | Cumulative since process start, never decremented; rates at read time |
| `meter(name, …)` → `.mark(n = 1)` | pm2's meter: the same counter |
| `histogram(name, {…, scale = 1})` → `.observe(v)` | Base-2 exponential buckets (below); `count`/`sum` cumulative; quantiles at read time, so a fleet can add them up |
| `.labels({k: v})` on any of the three | A child series with those labels added (prom-client style). Same name + labels = same object; cache the child, the lookup validates the labels each time. On the `overflow="true"` fold it returns the fold |
| `metric(name, fn, {…})` | A gauge read by calling `fn` at every flush — in the flush thread (Python, C++, Rust) or on the event loop (Node). A name is either a metric or a callback, never both |
| `event(name, {level, attrs, exc, fp})` | Sent at once; queued (64) while the connection comes up |
| `notify_error(err, {attrs, name})` | An `error` event carrying `exc.{type, message, stacktrace}` |
| `flush({wait_ms})`, `shutdown()` | `flush` is non-blocking unless `wait_ms` is given, then waits for the connection and the bytes (a short-lived script); `shutdown` does the same wait, then closes. Called from the SDK's own thread (a callback, `on_shutdown`) neither waits |

Names: `^[a-z][a-z0-9_.]{0,63}$`, dots for levels (`http.latency`,
`cv.xruns`), the unit **not** in the name (`http.latency` + `unit: "ms"`,
not `http.latency_ms`). Units are UCUM (`ms`, `s`, `By`, `1`, `%`, `Cel`),
descriptions ≤ 128 **bytes** (every cap here is in UTF-8 bytes, like the
daemon's; the SDKs cut on a character boundary and drop a unit or a
description holding a control character), both sent with every batch and
kept in the catalogue. The daemon announces `THREEPM_CTL_CAPS=sticky_meta`
to its apps: an entry without `u` or `d` then leaves the series' stored
ones unchanged, and an SDK that reads the capability sends them once per
series and connection (the Rust SDK since 0.37).

### Labels

Up to **3 keys** per metric (`^[a-z][a-z0-9_]{0,31}$`), values ≤ 64
bytes without control characters (numbers and booleans are stringified
the JSON way: `200` → `"200"`, `1.0` → `"1"`, `true` → `"true"`); the
twenty-three keys the fleet sets itself are refused (`cluster`, `node`,
`app`, `instance`, `ns`, `src`, `overflow`, `vmrange`, `le`, `hostname`,
`os`, `arch`, `agent_version`, `mount`, `window`, the host dimensions
`core`, `iface`, `blockdev`, `sensor`, `gpu`, `array`, and the process
tree's `proc`, `pid`). A
metric with an invalid label set is refused **by the SDK** (a silent
no-op series; `THREEPM_IO_DEBUG=1` says why).

Past **50 label sets** on one metric, the SDK folds the excess into a
single series labelled `overflow="true"` (the OpenTelemetry
`otel.metric.overflow` model): counters and histograms add up, gauges keep
the last value — sums stay exact, cardinality stays bounded, memory stays
bounded in the SDK and in the daemon. Design labels accordingly: a
`status` or a `route` pattern, never a user id.

### Histograms

Exponential buckets, base 2, the OpenTelemetry `ExponentialHistogram`
layout restricted to positive values: at `scale = s` bucket `i` covers
`(2^(i/2^s), 2^((i+1)/2^s)]`, upper bound inclusive (a value exactly on a
boundary belongs to the bucket below), `0` and subnormals in a zero
bucket, NaN/infinities/negatives ignored. The default `scale = 1` (√2
factor, ~6.6 buckets per decade, ±19 % on a quantile, ~27 buckets over
four decades) fits the 1 000-series budget with dozens of label sets;
`scale = 2` or `3` for one key metric that needs ±9 % or ±4 %.

A histogram carries at most **128 non-empty buckets** (128 doublings at
scale 0, 64 at scale 1, 8 at scale 4). Node, Python and Rust do what the
OpenTelemetry SDK does when a 129th bucket would open: they lower the
scale by one, merging bucket `i` into `⌊i/2⌋` (exact on this grid; the
daemon sees the new `s` and replaces the buckets), so a scale-4 histogram
fed a wide range ends up at the scale that fits — quantile error grows,
nothing is refused. C++ keeps its `observe` lock-free with a fixed array
of 128 slots at the requested scale, centred on 1 unless `min_exp` says
otherwise (scale 1: `[2^-32, 2^32)`, scale 4: `[2^-4, 2^4)`); a value
outside the window lands in the edge slot, so widen the window (lower
the scale or set `min_exp`) for a metric with another range.

The index is derived from the double's exponent and mantissa against a
frozen table — no `log2`, no libm call — and all four SDKs reproduce
`threepm_protocol::exphist::classify` **bit for bit**: the shared vectors
`packages/exphist-vectors.json` (generated by the Rust tests) are replayed
by every suite. Changing the grid is a `PROTOCOL_VERSION` bump.

Observe **per event** (a request, an audio callback), not per sample.
In C++ `count`, `sum` and the buckets are read without a lock at flush
time: a batch is eventually consistent, never exactly. A counter or a
sum that would overflow to infinity ignores the observation.

### Events

The OpenTelemetry log-record shape: `name` (same grammar as metrics),
`level` in `debug`/`info`/`warn`/`error`/`fatal` (`severity_number`
5/9/13/17/21), flat scalar `attrs` (≤ 16 keys, strings ≤ 256; no nested
objects — the log store indexes flat fields), an optional exception
(`type` ≤ 128, `message` ≤ 1 KiB, `stacktrace` ≤ 8 KiB; longer values are
cut, not refused) and an optional fingerprint `fp` (≤ 64). Without `fp`
the daemon derives one from the name, the exception type and the first
three frames of the trace.

The daemon stamps the time, replaces the instance's `cwd` and `$HOME` in
the trace by `.` and `~`, and rate-limits **per fingerprint** (1/s, burst
10) and per instance (10/s, burst 50): a crash loop shows up as
`repeat = N` on the next event of that fingerprint that passes, never as
a flood. What you put in a label, an attribute or an exception message
crosses the fleet link as is — no secrets there.

### Behavior you can rely on

- **Never blocking, never throwing.** Every call returns immediately.
  The outbound buffer is bounded (64 KiB): when the daemon does not read,
  the batch is dropped and the next one carries the totals. Nothing is
  logged above debug (`THREEPM_IO_DEBUG=1` to trace). (Node: the actions
  declaration and replies travel on their own budget, under the daemon's
  1 MiB line cap, so metrics can never crowd them out; `io.action` with a
  `3pm:` name is the one call that throws.)
- **Never keeps the process alive.** Sockets and timers are unref'd
  (Node), threads are daemons (Python) or joined by the destructor (C++).
  In Rust nothing holds a process up at all — `main` returning is the end,
  so a short-lived program flushes before returning and a worker that only
  answers actions ends its `main` with `io().serve()`.
- **Reconnection** 1 s → 30 s with doubling when the daemon goes away.
- **Gives up after 5 consecutive refusals** of `hello` — a connection the
  daemon closes right away means the process outlived its incarnation
  (a stale `THREEPM_GEN`: a `setsid` helper, a process that survived a
  restart). The SDK stops trying for good.
- **Bounds mirror the daemon's** (`docs/metrics.md`): 200 names, 3 label
  keys, 50 label sets then overflow, 128 buckets per histogram, 256
  entries and 32 KiB per line (longer registries are split over several
  lines, each under the outbound budget). What the SDK sends is never
  refused: `app_metrics_rejected` on the record stays at 0, and a line
  the daemon cannot parse is skipped by the daemon, never fatal.
- **Advanced knobs**, mainly for tests, same names in the four:
  `reconnect_min_ms`, `reconnect_max_ms`, `max_buffer_bytes`,
  `max_refusals`, `refusal_window_ms`.

## Node — `@3pm/api`

```js
const { io } = require('@3pm/api');
io.init({ source: 'worker', builtins: true, on_shutdown: () => server.close() });

const requests = io.counter('http.requests', { labels: { method: 'GET' } });
const latency  = io.histogram('http.latency', { unit: 'ms' });
const temp     = io.gauge('room.temp', { unit: 'Cel', description: 'studio temperature' });
io.metric('queue.depth', () => queue.length);

requests.labels({ status: res.statusCode }).inc();
latency.observe(elapsedMs);
temp.set(21.5);
io.event('deploy', { level: 'warn', attrs: { version: '1.2' } });
io.notifyError(err, { attrs: { where: 'worker' } });
```

`builtins: true` registers ordinary opt-in metrics, read once per flush
(a registry collector takes each snapshot once) or recorded as things
happen:

| Metric | Kind, unit | What |
|---|---|---|
| `nodejs.eventloop.delay` | histogram, ms | lag of a 100 ms timer, every sample |
| `nodejs.eventloop.delay.p99`, `.max` | gauge, ms | `monitorEventLoopDelay` (10 ms resolution) over the flush window; a window without a sample keeps the last value |
| `nodejs.eventloop.utilization` | gauge, 1 | busy time over wall time since the previous flush: saturation, which lag alone does not tell from OS throttling. Absent where libuv does not account the loop's time (Electron's main process runs Chromium's loop) |
| `nodejs.gc.duration{kind}` | histogram, ms | every GC pause, `kind` = `minor` / `major` / `incremental` / `weakcb` |
| `nodejs.heap.used`, `.committed`, `.limit` | gauge, By | V8 heap in use, committed (`heapTotal`), and its limit (out of memory past it) |
| `nodejs.memory.external`, `.array_buffers` | gauge, By | outside the V8 heap: C++-bound objects, (Shared)ArrayBuffers — Buffers, audio, images |
| `nodejs.handles.active` | gauge, 1 | live libuv handles and requests |
| `process.cpu.utilization` | gauge, 1 | CPU time over wall time since the previous flush (1 = one core) |
| `process.page_faults.major` | counter | page faults served from disk (memory pressure) |
| `process.context_switches.involuntary` | counter | preemptions by the kernel (CPU contention) |

`builtins: { heap_spaces: true }` adds `nodejs.heap.space.used{space}`
(the V8 heap in use per space: `old_space`, `new_space`,
`large_object_space`…, about 13 series). What a runtime lacks is not
registered. No built-in name ends in `total`, `count`, `sum` or
`bucket`: the fleet backend gives counters and histograms those
suffixes, and a gauge named so would collide — name your own gauges the
same way (`heap.committed`, not `heap.total`). Callback metrics and collectors run on the event loop at
flush; nothing keeps a short-lived script alive.

**Actions.** An app exposes functions to `3pm trigger` and to the fleet
backend (pm2's `io.action`):

```js
io.action('cache.flush', { description: 'drop the cache', timeout_ms: 5000 }, async (params, ctx) => {
  return { flushed: await cache.flush(params.all) };   // JSON, ≤ 512 KiB; a throw or a rejection is the failure
});
io.action('dump', async (params, ctx) => {             // a big result: a file
  await fs.promises.writeFile(ctx.file + '.tmp', JSON.stringify(state));
  return { bytes: await ctx.commitFile('json') };      // rename into place, name the extension
});
const off = io.action('reset', { remote: false }, () => reset());  // `3pm trigger` only; `off()` withdraws it
```

`name` is `^[A-Za-z0-9_.:-]{1,64}$`, 32 per source; a `3pm:` name
throws (the SDK's prefix); `remote` defaults to `true`; `timeout_ms`
(default 30 s, ceiling 10 min) bounds every call. The whole list is
declared right after each `hello` (`{"msg":"actions","list":[…]}`,
ahead of any queued event) and whenever it changes, once per tick, and
sent again every flush until a write of it succeeded. A call arrives as
`{"msg":"action","id","name","params","timeout_ms","file"}` and is
answered with `{"msg":"action_reply","id","ok","data?","error?",
"file?"}`; `ctx` carries `id`, `name`, `timeout_ms`, `file` (a path the
daemon prepared) and `commitFile(ext)`. Lone surrogates in a result,
an error message or a description become U+FFFD (the daemon would
refuse the line); a reply that cannot be written (over the line cap)
becomes a short `ok: false`, so the call fails instead of timing out.
Past the call's deadline `commitFile` deletes the `.tmp` and throws —
the daemon has already given up. A handler still running is not called
again (`busy`); a hung one is released at its timeout. `init()` declares
`3pm:profile` (a CPU or heap sampling profile of the process through
`node:inspector`, `{kind: cpu|heap, duration_ms: 1000..60000 (10 000),
sampling}`) unless `profile: false`: params are validated, not coerced
(an absent field takes its default), and a call whose timeout leaves
less than 15 s after the sampling fails before anything is sampled
(`3pm profile` asks for the sampling + 20 s). `3pm trigger <app>` lists
what is declared; `docs/protocol.md` has the bounds and error codes.

**Electron.** A renderer has no Unix socket, so its metrics travel over
IPC. Main process:

```js
const electron = require('electron');
const { io } = require('@3pm/api');
io.init({ source: 'electron-main', builtins: true, on_shutdown: teardown });
const bridge = io.ipcBridge(electron.ipcMain); // one control connection per renderer source
io.electron(electron);                         // Chromium processes, power, crashes, hangs
```

`io.electron(electron)` adds what only the main process sees (the
module is passed in, never required; `powerMonitor` waits for `ready`):

| Metric | Kind, unit | What |
|---|---|---|
| `electron.processes{type}` | gauge, 1 | Chromium processes by type (`Browser`, `Tab`, `GPU`, `Utility`…), one `app.getAppMetrics()` per flush |
| `electron.process.cpu{type}` | gauge, 1 | their CPU (1 = one core); a type gone drops to 0 |
| `electron.process.memory{type}` | gauge, By | their working set |
| `electron.power.on_battery` | gauge, 1 | 1 on battery, 0 on AC |
| `electron.power.thermal_state` | gauge, 1 | 1 nominal, 2 fair, 3 serious, 4 critical (where the OS reports it) |
| `electron.power.speed_limit` | gauge, % | CPU speed limit, once the OS announced one |
| `electron.process.gone{type,reason}` | counter | processes gone (`crashed`, `oom`, `killed`, `clean-exit`…) |
| `electron.unresponsive` | counter | times a window stopped responding |

and events: `electron.process_gone` (`error`, `info` for a clean exit;
type, reason, exit code), `electron.unresponsive` (`warn`) and
`electron.responsive` (`duration_ms`), `electron.power.suspend` /
`resume` / `on_battery` / `on_ac`, `electron.power.thermal_state`
(`warn` from `serious`), `electron.power.speed_limit` (`warn` under
100 %). A window is named by its id and kind, never its URL or title.
It returns an idempotent `stop()`, also run by `shutdown()`.

Renderer (or preload):

```js
const { createRendererIo } = require('@3pm/api/renderer');
const rio = createRendererIo(ipcRenderer, { source: 'renderer', builtins: true });
rio.event('view.change', { attrs: { view: 'main' } });
```

The renderer's `builtins: true` registers what the runtime offers, with
no Node builtin module: `renderer.eventloop.delay` (histogram, 100 ms
timer), `renderer.longtask.duration` (histogram, `PerformanceObserver`
long tasks), `renderer.heap.used` / `.committed` / `.limit` (Electron's
`process.getHeapStatistics()`, else Chromium's coarse
`performance.memory`), `renderer.blink.allocated` / `.committed` (Blink's
DOM, layout and style memory), `renderer.cpu.utilization`, and
`renderer.dom.nodes` (re-counted every 30 s: a tree walk).

Same surface; `3pm:metrics` carries `{source, m}` every flush,
`3pm:event` one event, `3pm:actions` the renderer's declarations. The
bridge validates what arrives, re-sends the last picture of each
renderer every second, drops entries not refreshed for 120 s, and caps
the number of sources (16); a renderer naming itself `main` or the main
process's own source is ignored (its hello would evict the main
process). Only the main process should register `on_shutdown`. With an
`ipcRenderer` that has `on`, `rio.action()` works too: the bridge
forwards calls on `3pm:action` and takes the answer on
`3pm:action_reply` (no `ctx.file` in a renderer; `shutdown()` removes
the listener). Several windows may share a source: each window's
declarations are its own (another window cannot withdraw them), a call
goes to the window that declared the action — the latest, when several
declared one name — and only that window's reply is taken; a window's
actions end when it declares again, navigates or reloads (main frame,
new document), crashes or is destroyed. The bridge declares
`3pm:profile` for every renderer source whose `webContents` has a
debugger and runs it over CDP (`Profiler` / `HeapProfiler`, each
command bounded to 15 s): only when exactly one live window holds the
source — with several, the call fails and asks for one source per
window — and never over a debugger already attached (DevTools open: an
immediate `debugger already attached`); it always detaches what it
attached. `3pm profile saw --src renderer`.

## Python — `threepm-io`

```python
import threepm_io as io

io.init(source="cv-engine")
xruns   = io.counter("cv.xruns", description="PortAudio over/underruns")
jitter  = io.histogram("cv.callback.jitter", unit="ms")
clients = io.gauge("cv.ws.clients")
io.metric("cv.queue.depth", lambda: len(queue))   # runs in the flush thread

xruns.inc()
jitter.observe(0.42)
clients.set(3)
io.event("midi.port_lost", level="warn", attrs={"port": "ES-9"})
try:
    open_device()
except Exception as e:
    io.notify_error(e, attrs={"device": "ES-9"})
```

Actions and calls, with a file in and a file out — the Node SDK's wire,
so either side may be the other language (full table:
`packages/python/README.md`):

```python
@io.action("segment", input={"max_bytes": 50_000_000, "ext": ["png", "jpg"]}, timeout_ms=60_000)
def segment(params, ctx):
    return model.run(ctx.input.path, params)      # its own thread; JSON ≤ 512 KiB, or ctx.commit_file(ext)
io.serve()                                        # block until 3pm stops the process

res = io.call("gpu-1", "vision", "segment", {"t": 0.7}, file="/tmp/capture.png")   # blocks; raises io.CallError
```

Not there yet: streams (`io.open`/`io.stream`), `input.stream`,
`3pm:profile`. `init()` without `source` is `main` (`pid-<pid>` for a
process it started), as in Node.

One daemon thread per instance writes and reads the socket; a second,
short-lived one per connection reads what the daemon sends. `metric()`
callbacks run in that thread — outside any asyncio loop, so read a plain
attribute or a `len()`, not a coroutine. `threepm_io.Io()` gives an
independent instance (another `source` in the same process). A
short-lived script should `io.flush(wait_ms=500)` before exiting.

## C++ — `threepm_io.hpp`

```cpp
#include <threepm_io.hpp>
using namespace threepm::io;

Io& io = instance();
InitOptions opts; opts.source = "plugin-host"; opts.on_shutdown = [] { request_stop(); };
io.init(opts);

auto late    = io.counter("host.late_callbacks");
HistogramOptions h; h.unit = "ms";
auto latency = io.histogram("host.callback.latency", h);
auto state   = io.gauge("host.state", {"", "0 idle, 1 loading, 2 running, 3 error", {}});

late.inc();                 // on the audio thread: atomic add, no lock, no allocation
latency.observe(0.42);      // exponent/mantissa lookup, no libm
EventOptions ev; ev.level = "warn"; ev.attrs.add("port", "ES-9");
io.event("midi.port_lost", ev);
io.notify_error("HostError", last_error, Attrs().add("plugin", name));
```

`set`, `inc` and `observe` are lock-free atomic operations on
preallocated storage (`observe` and `inc` add to a double by CAS: one
writer per series stays at a few nanoseconds, eight writers on the same
series cost microseconds — give each thread its own series). Creating a
metric or calling `.labels()` takes the registry mutex: do it at setup,
cache the handles, never on the audio thread. A histogram owns a fixed
array of 128 slots at its scale (window above). One `std::thread` per
instance does the I/O; `on_shutdown` and `metric()` callbacks run on it,
and may call `shutdown()`. The destructor flushes and joins.

## Rust — `threepm-io`

```rust
use serde_json::{json, Value};
use threepm_io::{io, Action, ActionCtx, Event, Init, Level, Metric};

io().init(Init::new().source("worker").on_shutdown(|| stop.store(true, Relaxed)));

let requests = io().counter("http.requests", Metric::new().label("method", "GET"));
let latency  = io().histogram("http.latency", Metric::new().unit("ms"));
let temp     = io().gauge("room.temp", Metric::new().unit("Cel"));
io().metric("queue.depth", Metric::new(), move || read.len() as f64);

requests.labels(&[("status", "200")]).inc(1.0);   // the step is explicit
latency.observe(elapsed_ms);
temp.set(21.5);
io().event("deploy", Event::new().level(Level::Warn).attr("version", "1.2"));
io().notify_error(&err, Event::new().attr("where", "worker"));

io().action("cache.flush", Action::new().timeout_ms(5_000),
    |params: Value, _: &ActionCtx| -> Result<Value, String> {
        Ok(json!({ "flushed": cache.flush(&params["prefix"]) }))
    });
io().serve();     // a worker whose only job is to answer actions
```

`io()` is the process's instance; `Io::new()` makes another. One thread
per instance writes and reads the socket, plus one per connection and one
per call in flight; `metric` callbacks and `on_shutdown` run on those
threads, so read a value there and do not block. `Series` is
`Clone + Send + Sync`: create the handles at startup and keep them.
Nothing returns a `Result` or panics on telemetry — a metric that cannot
exist comes back as an inert handle (`enabled() == false`).

Actions carry the same contract as Node's (`ctx.file` + `ctx.commit_file(ext)`
for a result too big for one line, `remote`, `timeout_ms`, `concurrent`,
`output_max_bytes`, a local `ctx.input`), with the handler's error type
being anything that prints. **Not implemented yet**: `io.call`, streams,
blob inputs and outputs (everything on `data.sock`), the built-in
`3pm:profile`, and Node's `builtins` runtime metrics. Differences,
bounds and the full surface: `packages/rust/README.md`.

## Writing your own client

The wire is plain NDJSON on `$THREEPM_CTL_SOCK` (`docs/protocol.md`,
"Control channel"): `{"msg":"hello","id":$THREEPM_ID,"gen":$THREEPM_GEN,
"source":"worker"}` first, then `{"msg":"metrics","m":[…]}` every second
and `{"msg":"event",…}` when something happens. `examples/metrics.py` is
the reference client in raw Python, histogram index included.

## Tests

`just sdks` runs the four suites — Node (`node --test`), Python
(`pytest`), C++ (`ctest`), Rust (`cargo test` + `clippy`) — each against a
fake control socket (hello and source, batches, events, reconnection and
backoff, the five-refusal give-up, the full outbound buffer, `shutdown`,
label overflow) and each replaying `packages/exphist-vectors.json`; it
also builds the C++ and Rust e2e clients.
`crates/threepm-cli/tests/e2e_io.rs` then starts them under a real daemon
and checks `show --json` and `events -f app`
(`cargo nextest run -p threepm-cli --test e2e_io`, part of `just e2e`);
`e2e_actions.rs` calls the Rust client's actions with `3pm trigger`.

</document>

<document url="https://3pm.edgecr4ft.net/docs/sdk/node.md" title="Node.js SDK: @3pm/api">

# @3pm/api

Node.js client for the [3pm](https://github.com/Unitech/3pm) process manager,
plus `io`, the application-metrics SDK (see below).
Same shape as pm2's programmatic API — `connect`, `start`, `list`, `stop`,
`restart`, `delete`, `describe`, `launchBus` — but promise-based, with zero
dependencies. It talks straight to the daemon's Unix sockets
(`rpc.sock` for commands, `pub.sock` for events) and auto-spawns the daemon
when none is running.

```console
npm install https://3pm.edgecr4ft.net/sdk/node/3pm-api-0.64.0.tgz
```

Published on <https://3pm.edgecr4ft.net> only, not on the npm registry:
every version and its sha256 in <https://3pm.edgecr4ft.net/sdk/sdk.json>.

Requires Node ≥ 18 and a `3pm` binary on `PATH` (or `THREEPM_BIN`) for
auto-spawn — not needed when a daemon is already up.

## Quickstart

```js
const { ThreePM } = require('@3pm/api');

const pm = await ThreePM.connect();          // spawns the daemon if needed

await pm.start({ script: 'app.js', name: 'api', instances: 2, env: { PORT: 3000 } });
console.table((await pm.list()).map(({ id, name, status, pid, restarts }) => ({ id, name, status, pid, restarts })));

const bus = await pm.launchBus({ process: 'api' });
bus.on('log:out',        (e) => process.stdout.write(`${e.name}[${e.instance}] ${e.line}\n`));
bus.on('log:err',        (e) => process.stderr.write(`${e.name}[${e.instance}] ${e.line}\n`));
bus.on('process:exit',   (e) => console.log('exit', e.exit_code, e.exit_signal, e.stderr_tail));
bus.on('process:online', (e) => console.log('online', e.name));
bus.on('daemon:shutdown',(e) => console.log('daemon going down:', e.reason));

await pm.restart('api');
await pm.stop('api');
await pm.delete('api');
pm.disconnect();                              // closes the rpc socket and every bus
```

## API

`new ThreePM({ home?, bin?, autoSpawn? = true, timeout? = 30000 })` — or `await ThreePM.connect(opts)`.

`home` resolves like the CLI (`resolvePaths`): the option, else `$THREEPM_HOME`, else the home a root daemon
hosts for this account (`/var/lib/3pm/users/<user>`, marked by a `hosted` file — `hostedHome()` returns it or
`null`), else `~/.3pm`. On a hosted home the client never spawns a daemon: when the root daemon is down it
throws `daemon_unreachable` with `err.hosted === true`. `start()` accepts `user` (run the app as a declared
account, root daemon) and `persist: false` (kept out of `dump.json`, never relaunched at boot); a hosted
account's daemon-level calls (`save`, `killDaemon`, `resurrect`, `internals`, `hostMetrics`) fail with
`forbidden`.

| Method | Wire request | Returns |
|---|---|---|
| `connect()` | `ping` (checks `protocol === 2`) | `this` |
| `ping()` | `ping` | `{pid, version, protocol}` |
| `start(opts \| opts[], {ifChanged?})` | `start` | `ProcessRecord[]` |
| `scale(name, n)` | `scale` | `ProcessRecord[]` |
| `stop(target)` | `stop` | `ProcessRecord[]` (returns once dead) |
| `restart(target, {toggleWatch?})` | `restart` | `ProcessRecord[]` |
| `delete(target, {flush?})` / `del` | `delete` | `ProcessRecord[]` |
| `reset(target)` | `reset` | `ProcessRecord[]` |
| `sendSignal(sig, target, {group?})` | `send_signal` | `ProcessRecord[]` |
| `list()` / `ls()` | `list` | `ProcessRecord[]` |
| `describe(target)` / `show()` | `describe` | `ProcessRecord[]` (every instance) |
| `save()` / `resurrect()` | `save` / `resurrect` | `{path, count}` / `ProcessRecord[]` |
| `killDaemon()` | `kill_daemon` | — (buses stay open to see `daemon:shutdown`) |
| `internals()` / `hostMetrics()` | `internals` / `host_metrics` | growable snapshots |
| `flushLogs(t?)` / `rotateLogs(t?)` / `reloadLogs()` | `*_logs` | touched file paths |
| `request(cmd, fields)` | anything | raw response — escape hatch for new requests |
| `launchBus(filter?)` | — | `EventBus` |

`target` is an id (`3` / `"3"`), a name (`"api"`), `"all"`, `{id}` or `{name}`.

### Start options

pm2 names, pm2 units: `script` (required; resolved against `cwd`), `name`, `cwd`, `args`
(string or array), `interpreter` (inferred from the extension — `.js/.ts→node`,
`.py→python3`, `.rb→ruby`, `.sh→bash`, `.pl→perl` — then resolved to an absolute path;
`'none'` = direct exec), `env`, `instances` (`'max'` = one per CPU), `autorestart` /
`restart_policy`, `max_memory_restart` (`"200M"`), `min_uptime`, `kill_timeout`,
`restart_delay`, `listen_timeout`, `wait_ready`, `watch` (`true`, path, list),
`ignore_watch`, `out_file`/`error_file`/`log_file` (`'NULL'` disables), `merge_logs`,
`log_date_format`, `stop_exit_codes`, `cron_restart`, `depends_on`, `max_restarts`,
`backoff_*`, `log_rotate`, … — the full list is in `index.d.ts` (`StartOptions`).
Unknown keys throw `invalid_spec` with a did-you-mean.

### Events

`launchBus({ subscribe?, process?, ids?, reconnect? })` sends the daemon an
`event_filter` so unwanted events never cross the socket:

- `subscribe`: globs over event names — `['process:*']`, `['log:err']`, `['process:exit', 'log*']`
- `process`: one app name; `ids`: process ids
- `reconnect: true | {delayMs, maxAttempts}`: re-subscribe after an EOF (e.g. across `3pm update`)

| Emitted as | Also as | Payload |
|---|---|---|
| `log:out`, `log:err` (from `stream`) | `log` | `{id, name, instance, line, stream, ts}` |
| `process:event` + `process:<kind>` (`start`, `online`, `exit`, `restart`, `stop`, `delete`, `errored`) | `process` | `{id, name, kind, exit_code?, exit_signal?, stderr_tail?}` |
| `log:rotated` | `log_rotated` | `{id, name, instance, path}` |
| `lagged` | — | `{missed}` — the subscriber fell behind |
| `daemon:shutdown` | `daemon_shutdown` | `{reason: 'kill' \| 'signal' \| 'home_removed'}` |
| `event` | — | every decoded event, unknown kinds included |
| `connect`, `reconnect`, `close({reason})`, `error` | — | bus lifecycle |

A `close` with `reason: 'eof'` (no `daemon_shutdown` first) is an unannounced daemon death.

### Errors

Every failure is a `ThreePMError` with `.code`: the daemon's codes (`not_found`,
`invalid_spec`, `spawn_failed`, `unknown_request`, `shutting_down`, `internal`) or
client-side `daemon_unreachable`, `protocol_skew` (run `3pm update`), `timeout`,
`disconnected`, `protocol`. Codes are exported as `CODES`.

## Application metrics and events: `io`

The other half of the package: what a process **started by 3pm** reports
about itself (pm2's `io.metric`/`io.counter`/`io.notifyError`, for 3pm).
Zero dependencies, never blocking, never throwing; outside 3pm (no
`THREEPM_CTL_SOCK` in the environment) every call is a no-op.

```js
const { io } = require('@3pm/api');
io.init({ source: 'worker', builtins: true });      // one connection per process of the app

const requests = io.counter('http.requests', { labels: { method: 'GET' } });
const latency  = io.histogram('http.latency', { unit: 'ms' });
const temp     = io.gauge('room.temp', { unit: 'Cel', description: 'studio temperature' });
io.metric('queue.depth', () => queue.length);        // read at every flush

requests.labels({ status: 200 }).inc();
latency.observe(12.5);
temp.set(21.5);
io.event('deploy', { level: 'warn', attrs: { version: '1.2' } });
io.notifyError(err, { attrs: { where: 'worker' } });  // an `error` event with the stack
```

Then `3pm show <app>` (section "Custom metrics") and `3pm events -f app`.
Full reference, bounds and the three languages: `docs/app-metrics.md` in
the 3pm repository.

| Call | Notes |
|---|---|
| `io.init({ source?, flush_ms? = 1000, builtins?, profile?, on_shutdown?, keep_alive? = true, connect_wait_ms? = 5000 })` | Connects in the background; idempotent. While the application declares an action of its own (`io.action`, `io.stream`; the SDK's `3pm:profile` does not count) the process is held up to serve it, so a worker that only answers calls needs no `setInterval`: `keep_alive: false` opts out, and the daemon's `shutdown` lets go. A call or a session placed before the connection is up waits for it up to `connect_wait_ms`; `await io.ready()` says when it is (`false` after the wait, or when not under 3pm). `source` defaults to a name unique per connection: `main` for the process 3pm spawned, `worker-<pid>-<threadId>` in a worker thread, `pid-<pid>` in a forked child (`child_process.fork`, cluster) or a child of the process holding `main`; two processes that both look spawned (`sh -c 'node a & node b'`) both claim `main`, and the one refused falls back to `pid-<pid>` — two connections of one instance under one source replace each other. The built-in `3pm:profile` action is declared on `main` only (a bare `3pm profile <app>` is never ambiguous); `profile: true` adds it on any source, `profile: false` withholds it. `builtins: true` adds the event loop (`nodejs.eventloop.delay` histogram, `.p99`/`.max`, `.utilization`), GC pauses (`nodejs.gc.duration{kind}`), memory (`nodejs.heap.used`/`.committed`/`.limit`, `nodejs.memory.external`/`.array_buffers`), `nodejs.handles.active`, `process.cpu.utilization`, and the counters `process.page_faults.major`, `process.context_switches.involuntary`; `builtins: { heap_spaces: true }` adds `nodejs.heap.space.used{space}` |
| `io.electron(require('electron'))` | Electron main process: `electron.processes`/`.process.cpu`/`.process.memory` by `type`, `electron.power.*` (battery, thermal state, speed limit), counters and events for crashed processes and hung windows, sleep/wake. Returns `stop()` |
| `createRendererIo(ipcRenderer, { source?, builtins? })` (`@3pm/api/renderer`) | Renderer/preload side of `io.ipcBridge(ipcMain)`. `builtins: true` adds `renderer.eventloop.delay`, `renderer.longtask.duration`, `renderer.heap.*`, `renderer.blink.*`, `renderer.cpu.utilization`, `renderer.dom.nodes` (what the runtime offers) |
| `gauge` / `counter` / `meter` / `histogram(name, { unit?, description?, labels?, scale? })` | Same name + labels = same object. Counters are cumulative; histograms are base-2 exponential (`scale` 0..4, default 1) and lower their scale by one (buckets merge in pairs) whenever a 129th bucket would open, so an entry always fits the daemon's cap. `.labels({...})` derives a child — cache it rather than calling it per request; past 50 label sets a metric folds into `overflow="true"`, and the fold's `.labels()` returns the fold |
| `metric(name, fn, opts?)` | Gauge by callback, evaluated on the event loop at flush. Callbacks and metrics share the 200-name budget: a name is one or the other |
| `event(name, { level?, attrs?, exc?, fp? })` | Sent at once (queued while the connection comes up) |
| `notifyError(err, { attrs?, name? })` | `error`-level event carrying `exc.{type,message,stacktrace}` |
| `action(name, { description?, remote? = true, timeout_ms? }, handler)` | Expose `handler(params, ctx)` to `3pm trigger <app> <name> [params]` and, unless `remote: false`, to the fleet backend. What it returns (JSON, ≤ 512 KiB, ≤ 64 levels deep) is the result; past either bound the call fails with code `too_large` — a bigger one is a file: write `ctx.file + '.tmp'` then `await ctx.commitFile(ext)` (past the call's timeout it deletes the `.tmp` and throws). Whatever the handler throws is the failure, even a value with no readable message. Returns `unregister`. A `3pm:` name throws (the SDK's prefix). At most 32 actions per source, `3pm:profile` included. `3pm:profile` (CPU/heap sampling profile, `{kind: cpu\|heap, duration_ms: 1000..60000, sampling}`) is declared on `main` by default; invalid params, or a timeout leaving less than 15 s after the sampling, fail the call before anything is sampled |
| `flush({ wait_ms? })`, `shutdown()` | The whole registry is sent every `flush_ms` anyway. `flush` resolves `true` when every batch was written (and, with `wait_ms`, left the process); `shutdown` is safe to call twice |

Strings are bounded in UTF-8 bytes, as the daemon measures them: label
values ≤ 64 bytes (else the series is refused), unit/description cut to
16/128 bytes, event attributes to 256, exception fields to 128/1024/8192.
Cuts land on character boundaries and lone surrogates become U+FFFD, so
nothing sent is ever refused. Everything else that goes wrong (bad name,
reserved label, wrong kind for a known name) returns a no-op metric
(`enabled: false`) and logs under `THREEPM_IO_DEBUG=1`.

`init` also accepts transport knobs, advanced and mainly for tests:
`reconnect_min_ms`, `reconnect_max_ms`, `max_buffer_bytes` (outbound
budget past which metrics and events are dropped, 64 KiB; the actions
declaration and replies have their own and are never crowded out),
`max_refusals`, `refusal_window_ms`. `io.stats` is a copy of `{ sent,
dropped, connects, closes, refusals }` (metrics and events lines).

Then `3pm trigger <app>` lists the actions, `3pm trigger <app> <name>
'{"json": true}'` runs one on every instance, `3pm profile <app>` takes a
profile and prints the hot frames; every call is a `proc_action` event
(`3pm events -f proc_action`).

Electron: the renderer has no Unix socket. In the main process,
`io.init({ source: 'electron-main' }); io.ipcBridge(ipcMain, { flush_ms?,
max_sources? = 16 })`; in a renderer, `const { createRendererIo } =
require('@3pm/api/renderer'); const rio = createRendererIo(ipcRenderer,
{ source: 'renderer' })` — same surface, one control connection per
renderer source. The renderer entry loads no Node builtin; the main
process re-validates every relayed entry and bounds what it keeps per
source; a renderer cannot take an automatic name (`main`, `pid-*`,
`worker-*`) or the main process's own source. A renderer's
`rio.action()` needs an `ipcRenderer` with `on` (calls come back on
`3pm:action`, acknowledgements on `3pm:actions_ack`: the declaration is
sent again until the bridge has it, so one made before `ipcBridge()`
still arrives; `shutdown()` withdraws the actions and removes the
listeners). Several windows may share a source: each keeps its own
actions — a call goes to the window that declared it (the latest, when
several declared one name) and only its reply is taken — until it
declares again, commits a navigation (a reload too; the next page is
asked for its own), crashes or is destroyed; a navigation that never
commits (prevented, a download, an HTTP 204) keeps them. The bridge profiles a renderer over
`webContents.debugger` under `3pm:profile` when exactly one live window
holds the source (give each window its own source to profile it), and
fails at once when the debugger is already attached (DevTools open).

## Fleet calls: `io.call`, `io.open`, `io.stream`

An action can take bytes and answer with bytes, on this machine or on
another node of the bucket. Declare what it takes:

```js
io.action('infer', { timeout_ms: 60_000, input: { max_bytes: 100e6, ext: ['jpg', 'png'] } },
  async (params, ctx) => {                    // ctx.input: { path, bytes, ext, sha256 } — fetched and checked
    return await model.run(await fs.promises.readFile(ctx.input.path), params)
  })

io.action('upscale', { input: { max_bytes: 50e6 } }, async (params, ctx) => {
  await fs.promises.copyFile(await render(ctx.input.path), `${ctx.file}.tmp`)
  await ctx.commitFile('png')                 // with an output ticket: PUT on the agent's socket
  return { seed: params.seed }
})
```

`input.max_bytes` (≤ 8 GiB) refuses a bigger call before a byte travels;
`ext` narrows what it accepts. The bytes arrive as a file the daemon
named, fetched from the agent's socket, cut at the size the call
declared and checked against its digest — a mismatch fails the call
rather than handing over half a file. The SDK then sends
`action_started`, so the transfer is not the handler's timeout, and
deletes the file once the reply is out. `concurrent` (1 by default, at
most 16) says how many calls of that action may run at once; past it a
call is `busy`.

`output.max_bytes` (an integer, 1 byte to 256 MiB) bounds the file the
handler commits: past it `ctx.commitFile` throws `too_large` naming both
sizes, deletes the `.tmp`, and nothing is sent. On the fleet, a caller
that asks for the file reserves that much of its blob quota while the
call runs — 256 MiB when the action declares nothing — so a small bound
lets many calls run at once:

```js
io.action('screenshot', { output: { max_bytes: 2_000_000 } }, async (params, ctx) => {
  await capture(`${ctx.file}.tmp`)
  await ctx.commitFile('png')
})
```

Any other `output` value leaves the action unregistered (logged), since
the daemon would refuse the source's whole list over it.

> **Not in `@3pm/api` 0.1.3** (the published SDK): `output` comes with
> the next published version (`[Unreleased]` in `CHANGELOG.md`); 0.1.3
> ignores it without a word — no check in `ctx.commitFile`, 256 MiB
> reserved. For the bound to reach the fleet, the node's daemon and
> agent must be of that release too, and the backend: an older one
> drops it and reserves 256 MiB, as before.

Place one:

```js
const res = await io.call('gpu-1', 'vision', 'infer', { threshold: 0.7 }, { file: '/tmp/capture.jpg' })
// { call_id, ok: true, data: { count: 3 }, duration_ms: 840, results: [...] }
const up = await io.call('gpu-1', 'vision', 'upscale', {}, { file: 'in.png', output: '/tmp/out.png' })
// up.file → { path: '/tmp/out.png', bytes, ext: 'png' }
await io.call(null, 'vision', 'infer', {}, { file })    // same machine: neither agent nor backend
```

`node` is a node id (`3pmn_…`) or a unique hostname; `null` is this
machine, and the file is then taken by path — nothing leaves it.
`'self'` is this machine through the fleet, by its own node id. `io.node`
is that node as `3pm link` recorded it (`{ id, cluster_id, server,
linked_at_ms }` from the agent's `link.toml`, `null` when the host is not
linked); name it by `io.node.id`, never by `os.hostname()`, which need not
match the hostname it enrolled under. One
instance answers (0 by default; `instance: null` asks every one, and is
refused when the call carries bytes). `opts`: `file`, `ext`, `sha256`,
`bytes`, `output` (a path, or `true` to be told the blob only),
`instance`, `source`, `timeout_ms`.

### One pass over a large input, not three

A hundred megabytes of input is mostly SHA-256 in JavaScript and a copy
to disk, not the road: `io.call` reads the file once to find its digest,
the callee writes it once into a staging file, and a handler that wants
the bytes reads it a third time. Two opt-ins remove two of those passes.
Both are off by default and neither changes what goes on the wire.

```js
// The caller already holds the digest — from a manifest, from whatever
// produced the file — so the SDK does not read the file to find it again.
await io.call('gpu-1', 'vision', 'infer', {}, { file, sha256, bytes })

// The handler reads the body as it arrives: nothing is staged on disk.
io.action('scan', { timeout_ms: 300_000, input: { max_bytes: 8e9, stream: true } },
  async (params, ctx) => {
    const h = crypto.createHash('sha256')
    for await (const chunk of ctx.input.stream()) h.update(chunk)
    return h.digest('hex')                       // only now are the bytes vouched for
  })
```

`sha256` and `bytes` go together; the size is checked against the file
(one `stat`, which catches a digest kept after the file changed under
it) and the digest is taken at your word — verifying it is the work
being skipped, and the callee checks it against what lands as ever.
Without `file`, or on a call to this machine (`node: null`, where the
daemon takes the file by path), they are ignored rather than refused.

`input.stream` changes three things worth knowing.

1. **`ctx.input.path` is `null` on a call from another node** — nothing
   was written. A call placed on this machine always names a path, and
   `stream()` there reads the daemon's own file, so one handler serves
   both without asking which it is.
2. **The size and the digest are checked at the end of the body**, not
   before the handler runs: a truncated or altered input throws out of
   the iteration. Nothing has vouched for the bytes until that iteration
   ends, so do not act on them before it does — a handler that `break`s
   early, or swallows the throw, has used bytes nobody checked.
3. **The transfer counts against `timeout_ms`.** The handler starts
   before a byte has moved, so the budget must cover the download as
   well as the work; the default path hands the handler its whole
   budget with the bytes already in. A hundred megabytes over a ten
   megabyte per second link is ten seconds of it.

The body is read once — a second `stream()` is `refused` — and a handler
that never calls it never fetches the bytes at all. The SDK still hashes
the body to check it (about 88 ms per hundred megabytes), so a handler
that also hashes pays for two passes; take the digest from the iteration
if that is all you wanted.

A failure is a `CallError` with a `code`: `timeout`, `refused`,
`too_large`, `busy`, `gone`, `failed`, `lost`, `node_offline`,
`node_unknown`, `ambiguous`, `forbidden`, `agent_too_old`,
`daemon_too_old`. `node_offline` is also what a call gets when this host
has no agent at all, and `daemon_too_old` when the daemon answered no
ticket within five seconds. A `CallError` prints with its code:
`String(err)` is `node_offline: …`, `console.error(err)` shows
`CallError [node_offline]: …`; `err.name` and `err.message` are unchanged.

### Streams

A stream is a call with two framed legs instead of one result:

```js
io.stream('analyze', { idle_ms: 30_000, concurrent: 8 }, async (session, params) => {
  for await (const f of session.input.frames())               // one JPEG per frame
    session.output.writeFrame(await model.annotate(f.payload), { key: true })
  return { done: true }
})

const s = await io.open('gpu-1', 'vision', 'analyze', { fps: 15 })
s.send(io.mjpeg(ffmpeg.stdout))                                // late frames are dropped, never queued
for await (const f of s.output.frames()) display(f.payload)    // ends on the callee's END
await s.close()
const { data } = await s.result                                // rejects if either leg died
```

A live session always goes through the agent's data socket, so
`io.open(null, …)` (or `'self'`) reaches this machine through the fleet,
as `io.node.id`; on a host that is not linked it rejects `node_offline`
(`this host is not linked: …`).

`idle_ms` (a stream only) is how long a session may stay silent, at most
24 h; it is the same wire field as `timeout_ms`, which the daemon reads as
a session's idle period, and wins over it.

**A leg that dies never kills the process.** A refusal from the agent — a
stale token, a quota, a lost link to the backend — is a routine event: the
sink keeps the failure, `await s.result` rejects with it, and `s.failure`
names it. `send()` and `pipe()` are both safe, with no `'error'` handler to
remember. Listen for `'error'` on `s.input` (or use `stream.pipeline`)
only when you want to know before the reply.

`await s.close()` means the bytes are in: it ends this side's leg, waits
for the agent to answer it (3 s by default, `opts.sink.settle_ms`; an
agent that does not answer by then has its leg taken back rather than
leaking a socket) and then stops reading. It never rejects — the failure
is on `s.result` and `s.failure`. On the caller's side, closed, the
session gives its slots back at once — its place among the 16 sessions
and the 16 calls in flight of the process — so the next `io.open` is not
`busy`; its call stays routed, so a reply that comes late is still
`s.result`, for `reply_grace_ms` (10 s at most), after which `result`
rejects `lost`. The daemon keeps its own count: a call placed on the
fleet holds one of the host's 64 forwards until it is answered or its
deadline passes, whatever its legs did — the callee answers once it sees
the `END`, which is what normally frees it. On the serving side the same rule
holds: a handler that returns happily while its output leg was refused
answers a failure, not `ok`. A stream handler must return: the session
holds one of the action's `concurrent` slots until it does, and nothing
reclaims it before `ACTION_STREAM_MAX_MS` (24 h).

**Ending a session.** `s.close()` also cuts the reading leg: a
`for await` over `frames()` in progress simply returns — it never throws
`ERR_STREAM_PREMATURE_CLOSE` — while a call that failed still throws its
failure there. A leg is read once: a second `frames()` while the first
reads throws (a closed or failed leg says why instead). To stop sending
but hear the rest of the answer, half-close
with `await s.end()` (both sides have it): it lets go of whatever is piped
in, writes `END`, waits for the agent like `close()`, and leaves the
reading leg open until the other side's own `END`. After the end a
`writeFrame` is ignored and returns `false`: a producer racing the close
is not an error. So is a frame the wire cannot carry — a payload over
1 MiB, a `ts_us` that is not an integer: `writeFrame` returns `false`,
the sink counts it in `stats.refused` and says it on `'refused'`, and
the leg goes on (a frame `pipe()`d in is refused the same way, never
failing the stream; `send()` counts only the frames that went).

**Feeding a source.** `s.send(readable, { end })` writes one frame per
chunk (`io.mjpeg(…)`'s images, or objects shaped like `writeFrame`'s)
and ends the leg with `END` when the source ends, fails or is closed —
`ffmpeg` killed mid-image included — or when `sending.stop()` is called
(the source is paused, never destroyed). `{ end: false }` keeps the leg
open for the next source. `await sending.done` never rejects: it is
`{ frames, error }`, `error` naming what broke the source (`null` for an
end, a close or a stop). `pipe()` still works; it just ends the session
with its source and says nothing when the source dies. `io.mjpeg` itself
never emits `'error'`: a source that fails or closes early ends the
images, drops the partial one, and leaves the reason on
`out.source_error`.

**Relaying.** A received frame is written on another leg as it came —
`session.output.writeFrame(f)` keeps its payload, flags (unknown bits
included) and `ts_us`; the sink gives it its own `seq`. A named flag
passed explicitly wins: `writeFrame(f, { key: false })`.

**What the path lost, and how late.** `s.stats.dropped` counts only this
process's abandon sink; the agents and the backend's window drop too.
Every one of them leaves a hole in `seq`, so the reading side counts
them: `s.stats` is `{ sent, dropped, bytes_out, received, bytes_in,
missing, discont }` (`missing` from the gaps, wrapping at 2³²; an `INIT`
replayed to a late reader is not a gap). The reading leg's own
`stats.latency` (`s.output.stats` on the caller, `session.input.stats`
on the callee) is `now − ts_us` over the last 256 frames, as `{ samples,
p50_ms, p95_ms, max_ms }` at the wall clock's millisecond resolution. It
is exact on a round trip, where one clock stamped and read the frame;
between two hosts it includes their clock skew, and can be negative. A
producer that wants to react sooner listens on the sink:
`s.input.on('drop', ({ seq, key, bytes, dropped }) => lowerFps())`.

Both sides write through an **abandon sink**: frames go out as fast as
the socket takes them and, when it says no more, they queue: a frame
that waited past a 100 ms window is dropped on its own age, never the
newest one (`s.stats.dropped` counts them) — `opts.sink.window_ms` moves
it, and it is this process's own window, unrelated to the backend's
`--live-window-ms`; past 8 MiB queued the oldest frame that is not a
`KEY` goes first, then the oldest `KEY` or superseded `INIT`, until the
queue fits. The newest `INIT` frame is always kept, as the agent's sink
keeps it: without it a reader that joins cannot decode at all; an older
one is an ordinary frame. Abandon starts at the source; back-pressure never travels
up a live stream. At most 16 sessions per process, as the daemon allows,
and at most 16 calls in flight on one connection.

A message is a 20-byte big-endian header and a payload of at most 1 MiB:
`u32 len | u8 flags | u8 hdr | u16 rsv | u32 seq | u64 ts_us`. Flags are
`KEY 0x01` (a message a reader may join on), `END 0x02` (the close),
`INIT 0x04` (decoding parameters — SPS/PPS, `ftyp+moov` — kept and given
to every reader that joins) and `DISCONT 0x08` (a gap: reset the
decoder). A longer header is accepted and skipped, a shorter one
refused, and unknown flag bits are carried. `ts_us` is a BigInt: the top
of its range does not fit a double. `io.mjpeg(readable)` cuts JPEG
images on `FFD8`/`FFD9` as they arrive — the `FFD9` looked for past the
headers, up to the start of the scan (`SOS`), so the thumbnail an EXIF
block carries never cuts its image short — and marks each one `KEY` (give
`ffmpeg` `-flush_packets 1`, or it adds 33 ms). It holds at most one
message: an image that never ends — a dead encoder, or a stream that is
not MJPEG but happens to carry `FF D8` — is given up on, the next start
marker resynchronises, and the frame that follows carries `DISCONT` so
the decoder resets. For H.264, keep the GOP under a second, drop
B-frames, and send SPS/PPS as an `INIT` frame.

`io.open` takes no `file`: a session carries frames, and a file belongs
to `io.call` (passing one is refused rather than ignored).

### `data.sock`

The SDK knows one path and never a network address: the agent's Unix
socket. This is the contract it speaks, and the whole of it.

**Where.** `$THREEPM_DATA_SOCK` when set. Otherwise, under
`$THREEPM_HOME` (or the home `resolvePaths` finds): `agent/data-<uid>.sock`
when one exists for this uid — a root daemon running a hosted user's app
leaves one socket per account, mode 0600, owned by that uid — else
`agent/data.sock`. A candidate counts only when `stat()` says it is a
socket. None of them: there is no agent here (and never one on Windows,
where a call with bytes is `node_offline` and nothing crashes).

**Requests.** HTTP/1.1 on that socket, `Authorization: Bearer <token>`
on every one — the local token the daemon put in the call's ticket, not
the node's. The blob id is percent-encoded in the path. Keep-alive is
asked for by everything except a tail, which takes a connection of its
own, and must be allowed, never required — a `PUT` asks for it so that
a refusal landing mid-body survives the answer.

| Request | Body | Answer |
|---|---|---|
| `PUT /blobs/<blob>` | the file, with `Content-Length` **and** `X-3pm-Sha256: <64 lowercase hex>` | `201` |
| `PUT /blobs/<blob>` | frames, `Transfer-Encoding: chunked`, no digest | `201` when the body ends |
| `GET /blobs/<blob>` | — | `200`, the bytes |
| `GET /blobs/<blob>?live=1` | — | `200`, the frames, until the writer's `END` |

Those four are the whole surface. `X-3pm-Sha256` is a **request header,
never a trailer**: the SDK hashes a file before it places the call (the
digest travels in the call itself, so the callee can check it), and sends
no HTTP trailer anywhere. A `Content-Type` of
`application/octet-stream` is sent and need not be read. The SDK never
sends `Expect: 100-continue`.

**Statuses.** `200` a `GET`; `201` a `PUT` taken. `401` the token is
unknown or expired; `404` no such blob, or not one of this call's; `409`
the blob is settled (a second `PUT`); `413` over the size the call
declared; `422` a malformed request (see below); `429` over a quota of
the bucket or the node; `503` the agent has no link to the backend, or
the backend no longer knows the agent's bearer (it takes a fresh one at
its next `welcome`, so the caller retries), or the agent is already
holding every connection it can — 512 for the host, shared by every app
on it (`this agent holds all the connections it can`). One connection
per transfer in flight is the rule, so that last one is a host running
hundreds of streams at once, not a busy app. Every other status the
backend answers with is relayed as it is when it is one of `403`, `404`,
`408`, `409`, `413`, `422`, `429`, and becomes `502` otherwise — that
is the agent's one rule, not a list of special cases. The SDK reads them as
`forbidden` (401, 403), `gone` (404, 409), `timeout` (408), `too_large`
(413, 422), `busy` (429), `node_offline` (503) and `failed` for anything
else; `ENOENT`, `ECONNREFUSED`, `EACCES` and `ENOTSOCK` on the socket
are `node_offline`. A refusal's body is read up to 1 KiB and its first
512 bytes go into the error message, so it is worth one plain line.

**Timeouts, per direction.** A response header is waited for 10 s on
every request **except a `PUT`**, sized or chunked: both are answered
only when the body ends — hours later for a stream, minutes for a file
of a few gigabytes — so neither is given a head budget at all. The agent
must therefore send a `GET`'s header as soon as it accepts the request —
`200`, flushed before the first byte of the body — and must not answer a
`PUT` early. That budget starts when the request gets a socket, not when
it is created; a request still queuing behind the pool has its own
budget of 120 s, after which it fails `timeout` rather than waiting for
ever. What holds a `PUT` instead is silence, twice over: 5 min without
the body moving ends it, and 5 min between the last byte and the status
ends it too. Both are long on purpose. A `GET` body gets 60 s
(`BLOB_IDLE`), but an upload stops moving for a reason that is not a
fault — the backend's window fills until a reader attaches — so the
write budgets guard against an agent that has gone away, never against
a reader who is slow. The SDK
sets `TCP_NODELAY` and keeps a pool of 8 keep-alive sockets for
requests that are over in a round trip and those alone: a read the
caller knows is settled **and** under a mebibyte. Every `PUT`, sized or
chunked, every read of a blob still being written, and every settled
read larger than that go on an unbounded agent instead, so that the
ninth of them never waits out its queue budget behind eight transfers
that have not finished. A `PUT` keeps `Connection: keep-alive` even so:
a refusal that lands while the body is still going out must survive,
and a connection closed on the spot takes the answer with it. Pooled
sockets are dropped after 8 s, under the agent's own 10 s, so that the
SDK is the side that closes first.

**Sizes.** Header lines stay under 8 KiB and the whole header block
under 16 KiB (the SDK sends about 300 bytes). The 16 KiB is enforced on
the **first** head of a connection, which the agent reads itself; the
heads 2..n of a keep-alive connection are bounded by the agent's own
read buffer, 256 KiB, so a later head between the two sizes is served
rather than refused. A `file` `PUT` is cut at the size the call
declared; a `live` frame is at most 1 MiB.

Sixteen details an implementation needs, settled here so both sides agree:

1. **No `Range`.** A blob is read whole. The SDK never sends one, so
   there is no `206` and no `416` to implement; an agent that wants to
   answer a `Range` from some other client may, but nothing here depends
   on it.
2. **A malformed `PUT` is `422`.** That means: neither `Content-Length`
   nor `Transfer-Encoding`, an `X-3pm-Sha256` without a `Content-Length`
   (a chunked body has no digest to check against), a digest that is not
   64 lowercase hex, a body that does not match the digest, a `live`
   frame over 1 MiB, and a chunked `live` body whose bytes are not
   frames at all or that stops in the middle of one (the agent says how
   many bytes into the message it ended). A body longer than the
   `Content-Length` it announced is what the backend refuses `413`, and
   the agent relays that as it relays every backend status (see
   *Statuses*). The SDK produces none of these; the agent refuses them
   rather than guessing.
3. **A tail ends with its body.** After relaying the writer's `END`
   frame the agent closes the response. The SDK's reader stops on the
   `END` flag *and* on the end of the body, so both work — the rule is
   written down so the agent and the backend do not pick different ones.
4. **`GET ?live=1` late.** A blob whose writer finished cleanly and
   whose ring the agent still holds answers `200` and an immediate `END`:
   a clean end, not an error. A blob that is unknown, or whose ring is
   gone, is `404` — which the caller reads as `gone`.
5. **A chunked `PUT` cut short** (the writing process died mid-body)
   seals the blob as truncated, never as complete: the reader's body ends
   without an `END` frame. A `file` `PUT` that stopped short of its
   `Content-Length` is discarded, so the reader sees fewer bytes than the
   call declared and fails the call rather than handing a half file to a
   handler.
6. **No `403`.** `401` covers the token and `404` covers "not a blob of
   this call". The SDK still maps `403` to `forbidden`, for a proxy that
   invents one; the agent does not use it.
7. **Response headers** stay under 8 KiB in total. Node's parser stops
   at 16 KiB and the SDK would read the failure as `failed`; it reads no
   response header other than the status.
8. **A blob id** is `^[A-Za-z0-9_-]{1,64}$` (the backend mints
   `bl_<hex>`). The SDK percent-encodes the path segment anyway, so the
   agent percent-decodes it and answers `404` to anything outside that
   alphabet.
9. **The agent's own request-header budget** is 10 s, answered `408`
   (which the SDK reads as `timeout`).
10. **Keep-alive.** The agent may close an idle socket whenever it likes
    — the SDK's pool holds them for 15 s and tolerates a close — but it
    must not answer `Connection: close` to a short request, or the pool
    of 8 is rebuilt at every call.

## The fleet backend's HTTP API: `@3pm/api/fleet`

Everything above is this host's daemon. A *fleet backend* — the server
side of `3pm link` — is a different thing: an HTTP API over every node
that reports to it. Its client is generated from that backend's own
route table and ships here as a subpath:

```js
const { Client } = require('@3pm/api/fleet');

const api = new Client({ baseUrl: 'https://fleet.example.com', token: '3pmu_…' });
const buckets = await api.listBuckets();
const nodes = await api.listNodes(buckets[0].cluster_id);

for await (const [kind, payload] of api.follow(buckets[0].cluster_id)) {
  if (kind === 'telemetry') console.log(payload);
}
```

A method per endpoint, `api.url(name, …)` for the routes that are bytes
rather than JSON (an artifact's file, a blob), `ApiError {status, code,
detail}` carrying the backend's own `ERR_*`, and `api.discover()` to
follow the table of whichever backend answers. A token is how you
authenticate — there is no browser session to lean on. The live feed is
read with `fetch` and a `ReadableStream`, the only way Node has.

The feed is cut on the server before it leaves. It comes **without**
`init.history` unless you ask for it, and takes the rest as a query:

```js
// Only the nodes whose hostname contains "envision", each telemetry frame
// cut to the host's gauges and a short row per process.
api.events(clusterId, handlers, { match: 'envision', view: 'live' });
for await (const item of api.follow(clusterId, { query: { view: 'live' } })) …
api.events(clusterId, handlers, { history: 1 }); // the per-process history back
```

On a bucket of nine hosts reporting every second that is ~100 KB/s down
to ~14 KB/s (five hosts, `view: 'live'`), and ~1 KB/s through a proxy
that gzips `text/event-stream`.

It is the browser client (`GET /api/v1/client.js`) generated a second
time as CommonJS, with the browser half — the token prompt, the
`EventSource` reader — cut out. Both come from
`3pm-backend/src/api/client.tmpl.js` via `just api` in that repository,
so neither is edited here.

## Compatibility

The wire is growable: unknown events are still surfaced on `event`, unknown
response tags become a `protocol` error rather than a hang, and the frozen
fixtures of `threepm-protocol` are part of this package's test suite.
`PROTOCOL_VERSION` is checked on every `connect()`. The `live` framing has
frozen vectors of its own
(`crates/threepm-protocol/tests/wire_samples/live_frames.json`), replayed
here.

A daemon that does not know how to place a call ignores the line: the
call fails `daemon_too_old` after five seconds, and nothing else
changes. A daemon that dies mid-call fails it `lost` at once, rather
than at its deadline. An older SDK never receives an input, since it
declares none.

With no agent on the host there is no socket to find, and a call that
carries bytes is `node_offline` before it is even placed; only the local
call (`node: null`) works. Every other fleet code — `agent_too_old`,
`node_unknown`, `ambiguous`, `forbidden`, `lost` — comes from the daemon
on the reply and is passed through as it arrived, newer ones included:
the SDK produces only `node_offline` (no socket, no daemon),
`daemon_too_old`, `busy` (its own caps), `refused` (what it can check
before placing a call) and `failed`.

## Development

```console
npm test                       # unit + wire fixtures + e2e (uses ../../target/{debug,release}/3pm or THREEPM_BIN)
npm run coverage               # the same, with per-file coverage
npm run check-types            # compile the published declarations and a consumer under --strict
```

`check-types` compiles `test/types/consumer.ts` — which imports only from
`index.d.ts` and exercises the documented lines, `io.mjpeg(…).pipe(s.input)`
included — together with the declarations, **without** `--skipLibCheck`,
which would skip the `.d.ts` files and so the whole point. The package has
no dependencies and gains none for this: TypeScript is taken from a local,
workspace or `npx`-cached install, and when there is none (or no
`@types/node`) the gate says so and exits 0. `node --test` type-checks
nothing, so without this a broken declaration ships with 217 green tests.

</document>

<document url="https://3pm.edgecr4ft.net/docs/sdk/python.md" title="Python SDK: threepm-io">

# threepm-io

Application metrics, events, **actions and calls** for a process started by
[3pm](https://github.com/Unitech/3pm) — pm2's `io.metric` / `io.counter` /
`io.notify_error`, for 3pm, in Python. Standard library only, Python ≥ 3.9.

```console
pip install --index-url https://3pm.edgecr4ft.net/simple/ threepm-io    # or: pip install -e 3pm/packages/python
```

Published on <https://3pm.edgecr4ft.net> only, not on PyPI. Keep
`--index-url` (a `requirements.txt` takes the same line), not
`--extra-index-url`, which would also let pip pick a same-named package on
PyPI.

```python
import threepm_io as io

io.init(source="cv-engine")                      # one connection per process of the app
xruns   = io.counter("cv.xruns", description="PortAudio over/underruns")
jitter  = io.histogram("cv.callback.jitter", unit="ms")
clients = io.gauge("cv.ws.clients")
io.metric("cv.queue.depth", lambda: len(queue))  # read at every flush, in the flush thread

xruns.inc()
jitter.observe(0.42)
clients.set(3)
io.event("midi.port_lost", level="warn", attrs={"port": "ES-9"})
try:
    open_device()
except Exception as e:
    io.notify_error(e, attrs={"device": "ES-9"})  # an `error` event with the traceback
```

Then `3pm show <app>` (section "Custom metrics") and `3pm events -f app`.

Outside 3pm (no `THREEPM_CTL_SOCK` in the environment) every call is a
no-op. Nothing blocks the caller and nothing raises: a daemon thread
connects, sends the whole registry every second, reconnects with backoff
(1 s → 30 s) and gives up after five refused `hello`s (the process outlived
its incarnation). Events raised before the connection is up are queued
(64, under the same 64 KiB outbound cap). A short-lived script should
`io.flush(wait_ms=500)` before exiting; `shutdown(wait_ms=500)` is that
flush followed by the close.

Cache the series you write to on a hot path (`requests = io.counter(...)`
once, `requests.inc()` per event): `.labels({...})` is a dictionary lookup
plus validation, fine per request, wasteful per sample. Past 50 label sets
a metric folds into one `overflow="true"` series; `.labels()` on that fold
returns the fold itself.

| Call | Notes |
|---|---|
| `init(source=None, flush_ms=1000, on_shutdown=None, connect_wait_ms=5000)` | Idempotent; `source` defaults to `main` (see below); `on_shutdown` runs when the daemon asks for a graceful stop (`shutdown_with_message`), in the reader thread — it may call `shutdown()`. Test knobs, in milliseconds like the Node and C++ SDKs: `reconnect_min_ms`, `reconnect_max_ms`, `max_buffer_bytes`, `max_refusals`, `refusal_window_ms` |
| `gauge` / `counter` / `meter` / `histogram(name, unit=None, description=None, labels=None, scale=1)` | Same name + labels = same object; `.labels({...})` derives a child; counters are cumulative; histograms are base-2 exponential (`scale` 0..4) and never carry more than 128 non-empty buckets — a value that would open a 129th lowers the scale by one (buckets merge pairwise, `count`/`sum` exact) until it fits; past 50 label sets a metric folds into `overflow="true"`. Non-finite numbers (NaN, infinities, integers beyond a double) are ignored |
| `metric(name, fn, ...)` | Gauge by callback — `fn` runs in the flush thread, outside any asyncio loop |
| `event(name, level=None, attrs=None, exc=None, fp=None)` | `level` in `debug`/`info`/`warn`/`error`/`fatal`; `attrs` flat scalars (≤ 16) |
| `notify_error(err, attrs=None, name=None)` | `error`-level event carrying `exc.{type,message,stacktrace}` |
| `flush(wait_ms=None)`, `shutdown(wait_ms=500)` | `flush` waits for the connection then the bytes when `wait_ms` is given (never from the flush thread itself); `shutdown` is `flush(wait_ms)` then the close |

## Actions and calls: `io.action`, `io.call`

A function another process — `3pm trigger`, `3pm call`, an `io.call` from
any SDK, the fleet backend — can call, on this machine or from another
node of the bucket, **with a file in and a file out**. Same wire as the
Node SDK (`packages/node/README.md`, "Fleet calls"), so a Node caller
reaches a Python action and the reverse.

```python
import threepm_io as io
io.init()                                   # source `main` by default, as in Node

@io.action("segment", input={"max_bytes": 50_000_000, "ext": ["png", "jpg"]}, timeout_ms=60_000)
def segment(params, ctx):                   # ctx.input: path, bytes, ext, sha256 — fetched and checked
    return model.run(ctx.input.path, params)            # JSON, ≤ 512 KiB, ≤ 64 levels deep

@io.action("upscale", input=50_000_000, output={"max_bytes": 20_000_000})
def upscale(params, ctx):
    render(ctx.input.path, ctx.file + ".tmp")           # a result too big for the line is a file
    ctx.commit_file("png")                              # renamed; PUT to the caller when it asked for it
    return {"seed": params.get("seed")}

io.serve()                                  # a worker that only answers calls: block until 3pm stops it
```

```python
res = io.call("gpu-1", "vision", "segment", {"threshold": 0.7}, file="/tmp/capture.png")
res["data"]                                 # {call_id, ok, data, duration_ms, results}
up = io.call("gpu-1", "vision", "upscale", file="in.png", output="/tmp/out.png")
up["file"]                                  # {"path": "/tmp/out.png", "bytes": …, "ext": "png"}
io.call(None, "vision", "segment", file=f)  # same machine: by path, neither agent nor backend
```

| Call | Notes |
|---|---|
| `action(name, handler=None, *, description, remote=True, timeout_ms, input, output, concurrent)` | `handler(params, ctx)` runs **in a thread of its own** (a coroutine function is run to completion there), so a slow handler never stalls the connection; what it raises is the failure (`"ValueError: …"`, cut to 1 KiB). Returns `unregister`; without `handler` it is a decorator. `input` is `{"max_bytes": n, "ext": [...]}` or a bare size; without it a call carrying bytes is refused by the daemon. `concurrent` (1..16) is how many calls of that action run at once; past it a call is `busy`. A bad name, a `3pm:` name, a 33rd action or an unreadable `output` is logged (`THREEPM_IO_DEBUG=1`) and ignored |
| `ctx` | `id`, `name`, `timeout_ms`, `deadline_epoch_ms`, `input` (`None` without bytes), `file`, `commit_file(ext)` → size, `committed`, `output`. An input blob is staged next to `ctx.file`, checked against the size and digest the call declared **before** the handler runs (a mismatch fails the call), and deleted once the reply is out; a local call's input is the caller's own file, never deleted. `commit_file` raises — and deletes the `.tmp` — past the call's deadline, and with `too_large` past the declared `output.max_bytes` |
| `call(node, app, action, params=None, *, file, ext, sha256, bytes, output, instance=0, source, timeout_ms)` | **Blocks** until the answer and **raises `CallError`** (`e.code`: `timeout`, `refused`, `too_large`, `busy`, `gone`, `failed`, `lost`, `node_offline`, `node_unknown`, `ambiguous`, `forbidden`, `agent_too_old`, `daemon_too_old`, or a newer one passed through; `str(e)` is `code: message`). `node` is a `3pmn_…` id or a unique hostname, `None` this machine. `sha256` + `bytes` skip the hashing pass when the digest is already held. `output` is a path to bring the action's file to, or `True` to be told the blob only. `instance=None` asks every instance and is refused with a `file`. At most 16 calls in flight per connection. Outside 3pm: `node_offline` |
| `ready(timeout_ms=None)` | Wait for the control connection (`connect_wait_ms`, 5 s, by default — what `call` does first); `False` outside 3pm |
| `serve()` | Block until the daemon's `shutdown` or `io.shutdown()`; returns at once outside 3pm. Nothing else keeps a Python process up: the SDK's threads are daemon threads |

The bytes never ride the control socket: they are `PUT`/`GET` on the
fleet agent's Unix socket (`$THREEPM_DATA_SOCK`, else
`agent/data-<uid>.sock` then `agent/data.sock` under the 3pm home, else
next to the control socket), with the token of the call's ticket. No
agent on the host: a call with a `file` to another node is
`node_offline` before it is placed; the local call still works.

**Not in the Python SDK yet:** streams (`io.open` / `io.stream`), a
streamed input (`input.stream`), the built-in `3pm:profile`.

**`init()` without `source` now names itself** — `main` for the process
3pm spawned, `pid-<pid>` for a process that one started (the
`THREEPM_IO_MAIN` mark it inherits) — as the Node SDK does: a sourceless
connection can report metrics but declare no action. Series of an app
that passed no `source` gain `src="main"`.

`threepm_io.Io()` gives an independent instance (another `source` in the
same process). Full reference, bounds and the other languages:
`docs/app-metrics.md` in the 3pm repository. Debug trace:
`THREEPM_IO_DEBUG=1`.

</document>

<document url="https://3pm.edgecr4ft.net/docs/sdk/cpp.md" title="C++ SDK: threepm_io.hpp">

# threepm_io.hpp

Application metrics and events for a process started by
[3pm](https://github.com/Unitech/3pm) — pm2's `io.metric` / `io.counter` /
`io.notify_error`, for C++17. Header-only, POSIX sockets, no JSON library,
no dependency.

```cmake
include(FetchContent)
FetchContent_Declare(threepm_io
  URL https://3pm.edgecr4ft.net/sdk/cpp/threepm-io-cpp-0.64.0.tar.gz
  URL_HASH SHA256=<sha256>)                 # in https://3pm.edgecr4ft.net/sdk/sdk.json
FetchContent_MakeAvailable(threepm_io)      # or add_subdirectory(3pm/packages/cpp), or copy include/
target_link_libraries(plugin_host PRIVATE threepm::io)
```

The tests build only when this directory is the top-level project, never
inside yours (`-DTHREEPM_IO_TESTS=ON` forces them).

```cpp
#include <threepm_io.hpp>
using namespace threepm::io;

Io& io = instance();                         // or your own `Io` (one per source)
InitOptions opts; opts.source = "plugin-host";
io.init(opts);

auto late    = io.counter("host.late_callbacks");
auto latency = io.histogram("host.callback.latency", [] { HistogramOptions h; h.unit = "ms"; return h; }());
auto state   = io.gauge("host.state", {"", "0 idle, 1 loading, 2 running, 3 error", {}});
io.metric("host.queue.depth", [&] { return double(queue.size()); });

late.inc();                                  // lock-free: fine on the audio thread
latency.observe(0.42);                       // no libm, no allocation, no lock
state.set(2);
EventOptions ev; ev.level = "warn"; ev.attrs.add("port", "ES-9");
io.event("midi.port_lost", ev);
io.notify_error("HostError", last_error, Attrs().add("plugin", name));
```

Then `3pm show <app>` (section "Custom metrics") and `3pm events -f app`.

Outside 3pm (no `THREEPM_CTL_SOCK` in the environment) every call is a
no-op. Nothing blocks the caller: one `std::thread` connects, sends the
whole registry every second, reconnects with backoff (1 s → 30 s) and
gives up after five refused `hello`s (the process outlived its
incarnation). Events raised before the connection is up are queued (64).
The destructor (or `shutdown()`) flushes and stops the thread.

| Call | Notes |
|---|---|
| `init(InitOptions{source, flush_ms, on_shutdown})` | Idempotent; `on_shutdown` runs (on the SDK thread) when the daemon asks for a graceful stop |
| `gauge` / `counter` / `meter(name, MetricOptions{unit, description, labels})` | Same name + labels = same series; `.labels({{k, v}})` derives a child; counters are cumulative; past 50 label sets a metric folds into `overflow="true"` |
| `histogram(name, HistogramOptions{…, scale, min_exp})` | Base-2 exponential, `scale` 0..4 (default 1). Fixed array of 128 buckets (`HISTOGRAM_SLOTS`, the daemon's cap) starting at `2^min_exp`; by default the window is centred on 1: scale 0 → [2^-64, 2^64), 1 → [2^-32, 2^32) (~2e-10 … 4e9), 2 → [2^-16, 2^16), 3 → [2^-8, 2^8), 4 → [2^-4, 2^4). Values outside fold into the edge buckets — that is what makes `observe` lock-free |
| `metric(name, fn, opts)` | Gauge by callback, evaluated on the SDK thread at every flush; it may use the registry, `flush()` or `shutdown()` (a nested flush renders the series only) |
| `event(name, EventOptions{level, attrs, exc, fp})` | `Attrs().add(key, string\|double\|any integer\|bool)`, ≤ 16 keys, a repeated key replaces; strings are made valid UTF-8 |
| `notify_error(const std::exception&, attrs)` / `notify_error(type, message, attrs, name, stacktrace)` / `notify_error(…, EventOptions, name)` | `error`-level event with `exc.{type,message,stacktrace}` (the type demangled); the `EventOptions` form reaches `level`, `fp` and the trace |
| `flush(wait_ms)`, `shutdown(wait_ms)`, `stats()` | `flush(wait_ms)` waits for the connection, then for the bytes — for a short-lived process; `shutdown` is `flush(wait_ms)` then stop, and is safe from `on_shutdown` |

`set`, `inc` and `observe` are lock-free atomic operations on preallocated
storage (`inc`/`observe` are a CAS loop: one writer per series for
real-time use — measured 7 ns alone, 2 µs with eight writers contending):
call them from a real-time thread once per callback, not once per sample.
`gauge()`, `counter()`, `histogram()`, `metric()` and `.labels()` take the
registry mutex: create the handles up front and keep them. `metric()`
callbacks and `on_shutdown` run on the SDK thread.
Full reference, bounds and the other languages: `docs/app-metrics.md` in
the 3pm repository. Tests: `cmake -S . -B build && cmake --build build &&
(cd build && ctest)`.

</document>

<document url="https://3pm.edgecr4ft.net/docs/sdk/rust.md" title="Rust SDK: threepm-io">

# threepm-io

Application metrics, events and custom actions for a process started by
[3pm](https://github.com/Unitech/3pm) — pm2's `io.metric` / `io.counter` /
`io.notifyError` / `io.action` / `io.stream` / `io.open`, for 3pm, in Rust. Two dependencies
(`serde`, `serde_json`), std threads, no runtime: it works the same in a
sync program, under tokio, or in an audio callback's host.

## Install

The crate is served from the 3pm download site, not from crates.io.
Declare the registry once, in `.cargo/config.toml` (beside `Cargo.toml`,
or in `$CARGO_HOME`):

```toml
[registries.threepm]
index = "sparse+https://3pm.edgecr4ft.net/sdk/cargo/"
```

then the dependency:

```toml
[dependencies]
threepm-io = { version = "0.64.0", registry = "threepm" }
```

`cargo add threepm-io --registry threepm` writes that line for you. The
registry is public and needs no token. Actions take and return
`serde_json::Value`: the crate re-exports it (`threepm_io::serde_json`),
or add `serde_json = "1"` beside it to write `serde_json::json!` as the
examples do; metrics and events need nothing else. Every version, its `.crate` file
and its sha256: <https://3pm.edgecr4ft.net/sdk/sdk.json>. In this
repository a path dependency works too:
`threepm-io = { path = "3pm/packages/rust" }`.

## Quickstart

```rust
use threepm_io::{io, Event, Init, Level, Metric};

io().init(Init::new().source("worker").on_shutdown(|| {
    // Asked to stop: signal your own loop, do not block here.
}));

let requests = io().counter("http.requests", Metric::new().label("method", "GET"));
let latency = io().histogram("http.latency", Metric::new().unit("ms"));
let temp = io().gauge(
    "room.temp",
    Metric::new().unit("Cel").description("studio temperature"),
);

requests.inc(1.0);
latency.observe(12.5);
temp.set(21.5);

io().event(
    "deploy",
    Event::new().level(Level::Warn).attr("version", "1.2"),
);
```

Then `3pm show <app>` (section "Custom metrics") and `3pm events -f app`.

Outside 3pm (no `THREEPM_CTL_SOCK` in the environment) every call is a
no-op: no socket, no thread, no timer. Nothing here blocks the caller,
returns a `Result` on telemetry, or panics — a metric that cannot exist
(an invalid name, a reserved label, a name already taken by another kind)
comes back as an inert handle, `enabled() == false`, plus one line on
stderr under `THREEPM_IO_DEBUG=1`.

`io()` is the process's instance, one control connection under one
`source`. `Io::new()` makes another, for a test or a second source.

## Metrics

```rust
let ok = requests.labels(&[("status", "200")]);
ok.inc(1.0); // keep the child, do not derive it per request

io().metric("queue.depth", Metric::new(), move || {
    read.load(std::sync::atomic::Ordering::Relaxed) as f64
});

io().meter("req.sec", Metric::new()).mark(1.0);
io().histogram("db.query", Metric::new().unit("ms").scale(3))
    .observe(0.8);
```

| Call | What |
|---|---|
| `gauge(name, Metric)` → `.set(v)` | The last value |
| `counter(name, Metric)` → `.inc(n)` | Cumulative since process start; never decremented, rates computed at read time. Rust has no default arguments: `inc(1.0)`, not `inc()` |
| `meter(name, Metric)` → `.mark(n)` | pm2's meter — the same counter under the name a rate is read by |
| `histogram(name, Metric)` → `.observe(v)` | Base-2 exponential buckets, `scale` 0..=4 (default 1). A value that would open a 129th bucket lowers the scale by one, merging pairwise: `count` and `sum` stay exact |
| `metric(name, Metric, f)` | A gauge read by calling `f` **on the flush thread**, once per `flush_ms`. `false` when the name could not be taken |
| `.labels(&[(k, v)])` | A child series with those labels added. Same name + labels = the same series |

`Metric::new().unit("ms").description("…").label(k, v).scale(2)` carries
the options; only the first declaration of a name is kept. `Series` is
`Clone + Send + Sync` — create the handles at startup and keep them:
`.labels()` takes the registry lock and validates, which is fine per
request and wasteful per sample. Non-finite values (NaN, infinities) are
ignored, as is a negative `inc`.

Names are `^[a-z][a-z0-9_.]{0,63}$`, the unit is **not** in the name
(`http.latency` + `unit: "ms"`). At most 3 label keys
(`^[a-z][a-z0-9_]{0,31}$`, values ≤ 64 bytes); the keys the fleet sets
itself (`app`, `instance`, `src`, `node`, `overflow`, …) are refused. Past
50 label sets a metric folds into one `overflow="true"` series, whose
`.labels()` returns the fold itself. Full model, bounds and the other
languages: `docs/app-metrics.md` in the 3pm repository.

### Families, removal, usage — for series made from what a process finds

A process that creates series from what it discovers (a monitoring
module: one series per service found) needs three more things:

```rust
use threepm_io::{io, Metric};

// A family: the name, unit and description — and no series. `gauge()`
// reports its unlabelled series from the first flush, even when only
// its children are ever set; a family never does.
let memory = io().gauge_family("redis.memory", Metric::new().unit("By"));
let used = memory.labels(&[("target", "redis@127.0.0.1:6379"), ("dim", "used")]);
used.set(1_048_576.0);

// The service is gone: stop reporting the series. The daemon drops it
// after its TTL (two minutes); asking for the same labels again starts
// a fresh one. `memory.remove_all()` removes every child.
used.remove();

// Budget against `threepm_io::limits` before creating more.
let u = io().usage(); // names, series, rejected, folded, removed
```

| Call | What |
|---|---|
| `gauge_family` / `counter_family` / `histogram_family(name, Metric)` → `Family` | The declaration alone; `.labels(&[(k, v)])` makes each child (the declaration's labels included) |
| `Series::remove()` | Stop sending it; the handle and its clones become inert until the labels are asked for again |
| `Family::remove_all()` | Every child; the name stays declared (it counts against the 200) |
| `Io::usage()` → `Usage` | Names and series held, declarations refused, label sets folded, series removed |

Under a daemon that announces `sticky_meta` in `THREEPM_CTL_CAPS`, a
series' unit and description travel once per connection (again after a
reconnect) instead of in every batch.

## Events and errors

```rust
io().event(
    "midi.port_lost",
    Event::new()
        .level(Level::Warn)
        .attr("port", "ES-9")
        .attr("retries", 3)
        .fp("midi.port_lost"),
);

if let Err(e) = std::fs::read("/does/not/exist") {
    io().notify_error(&e, Event::new().attr("where", "worker"));
}
```

An event is sent at once (and queued, 64 of them, while the connection
comes up). `Level` is `Debug` / `Info` / `Warn` / `Error` / `Fatal`;
`attr` takes a string, a bool or any integer or float, at most 16 keys,
strings cut to 256 bytes; `fp` (≤ 64 bytes) is the fingerprint the daemon
groups and rate-limits by, derived from the event when you name none.

`notify_error(&err, Event)` takes any `E: std::error::Error + ?Sized` and
sends an `error`-level event named `error` carrying:

- `exc.type` — `std::any::type_name::<E>()`, or `Event::name("…")` when
  you want something else (a `Box<dyn Error>` reports the box);
- `exc.message` — the error's `Display`;
- `exc.stacktrace` — its `source()` chain, then a backtrace **of the
  `notify_error` call site**, not of where the error was made, and only
  when `RUST_BACKTRACE` or `RUST_LIB_BACKTRACE` enables one. An empty
  stacktrace is left off the wire.

`Event::exception(type, message)` and `Event::stacktrace(…)` attach an
exception by hand to an event of any name.

## Actions

An action is a function of your process that `3pm trigger` and the fleet
backend can call (pm2's `io.action`).

```rust
// A JSON result: at most 512 KiB, at most 64 levels deep.
io().action(
    "cache.flush",
    Action::new()
        .description("drop the cache")
        .timeout_ms(5_000),
    |params: Value, _ctx: &ActionCtx| -> Result<Value, String> {
        let prefix = params["prefix"].as_str().unwrap_or("");
        Ok(json!({ "flushed": prefix }))
    },
);

// A failure: anything that prints. `Err` is the call's error message.
io().action("reindex", Action::new(), |_: Value, _: &ActionCtx| {
    std::fs::read_to_string("/etc/hostname")
        .map(|host| json!({ "host": host.trim() }))
        .map_err(|e| format!("reindex failed: {e}"))
});
```

The handler is `Fn(Value, &ActionCtx) -> Result<Value, E> + Send + Sync`
with `E: Display` — `String`, `io::Error`,
`Box<dyn Error + Send + Sync>`, `anyhow::Error`, your own type. It runs on
its own thread and may block for as long as its timeout allows. `Ok(v)` is
the call's `data` (`Ok(Value::Null)` answers success with no data), `Err(e)`
its error message (cut to 1024 bytes). A handler that panics answers
`the handler panicked: …` and the process lives on.

**A file, for a result too big for one line.** The daemon prepares a path;
write `<path>.tmp` and rename it into place with `commit_file`:

```rust
io().action(
    "dump",
    Action::new().output_max_bytes(16 << 20),
    |_: Value, ctx: &ActionCtx| -> Result<Value, String> {
        let tmp = ctx.tmp_file().ok_or("the call named no file")?;
        std::fs::write(&tmp, b"{}").map_err(|e| e.to_string())?;
        Ok(json!({ "bytes": ctx.commit_file("json").map_err(|e| e.to_string())? }))
    },
);
```

`commit_file(ext)` takes the extension (`^[a-z0-9]{1,16}$`, never `tmp`)
and returns the size. It refuses — deleting the `.tmp` — in the call's
last tenth (at most the last second), because the daemon has given up and
swept by then, and past the `output_max_bytes` the action declared.
`output_max_bytes` outside 1..=256 MiB leaves the action **unregistered**:
the daemon refuses a source's whole list over one bad declaration, so the
SDK refuses the one action instead.

**Local-only, and concurrency.**

```rust
let reset = io().action(
    "reset",
    Action::new().remote(false).concurrent(4),
    |_: Value, _: &ActionCtx| Ok::<Value, String>(json!("done")),
);
if !reset.enabled() {
    eprintln!("reset was refused: run with THREEPM_IO_DEBUG=1 to see why");
}
reset.unregister(); // explicit; dropping the handle withdraws nothing
```

`remote(false)` keeps the action out of the fleet backend's reach
(`3pm trigger` still calls it). `concurrent(n)`, 1..=16, is how many calls
of that action may run at once — past it a call is answered `busy`, and a
slot is held at most the call's timeout, so a hung handler frees it when
the daemon gives up. `Registration::unregister()` is explicit on purpose:
a silent `Drop` would withdraw an action by accident.

**Bytes the caller sent.**

```rust
io().action(
    "import",
    Action::new().input(8 << 20, &["json", "csv"]),
    |_: Value, ctx: &ActionCtx| -> Result<Value, String> {
        let input = ctx.input.as_ref().ok_or("this call carried no bytes")?;
        let body = std::fs::read(&input.path).map_err(|e| e.to_string())?;
        Ok(json!({ "read": body.len(), "ext": input.ext }))
    },
);
```

`input(max_bytes, ext)` is the declaration — at most 8 GiB, at most 16
extensions. `ctx.input` is `ActionInput { path, bytes, ext, sha256, live }`:
the bytes are already on this host, staged by the daemon. **Read them,
never delete them.** A call whose input is a *blob* (bytes the fleet
backend holds, fetched over `data.sock`) is refused at once with a message
saying so — that is not implemented yet, see below.

Names are `^[A-Za-z0-9_.:-]{1,64}$`, 32 per source; `3pm:` is the SDK's
prefix and a name under it is refused (an inert `Registration`, never a
panic). Registering a name twice replaces the handler and keeps its place;
the older `Registration` then withdraws nothing. The whole list is
declared right after every `hello` and again whenever it changes, and
re-sent every flush until a write of it succeeded. `3pm trigger <app>`
lists what is declared.

## Streams

A stream is a call with two framed legs instead of one result: the caller
and the callee each `send` frames the other `recv`s, until both ended.
Frames ride the agent's `data.sock` — agent to agent when the two nodes
are LAN neighbours, through the fleet backend otherwise — so both sides
must run on **linked** nodes (`3pm link`); a session to this very host
goes through the fleet too.

**The callee** declares a stream action; its handler runs once per
session, on its own thread, for as long as the session lasts:

```rust
io().stream(
    "echo",
    Action::new()
        .description("send every frame back")
        .concurrent(4)
        .idle_ms(60_000),
    |s: &Session, _params: Value| -> Result<Value, String> {
        while let Some(frame) = s.recv().map_err(|e| e.to_string())? {
            // Forwarded as it came: payload, flags and `ts_us`, so the
            // caller measures the round trip from its own stamp.
            s.send_frame(frame);
        }
        Ok(Value::Null)
    },
);
io().serve();
```

Returning closes the session; `Ok(Value::Null)` answers its stats.

**The caller** opens a session on a node (`Some("3pmn_…")`, a unique
hostname, or `None` / `"self"` for this host):

```rust
let s = io().open(node, "echo", "echo", json!({}), Open::new())?;
s.send(b"hello"); // no flag, stamped now
s.send_frame(Frame::new(b"a key frame".to_vec()).with(FLAG_KEY));
s.end(); // half-close: our END, we keep reading
while let Some(frame) = s.recv()? {
    println!("{} bytes back", frame.payload.len());
}
let reply = s.result(Duration::from_secs(5)); // Some(Ok(data)) once answered
s.close();
```

`open` blocks until the daemon's ticket (≤ 5 s, `daemon_too_old` past
it). A refusal before the ticket is its `Err`; one after it (`busy`, a
target whose `callers` say no) fails the session — `recv` returns that
error, `send` returns `false`, `result` says so.

| `Session` | |
|---|---|
| `send(&[u8])`, `send_frame(Frame)`, `send_shared(&Arc<[u8]>)` | Queue one frame, never block; `false` once this side ended or the session failed, or for a payload over 1 MiB. `send` copies the payload once, `send_frame` moves it, `send_shared` shares it — a frame broadcast to N sessions is held once |
| `recv()` | The next frame, blocking; `Ok(None)` at the other side's `END`, after `close`, and on a callee whose caller sends nothing; `Err(CallError)` when a leg broke, a minute passed without a byte, or the call failed |
| `end()` | Half-close: our `END`, then the agent's answer (≤ 3 s); keep reading |
| `close()` | `end()`, then stop reading: a `recv` blocked in another thread returns `Ok(None)`; the session's slot comes back, and the call stops counting against the 16 in flight (its reply, if it has not come, still has the reply grace, then `lost`) |
| `stats()` | `sent`, `dropped`, `bytes_out`, `received`, `bytes_in`, `missing` (holes in the other side's `seq`, every abandon point on the path), `discont` |
| `failure()`, `call_id()` | What killed the session; the call's id |
| `result(wait)` | Caller only: the callee's answer — its handler's data, or its stats when it returned `Null` |

`Session` is `Send + Sync` and `Clone`: one thread `recv`s while another
`send`s. Dropping the last clone ends the writing leg without waiting.

**Lossy by design, low latency first.** A frame is written as soon as it
is sent — one HTTP chunk per frame, on its own writer thread, so `send`
never waits; what is already queued when the writer comes round goes out
in one vectored write (up to 64 frames, 64 KiB), but nothing is held back
to make a batch. While that socket is blocked (the agent or
the link applies back-pressure) frames queue: a frame that waited past
**100 ms** is dropped on its own age (never the newest one), and past
**8 MiB** the oldest frame that is not a `KEY` goes first, then the
oldest `KEY` or superseded `INIT`, until the queue fits. Only the newest
`INIT` is kept whatever its age, as the agent's sink keeps it: drops
happen at the source and only while the socket is blocked, never on a
socket that keeps up. Mark the frames a
reader can start from `FLAG_KEY` and the decoding parameters `FLAG_INIT`;
an empty payload is a fine keepalive. A live leg keeps nothing for a
reader that has not joined yet: frames sent right after `open`, before the
callee's handler first `recv`s, can be lost on the way (they show up in the
callee's `missing`). A caller that needs its first frames delivered sends a
hello until the first frame comes back.

**Every leg must move at least once a minute.** A reading leg gives up
after 60 s without a byte — in either direction, and whatever `idle_ms`
the action declares (that one is the daemon's guard, this one the
backend's). A session that may stay quiet sends an empty frame every 15 s
or so.

`Open::window_ms` / `max_bytes`
move the window, `settle_ms` the `end` wait, `reply_grace_ms` how long the
reply has once a leg broke under the caller (10 s, then `lost`).

Each side holds two sockets and two threads per session (the writer, and
a watcher of the agent's answer, which turns an early `401`/`422`/`503`
into the session's failure at once). The idle period (`idle_ms`, the
declaration's `timeout_ms`) is enforced by the daemon; a session never
outlives 24 h. `io().node()` reads this host's node from the agent's
`link.toml` (`None` when it is not linked).

**A daemon reconnect does not end a session.** The legs live on the data
socket, not on the control connection: when that connection drops, a call
that already has its ticket gets the reply grace (`reply_grace_ms`, 10 s)
and is `lost` only if no reply came by then — the session keeps moving
meanwhile. A call still waiting for its ticket fails at once (`lost`).
The Node SDK fails both at once.

Errors are `CallError { code, message, call_id }`, printed `code:
message`; `code` is one of `CallError::{TIMEOUT, REFUSED, TOO_LARGE, BUSY,
GONE, FAILED, LOST, NODE_OFFLINE, NODE_UNKNOWN, AMBIGUOUS, FORBIDDEN,
AGENT_TOO_OLD, DAEMON_TOO_OLD}` or a newer one. A status of the agent maps
as the Node SDK's: `401`/`403` forbidden, `404`/`409` gone, `408` timeout,
`413`/`422` too_large, `429` busy, `503` node_offline. `examples/stream_echo.rs`
is both sides.

## Lifecycle

```rust
if io().ready(Duration::from_millis(500)) {
    io().event("boot", Event::new());
}
io().flush(Duration::from_millis(500));
io().shutdown(Duration::from_millis(500));
```

| Call | What |
|---|---|
| `init(Init)` | Connect in the background. Idempotent, and usable again after `shutdown`. Without `THREEPM_CTL_SOCK` the instance stays disabled |
| `enabled()` | Is a connection *configured* (not: up)? |
| `source()`, `stats()` | The name this instance said hello with; `{sent, dropped, connects, closes, refusals}` |
| `ready(Duration)` / `ready_default()` | Wait for the connection — for a program that wants to know before it reports. `ready_default` waits `connect_wait_ms` (5 s) |
| `flush(Duration)` | Send the whole registry now. With a non-zero wait, wait for the connection, then for the bytes to leave the process |
| `shutdown(Duration)` | `flush`, then close. `serve()` returns |
| `serve()` | Block until the daemon asks the process to stop |

**A Rust process ends when `main` returns.** No thread of this SDK holds
it up (Node's `keepAlive` has no equivalent), so a worker whose only job
is to answer actions ends its `main` with `io().serve()`: it blocks until
the daemon's `shutdown`, until `Io::shutdown`, or until the transport
gives up for good. On an instance that was never `init`ed it returns at
once.

**There is no `atexit`.** A short-lived program must call
`io().flush(Duration::from_millis(500))` — or `shutdown` — before
returning, or its last batch never leaves. A long-running one needs
neither: the registry goes out every `flush_ms`.

`Init` carries `source`, `flush_ms` (1 s), `connect_wait_ms` (5 s),
`on_shutdown` and `env` (read the identity from a map instead of the
process environment), plus the test knobs the three other SDKs spell the
same way: `reconnect_min_ms`, `reconnect_max_ms`, `max_buffer_bytes`,
`max_refusals`, `refusal_window_ms`. `on_shutdown` runs on the reader
thread — signal, do not block.

A source names one process within the app
(`^[a-z0-9][a-z0-9_-]{0,31}$`). Named none, the process 3pm spawned takes
`main`, and a process *it* started takes `pid-<pid>`; two that both claim
`main` settle it — the refused one falls back once, and that refusal never
counts toward giving up.

## Bounds

Everything the SDK holds itself to, so that nothing it writes is ever
refused. `threepm_io::limits` exposes each one as a constant.

| | |
|---|---|
| Metric names | 200 per instance, `^[a-z][a-z0-9_.]{0,63}$` |
| Labels | 3 keys per metric, values ≤ 64 bytes, 50 label sets then an `overflow="true"` fold |
| Histogram | 128 non-empty buckets, `scale` 0..=4 |
| Batches | 256 entries and 32 KiB per `metrics` line |
| Events | 16 attributes, strings ≤ 256 bytes, `exc.type` ≤ 128, `exc.message` ≤ 1 KiB, `exc.stacktrace` ≤ 8 KiB, `fp` ≤ 64 |
| Actions | 32 per source, `description` ≤ 128 bytes, `timeout_ms` ≤ 10 min (30 s by default), `concurrent` ≤ 16, result ≤ 512 KiB and ≤ 64 levels |
| Streams | 16 sessions per instance (both directions), 16 calls in flight per connection (a session holds one until its reply), frames ≤ 1 MiB, idle ≤ 24 h, a reading leg gives up after 60 s without a byte |
| Telemetry buffer | 64 KiB; past it a `metrics` line is dropped and the next one carries the same totals (counters and buckets are cumulative) |
| Control buffer | 5 × 1 MiB of its own: the actions declaration and the replies can never be crowded out by metrics |
| Reconnection | 1 s → 30 s, doubling; the instance gives up for good after 5 refused `hello`s (the process outlived its incarnation) |

Longer strings are cut on a UTF-8 character boundary, not refused. A
value that cannot be fixed (a bad name, a reserved label) makes the handle
inert instead.

## Differences from the Node SDK

- **Not implemented yet**: `io.call` (calling another node's action)
  and **blob** action inputs and outputs (a file that crosses the fleet).
  A call carrying a blob input is refused with a message that says so; an
  output ticket is ignored, and the handler's committed file is what the
  caller gets. Streams are there (`io().stream` / `io().open`).
- **Streams are `send` / `recv`**, not Node's `session.input` writable and
  `frames()` iterator: no `pipe`, no `mjpeg`, and `stats()` carries no
  latency percentiles. A session's slot comes back at `close` (or when it
  drops, or fails), not when the reply arrives.
- **A control reconnect gives an open session its reply grace** rather
  than failing it at once (see Streams).
- **No `3pm:profile`.** `3pm profile` returns nothing for a Rust process;
  `3pm:` names are refused rather than reserved for a built-in.
- **No built-in runtime metrics.** Node's `builtins: true` (event-loop
  delay, heap, GC, handles) has no Rust counterpart; declare what you
  want with `metric(…)`.
- **`Event::name` names the exception type**, not the event. Node uses
  `opts.name` as the event name and takes `exc.type` from the JS error
  class; Rust has no error class, and `event(name, …)` already takes the
  name as its first argument. So `notify_error` always sends the event
  `error`, and `Event::name` overrides `exc.type`. For a custom event name
  carrying an exception:
  `io().event("my.failure", Event::new().exception(kind, message))`.
- **`inc(n)` and `mark(n)` take their step**, where Node defaults to 1.
- **`metric` callbacks run on the flush thread**, not on an event loop.
  Read a plain value there; do not block, and do not take a lock the rest
  of your program holds while writing metrics.
- **An `action` line read past its deadline is not run and not
  answered.** The daemon has already answered `timeout` and swept the
  call's file, so a handler would work on a path that is gone and a reply
  would be discarded. Node runs it anyway.
- **`action_reply` carries no `code`.** `too_large` and `busy` travel as
  plain failures whose message says why; the wire has no such field, and
  the daemon ignores the one Node sends.
- **A `3pm:` name is an inert `Registration`, not a throw**, and no
  telemetry call returns a `Result` — an SDK must not kill a worker over
  a name.
- **No `worker-*` source**: a Rust thread shares its process's connection.

## Debugging

`THREEPM_IO_DEBUG=1` in the app's environment prints one `[3pm io]` line
on stderr for every refusal — an invalid name, a reserved label, a cut
description, an action the table would not take, a source that fell back.
It is read once, at `Io::new`. `io().stats()` gives the counters the
transport keeps.

## Tests

`cargo test` in this directory: unit tests against a fake `ctl.sock`,
integration tests decoding every line the SDK writes with
`threepm-protocol` (the crate the daemon itself uses), the frozen
`limits.json` and `exphist-vectors.json` gates, and proptests over the
histogram. `cargo build --examples` builds the snippets above
(`examples/quickstart.rs`, `examples/worker.rs`) and `examples/e2e_client.rs`,
which `crates/threepm-cli/tests/e2e_io.rs` and `e2e_actions.rs` run under
a real daemon.

</document>

<document url="https://3pm.edgecr4ft.net/docs/backend/api.md" title="HTTP API: every route">

# 3pm-backend HTTP API

The route table of 3pm-backend, exactly as a backend serves it for an LLM on `GET /api/v1/routes` (build {"server_version":"0.64.0","link_min":5,"link_max":5,"store":{"metrics":false,"events":false,"interval_ms":0,"sent":0,"dropped":0,"catalog_over_cap":0,"errors":0,"retries":0},"feed_app_events_dropped":0,"link":{"frames_invalid":0,"crc_full_mismatch":0}}).

- Paths below are relative to **the backend's origin**: the hosted one is `https://fleet.edgecr4ft.net`, a self-hosted one is wherever `3pm-backend` listens (default `http://127.0.0.1:9616`). Not relative to this documentation site.
- Authentication: `Authorization: Bearer 3pmu_<account token>` (or the dashboard's session cookie); a linked node may use its own `3pma_` bearer within the scopes its owner granted.
- A live backend answers the same text on `GET /api/v1/routes`, one route in full on `?route=<name>`, the JSON table on `?format=json`. The same full routes are here:

  - [`info`](https://3pm.edgecr4ft.net/docs/backend/routes/info.md): `GET /api/v1/info`
  - [`routes`](https://3pm.edgecr4ft.net/docs/backend/routes/routes.md): `GET /api/v1/routes`
  - [`client`](https://3pm.edgecr4ft.net/docs/backend/routes/client.md): `GET /api/v1/client.js`
  - [`register`](https://3pm.edgecr4ft.net/docs/backend/routes/register.md): `POST /api/auth/register`
  - [`login`](https://3pm.edgecr4ft.net/docs/backend/routes/login.md): `POST /api/auth/login`
  - [`logout`](https://3pm.edgecr4ft.net/docs/backend/routes/logout.md): `POST /api/auth/logout`
  - [`me`](https://3pm.edgecr4ft.net/docs/backend/routes/me.md): `GET /api/auth/me`
  - [`createUserToken`](https://3pm.edgecr4ft.net/docs/backend/routes/createUserToken.md): `POST /api/auth/tokens`
  - [`revokeUserToken`](https://3pm.edgecr4ft.net/docs/backend/routes/revokeUserToken.md): `DELETE /api/auth/tokens/{tokenId}`
  - [`listBuckets`](https://3pm.edgecr4ft.net/docs/backend/routes/listBuckets.md): `GET /api/buckets`
  - [`createBucket`](https://3pm.edgecr4ft.net/docs/backend/routes/createBucket.md): `POST /api/buckets`
  - [`getBucket`](https://3pm.edgecr4ft.net/docs/backend/routes/getBucket.md): `GET /api/buckets/{clusterId}`
  - [`setBucketSettings`](https://3pm.edgecr4ft.net/docs/backend/routes/setBucketSettings.md): `POST /api/buckets/{clusterId}/settings`
  - [`bucketToken`](https://3pm.edgecr4ft.net/docs/backend/routes/bucketToken.md): `GET /api/buckets/{clusterId}/enroll-token`
  - [`events`](https://3pm.edgecr4ft.net/docs/backend/routes/events.md): `GET /api/buckets/{clusterId}/events`
  - [`listNodes`](https://3pm.edgecr4ft.net/docs/backend/routes/listNodes.md): `GET /api/buckets/{clusterId}/nodes`
  - [`approveNode`](https://3pm.edgecr4ft.net/docs/backend/routes/approveNode.md): `POST /api/buckets/{clusterId}/nodes/{nodeId}/approve`
  - [`revokeNode`](https://3pm.edgecr4ft.net/docs/backend/routes/revokeNode.md): `POST /api/buckets/{clusterId}/nodes/{nodeId}/revoke`
  - [`deleteNode`](https://3pm.edgecr4ft.net/docs/backend/routes/deleteNode.md): `DELETE /api/buckets/{clusterId}/nodes/{nodeId}`
  - [`setNodeAccess`](https://3pm.edgecr4ft.net/docs/backend/routes/setNodeAccess.md): `POST /api/buckets/{clusterId}/nodes/{nodeId}/access`
  - [`listTokens`](https://3pm.edgecr4ft.net/docs/backend/routes/listTokens.md): `GET /api/buckets/{clusterId}/tokens`
  - [`createToken`](https://3pm.edgecr4ft.net/docs/backend/routes/createToken.md): `POST /api/buckets/{clusterId}/tokens`
  - [`revokeToken`](https://3pm.edgecr4ft.net/docs/backend/routes/revokeToken.md): `DELETE /api/buckets/{clusterId}/tokens/{tokenId}`
  - [`queryMetrics`](https://3pm.edgecr4ft.net/docs/backend/routes/queryMetrics.md): `POST /api/buckets/{clusterId}/metrics/query`
  - [`metricsCatalog`](https://3pm.edgecr4ft.net/docs/backend/routes/metricsCatalog.md): `GET /api/buckets/{clusterId}/metrics/catalog`
  - [`metricLabelValues`](https://3pm.edgecr4ft.net/docs/backend/routes/metricLabelValues.md): `GET /api/buckets/{clusterId}/metrics/labels/{label}/values`
  - [`metricsInventory`](https://3pm.edgecr4ft.net/docs/backend/routes/metricsInventory.md): `GET /api/buckets/{clusterId}/metrics/inventory`
  - [`metricsByServer`](https://3pm.edgecr4ft.net/docs/backend/routes/metricsByServer.md): `GET /api/buckets/{clusterId}/metrics/servers`
  - [`metricsByApp`](https://3pm.edgecr4ft.net/docs/backend/routes/metricsByApp.md): `GET /api/buckets/{clusterId}/metrics/apps`
  - [`eventsQuery`](https://3pm.edgecr4ft.net/docs/backend/routes/eventsQuery.md): `POST /api/buckets/{clusterId}/events/query`
  - [`logsQuery`](https://3pm.edgecr4ft.net/docs/backend/routes/logsQuery.md): `POST /api/buckets/{clusterId}/logs/query`
  - [`insights`](https://3pm.edgecr4ft.net/docs/backend/routes/insights.md): `GET /api/buckets/{clusterId}/insights`
  - [`insight`](https://3pm.edgecr4ft.net/docs/backend/routes/insight.md): `GET /api/buckets/{clusterId}/insights/{incident}`
  - [`nodeAppMetrics`](https://3pm.edgecr4ft.net/docs/backend/routes/nodeAppMetrics.md): `GET /api/buckets/{clusterId}/nodes/{nodeId}/app-metrics`
  - [`nodeActions`](https://3pm.edgecr4ft.net/docs/backend/routes/nodeActions.md): `GET /api/buckets/{clusterId}/nodes/{nodeId}/actions`
  - [`nodeHost`](https://3pm.edgecr4ft.net/docs/backend/routes/nodeHost.md): `GET /api/buckets/{clusterId}/nodes/{nodeId}/host`
  - [`nodeProcessTrees`](https://3pm.edgecr4ft.net/docs/backend/routes/nodeProcessTrees.md): `GET /api/buckets/{clusterId}/nodes/{nodeId}/trees`
  - [`nodeProcessTree`](https://3pm.edgecr4ft.net/docs/backend/routes/nodeProcessTree.md): `GET /api/buckets/{clusterId}/nodes/{nodeId}/procs/{procId}/tree`
  - [`nodeAction`](https://3pm.edgecr4ft.net/docs/backend/routes/nodeAction.md): `POST /api/buckets/{clusterId}/nodes/{nodeId}/actions`
  - [`nodeShell`](https://3pm.edgecr4ft.net/docs/backend/routes/nodeShell.md): `POST /api/buckets/{clusterId}/nodes/{nodeId}/shell`
  - [`nodeForward`](https://3pm.edgecr4ft.net/docs/backend/routes/nodeForward.md): `POST /api/buckets/{clusterId}/nodes/{nodeId}/forward`
  - [`nodeListen`](https://3pm.edgecr4ft.net/docs/backend/routes/nodeListen.md): `POST /api/buckets/{clusterId}/nodes/{nodeId}/listen`
  - [`nodeCall`](https://3pm.edgecr4ft.net/docs/backend/routes/nodeCall.md): `POST /api/buckets/{clusterId}/nodes/{nodeId}/calls`
  - [`putCallArtifact`](https://3pm.edgecr4ft.net/docs/backend/routes/putCallArtifact.md): `PUT /api/buckets/{clusterId}/calls/{callId}/artifacts`
  - [`getCall`](https://3pm.edgecr4ft.net/docs/backend/routes/getCall.md): `GET /api/buckets/{clusterId}/calls/{callId}`
  - [`listArtifacts`](https://3pm.edgecr4ft.net/docs/backend/routes/listArtifacts.md): `GET /api/buckets/{clusterId}/artifacts`
  - [`artifactFile`](https://3pm.edgecr4ft.net/docs/backend/routes/artifactFile.md): `GET /api/buckets/{clusterId}/artifacts/{artifactId}/file`
  - [`putBlob`](https://3pm.edgecr4ft.net/docs/backend/routes/putBlob.md): `PUT /api/buckets/{clusterId}/blobs/{blobId}`
  - [`getBlob`](https://3pm.edgecr4ft.net/docs/backend/routes/getBlob.md): `GET /api/buckets/{clusterId}/blobs/{blobId}`
  - [`viewBlob`](https://3pm.edgecr4ft.net/docs/backend/routes/viewBlob.md): `POST /api/buckets/{clusterId}/blobs/{blobId}/view`
  - [`enroll`](https://3pm.edgecr4ft.net/docs/backend/routes/enroll.md): `POST /api/v1/enroll`
  - [`link`](https://3pm.edgecr4ft.net/docs/backend/routes/link.md): `GET /api/v1/link`
  - [`pushLogs`](https://3pm.edgecr4ft.net/docs/backend/routes/pushLogs.md): `POST /api/buckets/{clusterId}/logs`
  - [`indexPage`](https://3pm.edgecr4ft.net/docs/backend/routes/indexPage.md): `GET /`
  - [`bucketPage`](https://3pm.edgecr4ft.net/docs/backend/routes/bucketPage.md): `GET /b/{clusterId}`
  - [`bucketSettingsPage`](https://3pm.edgecr4ft.net/docs/backend/routes/bucketSettingsPage.md): `GET /b/{clusterId}/settings`
  - [`asset`](https://3pm.edgecr4ft.net/docs/backend/routes/asset.md): `GET /assets/{path}`

---

tokens:
  claude: 15159
  o200k: 9914
api: 3pm-backend 0.64.0
about: "the fleet backend's HTTP API. Paths are relative to the origin that served this text; bodies and answers are JSON; an error is {error: {code, message}}"
auth: "auth=user: the route needs an account — Authorization: Bearer 3pmu_<token>, or the dashboard's session cookie — and one under /api/buckets/{clusterId} a bucket that account owns (another's answers 404); auth=public: no credential. A route may also be open to a linked node's own bearer (Authorization: Bearer 3pma_<token>, minted in every welcome): GET /api/v1/routes?route=<name> names the fleet scope it then needs (fleet:list, fleet:read, fleet:invoke), granted per node by its owner"
more: "GET /api/v1/routes?route=<name> answers one route in full: its details, its bounds, the types of its body and answer. ?format=json answers the whole table as JSON. other: routes that are not for an LLM (pages, the JavaScript client module, the agent's WebSockets). recipes: a body for each common question and how to read its answer, then the rules no type says. in_full: the routes given whole after them, their types once — what a bucket holds, then how to query it"
routes[45]{name,method,path,body,returns,auth,doc}:
  info,GET,/api/v1/info,"",Info,public,"Versions, the accepted link range, how the store is doing and what the link got wrong; no credentials needed."
  routes,GET,/api/v1/routes,"",ApiDescriptor,public,"Every route this backend serves: an index for an LLM by default, with `metricsInventory` and `queryMetrics` given whole; one route in full with `?route=<name>` (details, bounds, body and answer types), the JSON table with `?format=json` or `Accept: application/json` (what this client sends); `?version=` defaults to `latest`."
  register,POST,/api/auth/register,Register,Registered,public,"Create an account: its token comes back once (there is no other way in, keep it) with a session cookie. Only the first account unless the backend runs with `--registration open`; that first account adopts every bucket without an owner."
  login,POST,/api/auth/login,Login,LoggedIn,public,"Trade an account's token for a session cookie — the dashboard's way in; a script sends the token as `Authorization: Bearer` on each request instead."
  logout,POST,/api/auth/logout,"",void,user,Close this session and drop its cookie; the token stays valid.
  me,GET,/api/auth/me,"",Me,user,"The account behind this request: its name, its tokens (never their secrets) and how many buckets it owns."
  createUserToken,POST,/api/auth/tokens,CreateUserToken,UserTokenCreated,user,"Another token for the account (a laptop, a CI job); the secret comes back once."
  revokeUserToken,DELETE,/api/auth/tokens/{tokenId},"",void,user,Revoke one of the account's tokens and close every session it opened; never the last one (409 `ERR_LAST_TOKEN`).
  listBuckets,GET,/api/buckets,"",BucketView[],user,"The caller's buckets, oldest first."
  createBucket,POST,/api/buckets,CreateBucket,BucketView,user,"Create a bucket: a cluster id to paste, and nodes to enroll into it."
  getBucket,GET,/api/buckets/{clusterId},"",BucketDetail,user,"One bucket with its nodes, last telemetry and history — the shape the feed opens with; `history`, `match` and `view` cut it as they cut the feed's `init`."
  setBucketSettings,POST,/api/buckets/{clusterId}/settings,SetBucketSettings,BucketView,user,"Set the bucket's settings — `approval`: whether a host that links with a reusable token waits for `approveNode` (turning it off approves the hosts already waiting); `require_access_rights`: off, every active node may place a call on every other, as if each held `fleet:invoke`, `fleet:list` and `fleet:read`; `record_shells` (on by default): the screen of each shell session is kept as an asciicast artifact of its call. Answers the bucket as it now stands."
  bucketToken,GET,/api/buckets/{clusterId}/enroll-token,"",BucketToken,user,"The bucket's permanent enrollment token and its paste line: the same at every read, never expiring, any number of hosts, not revocable. Created on the first read; its hosts wait for `approveNode` unless the bucket's `approval` is off."
  events,GET,/api/buckets/{clusterId}/events,"",text/event-stream,user,"The bucket's live feed: `init`, then `node` / `telemetry` / `emit` / `call` / `artifact` — a call's state and an artifact's id and sizes only, never params, results or summaries (`getCall`, `listArtifacts`); `history`, `match` and `view` cut it to what the page draws."
  listNodes,GET,/api/buckets/{clusterId}/nodes,"",NodeView[],user,"Every enrolled node of the bucket, presence layered on."
  approveNode,POST,/api/buckets/{clusterId}/nodes/{nodeId}/approve,"",NodeView,user,Admit a pending node. Idempotent.
  revokeNode,POST,/api/buckets/{clusterId}/nodes/{nodeId}/revoke,"",NodeView,user,"Revoke a node, cut its live session and drop the bearer it was using. Idempotent."
  deleteNode,DELETE,/api/buckets/{clusterId}/nodes/{nodeId},"",void,user,"Delete a node: its row, its calls and its artifacts go, its live session is cut and the feed sends `reset`. Unlike `revokeNode` nothing is held against its key — the host may enroll again, as a new node."
  setNodeAccess,POST,/api/buckets/{clusterId}/nodes/{nodeId}/access,SetNodeAccess,NodeView,user,"What this node may ask of the bucket with its own bearer (`scopes`), who may place a call on it (`callers`), and what it lets go agent to agent with a LAN neighbour (`direct`); no scope by default, and an empty `callers` accepts every node holding `fleet:invoke`. Replaces each list it names, and takes effect on the node's next request."
  listTokens,GET,/api/buckets/{clusterId}/tokens,"",TokenView[],user,Every enrollment token of the bucket; never the secrets.
  createToken,POST,/api/buckets/{clusterId}/tokens,CreateToken,TokenCreated,user,Mint an enrollment token. The secret and its paste line are returned once.
  revokeToken,DELETE,/api/buckets/{clusterId}/tokens/{tokenId},"",void,user,"Revoke an enrollment token; the bucket's permanent one (`bucketToken`) is refused, 409."
  queryMetrics,POST,/api/buckets/{clusterId}/metrics/query,MetricsQuery,MetricsResult,user,"Time series for charts: metrics, filters, aggregation and step, scoped to the bucket."
  metricsCatalog,GET,/api/buckets/{clusterId}/metrics/catalog,"",MetricDef[],user,"Every metric a query may name: id, kind, unit, labels, default time function."
  metricLabelValues,GET,/api/buckets/{clusterId}/metrics/labels/{label}/values,"",LabelValues,user,"The values one label takes in this bucket over the last two UTC days, at most 5000."
  metricsInventory,GET,/api/buckets/{clusterId}/metrics/inventory,"",MetricsInventory,user,"What a dashboard or an LLM starts from, in one call: the metrics the bucket holds series for, with their descriptions, its hosts and their processes — text for an LLM by default, JSON with `?format=json` or `Accept: application/json` (what this client sends)."
  metricsByServer,GET,/api/buckets/{clusterId}/metrics/servers,"",MetricsByServer,user,"Which metric is stored on which server, for which app: each node with its host's own metrics, then each of its apps with theirs — text for an LLM by default, JSON with `?format=json` or `Accept: application/json` (what this client sends)."
  metricsByApp,GET,/api/buckets/{clusterId}/metrics/apps,"",MetricsByApp,user,"Which metric each app has on which server: each app with the servers it has series on and, on each, their metrics — text for an LLM by default, JSON with `?format=json` or `Accept: application/json` (what this client sends)."
  eventsQuery,POST,/api/buckets/{clusterId}/events/query,EventsQuery,EventsResult,user,"The persisted events of the bucket, newest first: a range, exact filters (kind, app, level, event_name…), a limit."
  logsQuery,POST,/api/buckets/{clusterId}/logs/query,LogsQuery,LogsResult,user,"The logs the bucket's nodes shipped over a range: their sources, the templates with their counts (worst first) and the shipped lines (newest first), filtered by node, source, owner, severity, template or text."
  insights,GET,/api/buckets/{clusterId}/insights,"",InsightsList,user,"What deserves a look on this fleet: incidents ranked by the insights detectors, best first — text for an LLM by default, JSON with `?format=json` or `Accept: application/json` (what this client sends)."
  insight,GET,/api/buckets/{clusterId}/insights/{incident},"",InsightDetail,user,"One incident of `insights` and the findings it groups, each with its detector, series, score and summary — text for an LLM by default, JSON with `?format=json` or `Accept: application/json`."
  nodeAppMetrics,GET,/api/buckets/{clusterId}/nodes/{nodeId}/app-metrics,"",AppMetricsView,user,"The application metrics of every instance of the node, as last received: sources, series, values and histogram buckets."
  nodeActions,GET,/api/buckets/{clusterId}/nodes/{nodeId}/actions,"",NodeActionsView,user,"The custom actions the apps of the node declare (`io.action`), per instance and source, with descriptions and timeouts — what `nodeAction` may run; none of the metrics, so a page may poll it all the time."
  nodeHost,GET,/api/buckets/{clusterId}/nodes/{nodeId}/host,"",NodeHostView,user,"The node's host as it is now, whole: the gauges and counters with the identity (hostname, OS, totals, interface and mount names) the feed's ticks blank; 404 before its first frame, and for a node that sends no host."
  nodeProcessTrees,GET,/api/buckets/{clusterId}/nodes/{nodeId}/trees,"",ProcessTreesView,user,"The process trees of every instance of the node, as last received — every process and the folded picture of `3pm ls --tree` — with the count of trees refused; what the dashboard's panel polls."
  nodeProcessTree,GET,/api/buckets/{clusterId}/nodes/{nodeId}/procs/{procId}/tree,"",ProcessTreeView,user,"The process tree of one instance of the node, as last received: every process, and the same folded picture as `3pm ls --tree`; 404 while the backend holds none — an instance running alone, or before its first tree."
  nodeAction,POST,/api/buckets/{clusterId}/nodes/{nodeId}/actions,NodeActionBody,CallView,user,"Run a custom action an app of the node declared (`io.action`), on every instance of the app or on one, optionally carrying bytes in and out; the call is answered as soon as it is placed — follow it with `getCall` or the feed's `call` events."
  nodeShell,POST,/api/buckets/{clusterId}/nodes/{nodeId}/shell,NodeShellBody,CallView,user,"Open an interactive login shell on the node (`3pm fleet ssh`), or run one command there (`command`, `3pm fleet ssh <node> -- <cmd>`): the answer is the call, whose `input.blob` you `PUT` keystrokes (or stdin) to and `output.blob` you `GET` the screen (or stdout and stderr) from; it replies `exited` when the shell or the command ends. Never from a page on another site."
  nodeForward,POST,/api/buckets/{clusterId}/nodes/{nodeId}/forward,NodeForwardBody,CallView,user,"Open one TCP connection from the node to `host:port` (`3pm fleet ssh -L`, one call per local connection): the answer is the call, whose `input.blob` you `PUT` the bytes to send and `output.blob` you `GET` the bytes that come back; it replies `exited` once both ways are closed. Never from a page on another site."
  nodeListen,POST,/api/buckets/{clusterId}/nodes/{nodeId}/listen,NodeListenBody,CallView,user,"Make the node listen on a port for you (`3pm fleet ssh -R`): the answer is the call, whose sealed legs carry the port it bound and one `accepted` message per connection, each of which you take over with a `nodeForward` naming it (`accepted`); it replies `exited` when you close it. Never from a page on another site."
  nodeCall,POST,/api/buckets/{clusterId}/nodes/{nodeId}/calls,NodeCallBody,CallView,user,"Send one generic command to the node's agent — `{\"op\": \"restart\", \"target\": {\"name\": \"api\"}}`, `stop` or `ping` (an action goes through `nodeAction`); answered as soon as it is placed — follow it with `getCall`."
  getCall,GET,/api/buckets/{clusterId}/calls/{callId},"",CallView,user,"One remote command: its state (`sent`, `acked`, `replied`, `unacked`, `refused`, `lost`), who placed it, the blobs it carries, its result once replied (`result.ok` says the action ran) and the artifacts it left. A node's bearer sees the calls it placed itself and nothing else."
  listArtifacts,GET,/api/buckets/{clusterId}/artifacts,"",ArtifactView[],user,"The files actions left (profiles, dumps), newest first, at most 200; `node` (a node id) and `app` (a name) narrow. Kept 30 days and 1 GB per bucket. A node's bearer sees what its own calls left and nothing else."
  viewBlob,POST,/api/buckets/{clusterId}/blobs/{blobId}/view,ViewBlobBody,ViewBlobUrl,user,"Mint the one credential a browser tag can carry: a `getBlob` URL with `?k=` on it, good for sixty seconds, for this blob, for reading — `{as: \"mjpeg\"}` adds the rendering an `<img>` needs."
other[12]{name,kind,path}:
  client,static,/api/v1/client.js
  putCallArtifact,agent-http,/api/buckets/{clusterId}/calls/{callId}/artifacts
  artifactFile,static,/api/buckets/{clusterId}/artifacts/{artifactId}/file
  putBlob,static,/api/buckets/{clusterId}/blobs/{blobId}
  getBlob,static,/api/buckets/{clusterId}/blobs/{blobId}
  enroll,agent-http,/api/v1/enroll
  link,agent-ws,/api/v1/link
  pushLogs,agent-http,/api/buckets/{clusterId}/logs
  indexPage,static,/
  bucketPage,static,/b/{clusterId}
  bucketSettingsPage,static,/b/{clusterId}/settings
  asset,static,/assets/{path}
in_full[3]: metricsInventory,queryMetrics,eventsQuery

## Recipes

A body for each common question, and how to read its answer. Every body passes the validation its route applies; the metrics are static catalog lines, the node ids placeholders.

### The value right now — which processes are down

POST /api/buckets/{clusterId}/metrics/query (queryMetrics)
{"range":{"from":"now-10m"},"queries":[{"ref":"UP","metric":"process.up"}],"options":{"mode":"summary"}}
read: one series per instance; `summary.last` is its latest value (`process.up`: 1 up, 0 down). Keep the rows whose `last` is 0 yourself: a query has no value predicate.

### A total over a period — restarts per app over the last 7 days

POST /api/buckets/{clusterId}/metrics/query (queryMetrics)
{"range":{"from":"now-7d"},"queries":[{"ref":"R","metric":"process.restarts","time":{"fn":"increase","window":"7d"},"aggregate":{"fn":"sum","by":["app"]}}],"options":{"mode":"summary"}}
read: `summary.last` per app is the total: the window is the whole range, so the last point covers it. Never add the points up — each one is already a window's total.

### The top N — the 5 apps using the most CPU over the last hour

POST /api/buckets/{clusterId}/metrics/query (queryMetrics)
{"range":{"from":"now-1h"},"queries":[{"ref":"CPU","metric":"process.cpu.utilization","aggregate":{"fn":"sum","by":["app"]}}],"options":{"max_series":5,"sort":"avg","mode":"summary"}}
read: the 5 highest by average, best first, `summary.avg` each. `truncated: true` says more existed; `series_total` is then a floor (the store is asked for `max_series` + 1), not a count.

### A percentage from two metrics — memory used per host, in %

POST /api/buckets/{clusterId}/metrics/query (queryMetrics)
{"range":{"from":"now-6h"},"queries":[{"ref":"U","metric":"system.memory.used","hidden":true},{"ref":"T","metric":"system.memory.total","hidden":true}],"formulas":[{"ref":"PCT","expr":"U / T * 100","unit":"percent"}]}
read: one `PCT` series per node: a formula pairs the series of its refs on equal label sets (a lone series applies to all). Add `"options":{"mode":"summary"}` for one number per host — formulas run there too.

### A counter as a rate — network receive per host and interface

POST /api/buckets/{clusterId}/metrics/query (queryMetrics)
{"range":{"from":"now-3h"},"queries":[{"ref":"RX","metric":"system.interface.receive","time":{"fn":"rate"}}]}
read: bytes per second, one series per `node` and `iface`. `rate` is a counter's default; `increase` gives a count per window instead.

### One host by its name — resident memory per app on that host

POST /api/buckets/{clusterId}/metrics/query (queryMetrics)
{"range":{"from":"now-6h"},"queries":[{"ref":"RSS","metric":"process.memory.rss","filters":[{"label":"node","op":"eq","value":"3pmn_aaaaaaaaaaaaaaaa"}],"aggregate":{"fn":"sum","by":["app"]}}]}
read: a filter names a host by `node`, its node_id, never by hostname: find the hostname in the inventory's `hosts[]` and put its `node_id` in `value`.

### One value of a label — the 1-minute load average per host

POST /api/buckets/{clusterId}/metrics/query (queryMetrics)
{"range":{"from":"now-24h"},"queries":[{"ref":"L","metric":"system.load","filters":[{"label":"window","op":"eq","value":"1"}]}]}
read: what a label holds is the inventory's `label_values` (`system.load`'s `window`: 1, 5, 15); a filter takes a value as written there.

### Error events — the error-level application events of the last day

POST /api/buckets/{clusterId}/events/query (eventsQuery)
{"range":{"from":"now-24h"},"filters":{"kind":"app","level":"error"},"limit":100}
read: `events`, newest first; `truncated: true` says more matched than `limit`. An empty list is an answer: none happened.

### Link two machines, step 1 — let node A place calls (on the caller)

POST /api/buckets/{clusterId}/nodes/{nodeId}/access (setNodeAccess)
{"scopes":["fleet:list","fleet:read","fleet:invoke"]}
read: sent to `/nodes/{A}/access`: the `NodeView` of A with its new `scopes`. `fleet:invoke` alone is what `io.call` / `io.open` / `3pm fleet call` need; the two others let A see the fleet. `listNodes` gives each machine's `node_id` from its `hostname`. A list left out of the body is kept as it is; a list sent replaces the old one whole.

### Link two machines, step 2 — let node A call one app's actions on node B (on the target)

POST /api/buckets/{clusterId}/nodes/{nodeId}/access (setNodeAccess)
{"callers":[{"from":"3pmn_aaaaaaaaaaaaaaaa","app":"kiosk","action":"show"},{"from":"3pmn_aaaaaaaaaaaaaaaa","from_app":"mac-camera","app":"tabloid-mac_camera","action":"video"}]}
read: sent to `/nodes/{B}/access`: the `NodeView` of B with its `callers`. An empty `callers` lets in every node holding `fleet:invoke`; once it holds a rule, only what a rule admits goes through (403 `ERR_FORBIDDEN` "not allowed to call <app>/<action>" otherwise). A field left out is `*`; `from_app` is the 3pm app name of the caller as its daemon declares it; a stream action (`io.stream`) is admitted like any other. The list replaces the old one: read B's `callers` in `listNodes` first and send them back with the new rule. A node calling itself needs both steps on itself.

### Notes

- Legends: every series carries `node` = a node_id; the inventory's `hosts[]` gives its hostname, and its `processes` the `app` / `instance` / `ns` values.
- No value predicate: a query cannot keep the series above or below a threshold (`up == 0`, disk > 90 %) — ask `mode: summary` and filter the rows yourself.
- A bool gauge (unit `bool`, as `process.up`) reads per instance: its `avg` across instances is a share of them, not a state.
- An `app.*` metric: the inventory's `apps` column lists the apps that emit it; when it lists several, put `app` in `aggregate.by` or in a filter.

# GET /api/buckets/{clusterId}/metrics/inventory (metricsInventory)

What a dashboard or an LLM starts from, in one call: the metrics the bucket holds series for, with their descriptions, its hosts and their processes — text for an LLM by default, JSON with `?format=json` or `Accept: application/json` (what this client sends).

- auth: an account — Authorization: Bearer 3pmu_<token>, or the dashboard's session cookie
- answer: MetricsInventory (JSON)

## Details

Text for an LLM by default; the generated client asks for the JSON (`Accept: application/json`), and so does `?format=json`. The JSON is `{ tokens, metric_groups, label_values, hosts }`, shaped for few tokens: nothing empty, nothing derivable, no join to make. `tokens` comes first: what the whole answer costs an LLM, estimated from its bytes with the ratio measured for its shape on Claude's tokenizer (about 2.4 bytes a token; OpenAI's counts 1.5–1.7× fewer). `metric_groups` are the catalog lines that have at least one series stored over the last two UTC days — the catalog says what a query may name, this says what would draw something — grouped by the labels they carry: `{ labels, metrics: [{ id, kind, unit?, default_time?, description?, apps? }] }`. `default_time` is left out when it is the kind's (`avg_over_time` for a gauge, `rate` for a counter, `quantile` for a histogram), `unit` and `description` when empty; a line is an application's exactly when its id starts with `app.`, and `apps` lists the apps whose series carry it (one read of every application series but a histogram's buckets, at most 5000: past that, the lists may be short). `label_values` are `{ label, values, more? }`: what every other label of those lines holds over the same window (not `node`, `app`, `instance`, `ns` — `hosts` has them — nor `pid`), sorted, at most 50 values, `more` counting the rest. `hosts` are every enrolled node, `{ node_id, hostname, os, arch, state, processes: [{ app, instance, ns? }] }`: the instances the store holds a series for over the same window, label values that go into a query's `filters` as they are, next to `node` = `node_id`. Every series carries `node` = a `node_id`: a query names a host by it, never by hostname, and `hosts` turns a legend's `node` into a name. `?detail=index` leaves the descriptions out, about 40 % fewer tokens. The text (`?format=llm`, the default) is the same answer as TOON-style `text/plain` after an `about:` line saying how the lists join, about a fifth fewer tokens than the JSON (more than half fewer with `detail=index`): a header per list with its count and fields (`metrics[140]{id,kind,unit,default_time,description}:`), one row per entry, every `default_time` spelled out. An unknown parameter or value is a 400. The store is read for at most 5000 series names and 5000 processes. Without `--metrics-url` the route answers 409 `ERR_STORE_DISABLED`.

# POST /api/buckets/{clusterId}/metrics/query (queryMetrics)

Time series for charts: metrics, filters, aggregation and step, scoped to the bucket.

- auth: an account — Authorization: Bearer 3pmu_<token>, or the dashboard's session cookie
- node: also open to a node's own bearer (3pma_…) when its row is granted fleet:read; that node's bucket only, and for the calls it placed itself
- body: MetricsQuery (JSON)
- answer: MetricsResult (JSON)

## Details

Discovery first: `metricsCatalog` gives every metric id, the labels each one carries and its `default_time`; `metricLabelValues` gives what a label holds in this bucket. A metric the catalog lacks, a label the metric does not carry, `rate` on a gauge, a regex that does not compile or any unknown field is a 400 `ERR_BAD_REQUEST` naming what was wrong.

Body `{ range, queries, formulas?, options? }`:

- `range` = `{ from, to?, step? }`. `from` and `to` are epoch milliseconds or `now` / `now-<duration>` (`now-90m`, `now-7d`); `to` defaults to `now`. `step` is `30s` / `5m` / `1h` or milliseconds, and is optional: left out, it is derived from `options.max_points`. Both ends are aligned to it, and the answer says which step was used (`range.step_ms`).
- `queries` = 1 to 8 `{ ref, metric, filters?, time?, aggregate?, hidden? }`. `ref` matches `^[A-Z][A-Z0-9]{0,7}$` and is unique across queries and formulas. `metric` is a catalog id (`process.cpu.utilization`, or `app.<name>` for an application's own). `filters` are AND-ed `{ label, op, value | values }` over `eq | neq | in | nin | regex | nregex`. `time` is the time function and its window — `rate` / `increase` / `delta` on a counter, the `*_over_time` family on a gauge, `quantile` (with `q`) on a histogram, `raw` on either — the window defaulting to twice the step. `aggregate` = `{ fn: sum | avg | min | max | count | quantile, by: [labels] }` collapses the series that share `by`; without it, one series comes back per label combination the metric carries.
- `formulas` = up to 8 `{ ref, expr }` over the other refs: `A / B * 100`, with `+ - * /` and parentheses. A query marked `hidden: true` is computed for a formula and not returned.
- `options`: `max_points` (the step is raised until the range fits), `max_series` with `sort` (`avg` or `last`) deciding which are kept, `fill` (`null` | `zero` | `previous`), `mode` (`series`, or `summary` to drop the values and keep min/max/avg/last/count per series), `explain` to get the compiled expression back.

Answer `{ range, times, results, took_ms }`: one `times[]` shared by every series, then one entry per visible ref with `series[].labels`, `series[].values` aligned on `times`, `series[].summary`, and `series_total` / `truncated` when the ranking cut some off.

The bounds are `limits.queryMetrics` of `GET /api/v1/routes`, and the step may not go below the store's resolution (`store.interval_ms` of `GET /api/v1/info`). They bound the grid, not the window, and what a step is measured against is `options.max_points` — 600 by default, 2000 at most: 30 days at `5m` is 8 640 points and is refused whatever it is set to. The same 30 days draws at `2h` on the default and at `30m` (1 442 points) with `max_points: 2000`; a `5m` step fits two days on the default and just under seven at 2000. Leave `step` out and the finest step of the ladder that fits is picked for you.

Example — 24 hours per app: the peak resident memory, and the average CPU turned into a percentage by a formula (the ratio itself computed and not returned). Every metric it names is a static catalog line, so it runs against any bucket:

{"range":{"from":"now-24h"},"queries":[{"ref":"CPU","metric":"process.cpu.utilization","filters":[{"label":"app","op":"in","values":["api","worker"]}],"aggregate":{"fn":"avg","by":["app"]},"hidden":true},{"ref":"RSS","metric":"process.memory.rss","aggregate":{"fn":"max","by":["app"]}}],"formulas":[{"ref":"PCT","expr":"CPU * 100","unit":"percent"}],"options":{"max_points":300}}

## Bounds

- default_points: 600
- default_series: 200
- max_body_bytes: 65536
- max_filters: 16
- max_formulas: 8
- max_points: 2000
- max_queries: 8
- max_range_ms: 2678400000
- max_series: 2000
- max_value_len: 256
- max_values: 64
- step_floor_from: "info.store.interval_ms"

# POST /api/buckets/{clusterId}/events/query (eventsQuery)

The persisted events of the bucket, newest first: a range, exact filters (kind, app, level, event_name…), a limit.

- auth: an account — Authorization: Bearer 3pmu_<token>, or the dashboard's session cookie
- body: EventsQuery (JSON)
- answer: EventsResult (JSON)

## Details

Body `{ range, filters?, limit? }`. `range` is the metrics query's, minus `step` (there is no grid): `from` / `to` as epoch milliseconds or `now-<duration>`. `filters` are exact matches, AND-ed, on the fields the store writes — `kind` (`process`, `daemon_shutdown`, `app`, `action`, `shell`), `node`, `app`, `instance`, `src`, `level`, `event_name`, `fingerprint`, `call` (every line of one action or shell session) — no operators, no regex. `limit` is 1 to 1000 (default 200).

Answer `{ events, truncated, took_ms }`, newest first, each event the record as stored (`_time`, `_msg`, `kind`, and the fields above); `truncated` says more matched than `limit`. Bounds: `limits.eventsQuery` of `GET /api/v1/routes`.

Example — the last 50 error-level application events of one app over a day:

{"range":{"from":"now-24h"},"filters":{"kind":"app","app":"api","level":"error"},"limit":50}

## Bounds

- default_limit: 200
- max_body_bytes: 16384
- max_limit: 1000
- max_range_ms: 2678400000
- max_value_len: 256

## Types

```ts
/** `metricsInventory(clusterId)`: what a dashboard or an LLM starts from. The route answers TOON-style text by default; this client asks for this JSON (`Accept: application/json`), as does `?format=json`. `?detail=index` leaves the descriptions out. */
interface MetricsInventory {
  /** What this whole answer costs an LLM; first, so a reader knows before the rest. */
  tokens: TokenCount;
  /** The catalog lines with a series stored over the last two UTC days, grouped by the labels they carry; groups and lines in catalog order. */
  metric_groups: MetricGroup[];
  /** What every other label of those lines holds over the same window — not `node`, `app`, `instance`, `ns` (`hosts` has them) nor `pid` —, in the order the lines name them; a label that holds nothing is left out. */
  label_values: InventoryLabel[];
  /** Every enrolled node of the bucket, whatever its state, with its processes; a node only the store knows comes last, `state: "unknown"`. Every series carries `node` = `node_id`: a query names a host by it, never by hostname, and this list names a legend's `node`. */
  hosts: InventoryHost[];
}

/** `POST /api/buckets/{id}/metrics/query`. */
interface MetricsQuery {
  range: TimeRange;
  /** 1–8. */
  queries: QuerySpec[];
  /** ≤ 8. */
  formulas?: Formula[];
  options?: QueryOptions;
}

interface MetricsResult {
  range: ResultRange;
  /** One timestamp per point, shared by every series. */
  times: EpochMs[];
  results: ResultSet[];
  took_ms: number;
}

/** `POST /api/buckets/{id}/events/query`: the persisted events, newest first. */
interface EventsQuery {
  /** `step` is ignored. */
  range: TimeRange;
  filters?: EventFilters;
  /** ≤ 1000, default 200. */
  limit?: number;
}

interface EventsResult {
  /** The stored fields as they are (`_time`, `_msg`, `kind`, `app`, `level`, `a.*`, `exception.*`, `fingerprint`, `repeat`…). */
  events: Record<string, unknown>[];
  /** More than `limit` matched. */
  truncated: boolean;
  took_ms: number;
}

/** What an answer costs to put in a prompt: the first field of every answer that carries it (`metricsInventory`, `GET /api/v1/routes`). */
interface TokenCount {
  /** Claude's tokenizer, estimated from the bytes with a ratio measured per shape through the `count_tokens` API — within a few percent. */
  claude: number;
  /** OpenAI's `o200k_base`, exact. Claude counts 1.5–1.7× more on this backend's answers. */
  o200k: number;
}

interface MetricGroup {
  /** What `filters` and `aggregate.by` may name for every metric of the group. */
  labels: QueryLabel[];
  metrics: InventoryMetric[];
}

interface InventoryLabel {
  label: QueryLabel;
  /** Sorted, at most 50: values a filter takes as written. */
  values: string[];
  /** How many more there were past the 50; absent when none. */
  more?: number;
}

interface InventoryHost {
  node_id: NodeId;
  hostname: string;
  os: string;
  arch: string;
  /** As `NodeView.state`; `"unknown"` for a node only the store knows. */
  state: DisplayState | "unknown";
  /** The instances the store holds a series for, sorted: label values, ready for `filters` next to `node` = `node_id`. */
  processes: InventoryProcess[];
}

interface TimeRange {
  from: TimeValue;
  /** Default `now`. */
  to?: TimeValue;
  /** `30s`, `5m`, `1h` or milliseconds (what `step_ms` answers); omitted ⇒ derived from `options.max_points`. Both ends are aligned to it. */
  step?: DurationValue;
}

interface QuerySpec {
  /** `^[A-Z][A-Z0-9]{0,7}$`, unique across queries and formulas. */
  ref: string;
  /** A catalog id, `process.cpu.utilization`. */
  metric: string;
  /** AND-ed; at most 16. */
  filters?: Filter[];
  /** Default: the catalog's `default_time` for the metric. */
  time?: TimeFn;
  aggregate?: Aggregate;
  /** Computed for a formula, not returned. */
  hidden?: boolean;
}

interface Formula {
  ref: string;
  /** Refs, numbers, `+ - * /` and parentheses: `A / B * 100`. */
  expr: string;
  unit?: string;
}

interface QueryOptions {
  /** 2–2000, default 600: the step is raised until the range fits. Every bound here is also `limits.queryMetrics` of `GET /api/v1/routes`, which is generated from the validator. */
  max_points?: number;
  /** 1–2000, default 200: the series kept are the top ones by `sort` over the whole range (the engine ranks, the backend re-ranks the same way). */
  max_series?: number;
  /** What ranks series for `max_series`, descending; default `avg`. */
  sort?: "avg" | "last";
  /** Default `null`. */
  fill?: "null" | "zero" | "previous";
  /** `series` (default) or `summary` (no values). */
  mode?: "series" | "summary";
  /** Add `results[].expr`: the expression as compiled for the engine. */
  explain?: boolean;
}

interface ResultRange {
  /** Aligned to `step_ms`: `from` rounded down, `to` rounded up. */
  from: EpochMs;
  to: EpochMs;
  step_ms: number;
}

/** Milliseconds since the Unix epoch. */
type EpochMs = number;

interface ResultSet {
  ref: string;
  /** The catalog id; absent on a formula. */
  metric?: string;
  kind: MetricKind;
  /** `ratio`, `bytes`, `seconds`, `count`, `bool`, or the formula's; absent on a formula that gave none. */
  unit?: string;
  series: Series[];
  /** The series the engine answered, at most `max_series + 1`: `truncated` is exact, this is a floor, not the bucket's count. */
  series_total: number;
  truncated: boolean;
  /** With `options.explain`: the expression as compiled for the engine. */
  expr?: string;
}

/** Exact matches on the stored fields. */
interface EventFilters {
  /** `process`, `daemon_shutdown`, `app`, `action`, `shell`. */
  kind?: string;
  node?: NodeId;
  app?: string;
  instance?: number;
  src?: string;
  level?: string;
  event_name?: string;
  fingerprint?: string;
  /** A call id: every line of one action or shell session. */
  call?: string;
}

/** The labels a query may filter or group on; the bucket itself is never one of them. */
/** A built-in label, or an application's own label name (`^[a-z][a-z0-9_]{0,31}$`, not reserved) on a metric that carries it. */
type QueryLabel = "node" | "app" | "instance" | "ns" | "mount" | "window" | "src" | "core" | "iface" | "blockdev" | "sensor" | "gpu" | "array" | "proc" | (string & {});

/** A catalog line, minus what is empty or derivable: it is an application's exactly when `id` starts with `app.`. */
interface InventoryMetric {
  id: string;
  kind: MetricKind;
  /** Absent when the metric has none. */
  unit?: string;
  /** Absent when it is the kind's: `avg_over_time` for a gauge, `rate` for a counter, `quantile` for a histogram. */
  default_time?: TimeFnName;
  /** Absent when the application gave none, and with `?detail=index`. */
  description?: string;
  /** An application line only: the apps whose series carry it, sorted — absent when none did. With several, put `app` in `aggregate.by` or a filter. */
  apps?: string[];
}

/** `3pmn_…` — a node. */
type NodeId = string;

/** `NodeView.state`: the enrollment state, or the presence of an active node. */
type DisplayState = NodeState | "online" | "stale" | "offline";

interface InventoryProcess {
  app: string;
  instance: string;
  ns?: string;
}

/** Epoch milliseconds, or `now` / `now-<duration>` (`now-1h`, `now-7d`). */
type TimeValue = EpochMs | string;

/** Milliseconds, or `30s` / `5m` / `1h` / `7d` / `2w`. */
type DurationValue = number | string;

interface Filter {
  label: QueryLabel;
  op: FilterOp;
  /** `eq`, `neq`, `regex`, `nregex`. ≤ 256 characters. */
  value?: string;
  /** `in`, `nin`: 1–64 values. */
  values?: string[];
}

interface TimeFn {
  fn: TimeFnName;
  /** `1m` or milliseconds, like `range.step`; default twice the step. Not for `raw`. */
  window?: DurationValue;
  /** 0..1, `quantile_over_time` and `quantile` (a histogram's) only. */
  q?: number;
}

interface Aggregate {
  fn: AggregateFnName;
  /** Labels the metric carries. */
  by?: QueryLabel[];
  /** 0..1, `quantile` only. */
  q?: number;
}

type MetricKind = "gauge" | "counter" | "histogram";

interface Series {
  /** The `by` labels, or every public label without an aggregation — never the bucket. */
  labels: Record<string, string>;
  /** Aligned on `MetricsResult.times`; absent in `summary` mode. */
  values?: (number | null)[];
  summary: Summary;
}

/** `rate`, `increase`, `delta` for counters; the `*_over_time` family for gauges; `raw` for either; `quantile` (with `q`), `rate` or `increase` for a histogram. */
type TimeFnName =
  | "raw"
  | "rate"
  | "increase"
  | "delta"
  | "avg_over_time"
  | "min_over_time"
  | "max_over_time"
  | "last_over_time"
  | "quantile_over_time"
  | "quantile";

/** Enrollment state in the database, presence aside. */
type NodeState = "active" | "pending" | "revoked" | "unknown";

type FilterOp = "eq" | "neq" | "in" | "nin" | "regex" | "nregex";

type AggregateFnName = "sum" | "avg" | "min" | "max" | "count" | "quantile";

/** Over the points the engine returned on the grid — before `fill`, so `sort` ranks what the engine ranked. */
interface Summary {
  min: number | null;
  max: number | null;
  avg: number | null;
  last: number | null;
  /** Points the engine returned (not the filled ones). */
  count: number;
}
```

</document>

<document url="https://3pm.edgecr4ft.net/docs/backend/cli.md" title="Flags and environment">

# 3pm-backend flags and environment

`3pm-backend --help`: every flag, the environment variable that sets it, its default.

```text
3pm fleet backend: enrolls agents, ingests telemetry, serves the dashboard

Usage: 3pm-backend [OPTIONS]

Options:
      --listen <LISTEN>
          Address to bind
          
          [env: THREEPM_BACKEND_LISTEN=]
          [default: 127.0.0.1:9616]

      --advertise <ADVERTISE>
          Where agents connect (goes into the pasted `3pm link` line): `host:port`, or a URL whose scheme wins — `wss://fleet.example.com` behind a proxy that terminates TLS. Defaults to this machine's hostname and the listen port
          
          [env: THREEPM_BACKEND_ADVERTISE=]

      --db <DB>
          SQLite database path (created on first run)
          
          [env: THREEPM_BACKEND_DB=]
          [default: 3pm-backend.sqlite]

      --call-margin-ms <CALL_MARGIN_MS>
          Over a remote command's own timeout, the margin before this backend settles it (`unacked` / `lost`), ms: the agent's 5 s and the link's round trips
          
          [env: THREEPM_BACKEND_CALL_MARGIN_MS=]
          [default: 15000]

      --data-dir <DATA_DIR>
          Where the files actions leave (profiles, dumps) are stored; default `<db>.d/` next to the database
          
          [env: THREEPM_BACKEND_DATA_DIR=]

      --dev
          Development mode: clear-text HTTP/WS, and localhost is an acceptable advertise address
          
          [env: THREEPM_BACKEND_DEV=]

      --history-points <HISTORY_POINTS>
          Samples kept per process for the dashboard charts (1 Hz)
          
          [env: THREEPM_BACKEND_HISTORY_POINTS=]
          [default: 300]

      --stale-after-ms <STALE_AFTER_MS>
          A node is `stale` after this many ms without a frame
          
          [env: THREEPM_BACKEND_STALE_AFTER_MS=]
          [default: 3000]

      --offline-after-ms <OFFLINE_AFTER_MS>
          A node is `offline` after this many ms without a frame
          
          [env: THREEPM_BACKEND_OFFLINE_AFTER_MS=]
          [default: 30000]

      --handshake-timeout-ms <HANDSHAKE_TIMEOUT_MS>
          The link handshake (challenge → hello) must finish within this many ms, else 4408 `ERR_HANDSHAKE_TIMEOUT`
          
          [env: THREEPM_BACKEND_HANDSHAKE_TIMEOUT_MS=]
          [default: 10000]

      --ping-every-ms <PING_EVERY_MS>
          Ping every live link this often, ms
          
          [env: THREEPM_BACKEND_PING_EVERY_MS=]
          [default: 20000]

      --silence-ms <SILENCE_MS>
          A link without any frame (pongs count) for this many ms is closed
          
          [env: THREEPM_BACKEND_SILENCE_MS=]
          [default: 60000]

      --enroll-pending-per-ip <ENROLL_PENDING_PER_IP>
          Enrollment handshakes one client address may have pending at once; beyond it `/api/v1/enroll` answers `ERR_RATE_LIMITED`
          
          [env: THREEPM_BACKEND_ENROLL_PENDING_PER_IP=]
          [default: 16]

      --link-pending-per-ip <LINK_PENDING_PER_IP>
          Link handshakes one client address may have pending at once (an IPv6 one counted by its /64); beyond it `/api/v1/link` answers 503 `ERR_LINK_BUSY`. Behind a reverse proxy, without `--trust-proxy`, every agent is the proxy and shares it
          
          [env: THREEPM_BACKEND_LINK_PENDING_PER_IP=]
          [default: 64]

      --header-timeout-ms <HEADER_TIMEOUT_MS>
          How long a connection may take to send its request headers before it is dropped (slowloris); `0` disables the timeout
          
          [env: THREEPM_BACKEND_HEADER_TIMEOUT_MS=]
          [default: 30000]

      --trust-proxy <TRUST_PROXY>
          Peers whose `X-Forwarded-For` is believed (the reverse proxy in front, `127.0.0.1`): the per-address enrollment budget and the link logs then key on the client it names. Leave it empty when the port is reachable directly: anyone can send the header
          
          [env: THREEPM_BACKEND_TRUST_PROXY=]

      --registration <REGISTRATION>
          Who may create an account (`POST /api/auth/register`): `closed` lets the first account in — it adopts every bucket without an owner — and no other; `open` lets anyone who reaches the port. `--dev` means `open`

          Possible values:
          - closed: The first account only
          - open:   Anyone who reaches the port
          
          [env: THREEPM_BACKEND_REGISTRATION=]
          [default: closed]

      --direct <DIRECT>
          Whether the bytes of a call between two LAN neighbours may go agent to agent (`on`), or always through this backend (`off`). Each node's own setting (`setNodeAccess.direct`) narrows it
          
          [env: THREEPM_BACKEND_DIRECT=]
          [default: on]
          [possible values: on, off]

      --cookie-secure
          Mark the session cookie `Secure`: set it when the dashboard is reached over HTTPS (behind a TLS proxy), so a browser never sends the cookie in clear text. Over plain HTTP a `Secure` cookie is never set, and nobody can log in
          
          [env: THREEPM_BACKEND_COOKIE_SECURE=]

      --metrics-url <METRICS_URL>
          Base URL of the metrics engine (VictoriaMetrics, `http://127.0.0.1:8428`): the telemetry is persisted there. Unset: nothing is persisted
          
          [env: THREEPM_BACKEND_METRICS_URL=]

      --events-url <EVENTS_URL>
          Base URL of the events engine (VictoriaLogs, `http://127.0.0.1:9428`): process lifecycle events go there
          
          [env: THREEPM_BACKEND_EVENTS_URL=]

      --insights-url <INSIGHTS_URL>
          Base URL of the insights apps' `triage` (`http://127.0.0.1:9700`): `GET /api/buckets/{id}/insights` relays its ranked incidents. Unset: that route answers 409 `ERR_INSIGHTS_DISABLED`
          
          [env: THREEPM_BACKEND_INSIGHTS_URL=]

      --store-interval-ms <STORE_INTERVAL_MS>
          Stored resolution, ms: one sample per series per interval (the engine's dedup interval must match). At least 1000
          
          [env: THREEPM_BACKEND_STORE_INTERVAL_MS=]
          [default: 5000]

      --store-query-concurrency <STORE_QUERY_CONCURRENCY>
          Chart queries in flight against the metrics engine at once; past it a query waits a few seconds, then answers 503
          
          [env: THREEPM_BACKEND_STORE_QUERY_CONCURRENCY=]
          [default: 8]

      --store-query-cache-mb <STORE_QUERY_CACHE_MB>
          Memory the query cache may hold, MB: the engine's answers kept for a step (a minute at most), so the same chart refreshed by several viewers costs the engine one request. 0 turns it off
          
          [env: THREEPM_BACKEND_STORE_QUERY_CACHE_MB=]
          [default: 64]

      --app-metrics-every-s <APP_METRICS_EVERY_S>
          How often, in seconds, an agent asks its daemon for the application metrics that moved (`welcome.pace.metrics_s`: the pace of the `app/<id>` sections); 0 reads as every tick
          
          [env: THREEPM_BACKEND_APP_METRICS_EVERY_S=]
          [default: 1]

      --trees-every-s <TREES_EVERY_S>
          How often, in seconds, an agent sends the figures of its apps' process trees (`welcome.pace.trees`: `figures`); 0 sends the shape only, when it changes (`shape`)
          
          [env: THREEPM_BACKEND_TREES_EVERY_S=]
          [default: 1]

      --state-every-s <STATE_EVERY_S>
          How often, in seconds, an agent sends what moved in its state — the host, the process table, the application metrics, the process trees — as one `state` frame (`welcome.pace.state_s`). 1 at least: the stream is the node's only one and its heartbeat, so `0` — which used to mean "the older streams" — is refused rather than read as something else
          
          [env: THREEPM_BACKEND_STATE_EVERY_S=]
          [default: 1]

      --blob-quota-mb <BLOB_QUOTA_MB>
          Bytes of blobs one bucket may have in flight at once, MB
          
          [env: THREEPM_BACKEND_BLOB_QUOTA_MB=]
          [default: 4096]

      --blob-disk-mb <BLOB_DISK_MB>
          Bytes every blob together may hold on disk, MB: a `file` blob spills only while a reader is behind, and this is the ceiling
          
          [env: THREEPM_BACKEND_BLOB_DISK_MB=]
          [default: 16384]

      --blob-window-mb <BLOB_WINDOW_MB>
          How far a `PUT` may run ahead of the slowest reader of its blob, MB; past it the body simply stops being read
          
          [env: THREEPM_BACKEND_BLOB_WINDOW_MB=]
          [default: 64]

      --live-window-ms <LIVE_WINDOW_MS>
          How much of a `live` blob the memory ring keeps, ms
          
          [env: THREEPM_BACKEND_LIVE_WINDOW_MS=]
          [default: 100]

      --blob-max-mb <BLOB_MAX_MB>
          Where any one blob is cut, MB
          
          [env: THREEPM_BACKEND_BLOB_MAX_MB=]
          [default: 4096]

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version
```

</document>

<document url="https://3pm.edgecr4ft.net/docs/backend/features.md" title="Backend features">

# Features

User-facing features of the fleet backend, one section each: what you
see, how to get it, and where it comes from. Internals live in
OV `3pm-backend-docs-archived/backend`; the wire contract with the agent is in
3pm's `docs/backend-agent/`.

## Buckets and the pasted line

**What you see.** A bucket is a fleet: a name and a `3pmc_…` id. `/`
lists and creates them; `/b/<id>` mints an enrollment token and prints
the one line to paste on a host:

```
$ 3pm link 3pmc_<cluster> 3pmk_<token> ws://127.0.0.1:9616
```

and, for a host without 3pm, the same link through the installer
(`install_line` beside `paste_line` in `createToken` and `bucketToken`;
under *Link a host* the bucket page shows it with the permanent token,
and the settings page too):

```
$ curl -fsSL https://3pm.edgecr4ft.net/install.sh | sh -s -- link 3pmc_<cluster> 3pmk_<token> wss://…
```

A token is short-lived and counted: `{ttl_s: 3600, uses: 1}` by
default, up to 30 days and 10 000 uses. The secret is shown once and
never stored — only its SHA-256 — and `DELETE …/tokens/{id}` revokes
it. `pre_approved` is decided at creation (`uses == 1 && ttl_s <=
3600`): a single-use, short-lived token admits its host at once, a
reusable or longer-lived one produces a *pending* node an operator has
to admit. Minting refuses a local advertise address outside `--dev`
(409 `ERR_ADVERTISE_LOCAL`), so a pasted line never points at
`127.0.0.1` by accident. The bucket page shows the address the line
ends with (`ws://…` or `wss://…`).

**Behind a TLS proxy.** The backend speaks clear text; nginx (or any
proxy) terminates TLS in front of it. `--advertise` takes a URL whose
scheme wins, so the pasted line says what agents must dial:

```
$ 3pm-backend --listen 127.0.0.1:9616 --advertise wss://fleet.example.com --trust-proxy 127.0.0.1
$ 3pm link 3pmc_<cluster> 3pmk_<token> wss://fleet.example.com:443
```

The agent checks the proxy's certificate against the public roots,
nothing to configure. A bare `host:port` keeps the old rule — `ws://`,
since the backend serves no TLS of its own — and outside `--dev` the
backend warns at startup that agents refuse clear text to a remote
host without `--insecure`. An invalid URL stops the start.
`--trust-proxy <ip>[,<ip>…]` believes those peers' `X-Forwarded-For`
(the rightmost hop that is not a trusted proxy), so the per-address
budgets (enrollment, link handshakes) and the link logs name each agent
rather than the proxy; the proxy must set it (`proxy_set_header
X-Forwarded-For $proxy_add_x_forwarded_for;`). Without the flag the
header is ignored. **Behind a reverse proxy the flag is not optional:**
without it every agent is the proxy's address, and the whole fleet
shares one address's budget — 64 pending link handshakes, 16 pending
enrollments.

**Where it comes from.** `crypto/tokens.rs`, `db/enroll.rs` (one
transaction: consume the use and upsert the node — no burnt use, and
two hosts racing for the last one never both win it), `http/paste.rs`,
`advertise.rs`, `http/client_ip.rs`. `POST /api/v1/enroll`
(`http/api_enroll.rs`) is the agent's side: a token, a public key and a
signature over both in, a node id out. A host is
`(cluster_id, node_id, Ed25519 key)` on its row — there is no
certificate and no CA, so nothing expires and nothing is re-issued: the
row is read again at every handshake, and an approval or a revoke lands
on the next connection.

## Admitting and revoking hosts

**What you see.** Every node of a bucket is a row: its id, host, OS,
agent version, state, last seen and receive rate.
`Approve` admits a pending host, `Revoke` cuts it — the live session is
closed 4403 `ERR_NODE_REVOKED` and the agent parks itself and probes
hourly. Both are idempotent, and approving again re-admits a revoked
node with the key already on record.

`Delete` is offered on every node that is not `online` — offline,
stale, pending or revoked; a node that is talking to the backend shows
no such button, because deleting it cuts a link its agent cannot mend
by itself (it would retry forever under an id nobody knows). The route
itself refuses nothing: a script may still delete an online node.
`Delete` forgets a node, after a confirmation: its row, its calls and
its artifacts (the files too) go, its session is closed 4401
`ERR_NODE_UNKNOWN` and its bearer dropped. It is the opposite of a
revoke in one respect: nothing is kept against the key, so the host may
enroll again — as a new node, with a new id. Use it for a machine that
is gone, or a test enrollment; use `Revoke` to keep a host out. Every
open page of the bucket redraws without the node (the feed sends
`reset`, the page asks for a fresh `init`). The metrics and events the
store already holds for it stay until the store's retention takes them.

A host that comes back under a new key — reinstalled from scratch, its
`node.key` gone with the disk — enrolls as a new node, and its old row
is **deleted on its own** when the new one becomes active (enrolled
with a pre-approved token or approval off, approved, or admitted by
turning approval off): every older row of the bucket with the same
hostname (case ignored) that has no live session and is `offline` goes,
exactly as `Delete` would. A row someone is linked through, a `stale`
one (it may be reconnecting) and a revoked one are kept; a pending
node replaces nothing until it is admitted; of several twins admitted
at once, the newest stays. The hostname is the agent's word, not a
proof: two homes of one machine linked to the same bucket share it, and
the offline one is deleted when the other enrolls.

A host that knocks while pending or revoked leaves `last_auth_error`
on its row (`ERR_NODE_PENDING`, `ERR_NODE_REVOKED`, with a timestamp),
so the page shows who is waiting instead of failing silently — one
click admits it.

**Where it comes from.** `http/api_nodes.rs` and the hub
(`hub/registry.rs`): the database row changes, the hub's view is
replaced, and a `node` event goes out on the feed. A delete is
`db::nodes::delete` (the row and the paths of its files, one
transaction), then `Hub::forget`, whose `reset` `http/sse.rs` turns into
the end of every stream. The automatic delete is `http/supersede.rs`
(`is_replaced_by`, the rule; `sweep`, called by enrollment, `approveNode`
and `setBucketSettings` before the node is published), through
`api_nodes::forget`, the body `deleteNode` shares.

### Bucket settings: approval, on or off

**What you see.** `settings`, in the header of a bucket's page, opens
`/b/{cluster_id}/settings`: one box, *Require approval for new hosts*.
On (the default), a host linked with a reusable token waits for
`Approve`; a one-use token of an hour or less is active at once, as
always. Off, every host that links with a valid token of the bucket is
active at once — what enrolling twenty tablets with one reusable token
wants — and the token is then the only gate. Turning it off approves
the hosts that were already waiting (the page says how many before you
do); turning it back on leaves them in. The bucket page follows the
setting: the note next to *reusable* and the line under a new token say
whether the host will wait.

**Calls between nodes** is a second box: *Require nodes to specify
access rights*, ticked by default. Ticked, a node places calls only once
you tick *Place calls* in its Access panel. Unticked, every active node
may run any action of any other and list the bucket's nodes to find
them by hostname, and read each other — as if each held `fleet:invoke`,
`fleet:list` and `fleet:read` —
with nothing to grant node by node: a tablet that enrolls tomorrow
talks to the others at once. It opens *Read the fleet* too (hosts,
metrics, the catalog of actions: a caller checks what a peer declares
before calling it), it writes nothing on the nodes (their
Access panels still show what you granted, and ticking the box again
takes back exactly what it gave), and a node's own *Who may call this
node* rules still apply. Both ways are felt on a node's next request.

Below it, the bucket's **permanent enrollment token**: one
`3pm link …` line with a `copy` button, the same every time you open
the page. It never expires, enrolls any number of hosts, and cannot be
renewed or revoked — paste it on a host, or give it to the script that
provisions your machines. It goes through the same approval as any
reusable token: with the box ticked its hosts wait for `Approve`, with
the box unticked they are active at once and the token is the bucket's
only gate, so keep it like a password. The one-use and reusable tokens
of the bucket page still exist next to it.

Over the API: `bucketToken(clusterId)` answers `{ token_id, token,
paste_line, created_at }`; `setBucketSettings(clusterId, { approval: false })`
answers the `BucketView`, which carries `approval` everywhere a bucket
is read. A node's bearer never sees it.

**Where it comes from.** `http/api_settings.rs` over
`db::buckets::set_approval` (the setting and the queue it empties in
one transaction), read at enrollment in `http/api_enroll.rs`; `require_access_rights`
is read by `db::effective_scopes`, applied where a node's scopes are read: the
node gate (`auth/node_gate.rs`, on the bucket row of that request) and
the `call_result` push (`calls/result.rs`, which reads the setting
fresh); the
permanent token is `db::tokens::permanent` (created once, behind a
partial unique index; `SQL_CONSUME` takes a use from it without
spending one, `SQL_REVOKE` leaves it out), the only token whose secret
is stored; the page
is `assets/settings.html` + `assets/pages/settings.js`.

## The live dashboard

**What you see.** `/b/<id>` opens one SSE feed and stays current
without polling: presence goes `online` → `stale` (3 s without a
frame) → `offline` (30 s), each node's process table refreshes at 1 Hz
with CPU, memory (of the app's whole process tree, as 3pm reports
them), restarts and uptime, `+N` after an app with other processes
and a **Process trees** panel that draws them (its own section
below), the last raw `telemetry` frame
sits under a `<details>`, and lifecycle events (`process`,
`daemon_shutdown`) scroll in a log. Each node also shows what it costs:
`rx_bps` (bytes received over its link, averaged over the last 5
seconds) and the total since the backend started.

Next to each card's hostname, the node's addresses: `public <ip>`,
where its link comes from as this backend sees it (behind a
`--trust-proxy` proxy, the forwarded client; kept while the node is
offline, forgotten when the backend restarts), and `private <ip>, …`,
the private addresses of its interfaces (RFC 1918, CGNAT `100.64/10`,
IPv6 ULA; loopback, link-local and container or VM bridges — `docker*`,
`br-*`, `veth*`, `virbr*`… — left out), from its last complete host
snapshot. The node view carries them as `peer_ip` and `private_ips`.

At the bottom of the page, **Metric keys** lists every key the bucket
holds — the store's own telemetry and the application metrics seen
here over the last thirty days — with its kind, unit, default time
function, the labels a query may filter or group on, and where the key
comes from (`static` or `app`). Applications come first, the count
next to the title says how many of the keys are theirs, and the list
is asked again every 30 s while the tab is visible, so an app's first
sample shows up on its own.

**Live JSON.** Every row of the node table has a **Live JSON** button,
and every app of a card a `JSON` one: a dialog shows the node's
documents exactly as the API serves them, refreshed every second while
it is open — the host now (`GET …/nodes/{node}/host`, `nodeHost`, what
the node button opens on), the last telemetry frame as the feed sent
it (no request), `nodeAppMetrics` (custom metrics),
`nodeProcessTrees`, `nodeActions`, or one app composed from the three
(`{ process, app_metrics, tree }`, what the per-app button opens on).
A second selector narrows any document to one app. The document is a
collapsible tree (json-formatter-js, vendored under `assets/vendor/`):
the top-level keys open, everything under them folded behind a `►`,
a preview of what a fold holds on hover, keys, strings, numbers,
booleans and nulls each in their colour; what the reader unfolds stays
unfolded across the one-second repaints (every row carries its path,
and the fresh tree is unfolded along the same paths). A **search**
box cuts the document down to every key or scalar value holding the
text (case-insensitive): each hit with its parent whole — every
attribute of the parent, as it is — under the chain of keys that
leads there, everything unfolded down to the hit and the hit's key on
an amber ground; an array on the way shows as its indices, so they
stay true. The count says how many hits; the search follows the live
document. **text** shows
the raw JSON instead, **stop** freezes the document to study it (the
title says when it was frozen; changing the source or the app loads
that document once and stays frozen; **resume** follows it again),
`copy` puts the text on the clipboard whatever the view, and the
reader's scroll survives a repaint.

The frame and the host differ on purpose. To save 5 KiB a second per
node and viewer, the feed carries the host's identity — hostname, OS,
kernel, arch, boot time, CPU brand and counts, memory and swap totals,
the interfaces' names, MACs and IPs, the mounts, the GPUs' models —
only on a node's first frame, when it changes, and once a minute
(`host_full`); every other frame is a tick whose identity fields read
blank (`host_tick`). `nodeHost` answers the host whole — the one the
backend rebuilds from the node's state (see "The host, received as a
diff"), `identity` always `patched`. 404 before the node's first frame.

**Events and Metric keys, folded and filtered.** Both lists fold
behind their title (closed by default, the count next to it: `(200
events)`, `(218 keys, 3 from applications)`), with a filter box
inside: words, every one of which must appear in the line — `api
error` keeps the error events of `api`; `counter iface` the counters
carrying the `iface` label (a key's id, kind, unit, labels, source and
description are all matched). The count says what is shown of the
whole (`(12 of 200 events)`).

**SDK guide.** At the bottom of the page (**SDK guide** in the
header), six folds: start an app with the SDK, custom metrics, custom
events, custom actions, profiling, and the `3pm` commands to know
(`start`, `ls --tree`, `show`, `events -f app` / `-f proc_action`,
`trigger`, `profile`, `link`). The examples are runnable as they are:
the `curl` and JavaScript-client snippets name this bucket, its first
online node and that node's first app, and follow the feed when they
appear (`assets/pages/guide.js`, whose escaping and filling
`tests/it/dashboard_render.rs` holds). Every example is coloured by
highlight.js (vendored under `assets/vendor/hljs/`: core and the
`bash`, `javascript` and `json` grammars, loaded after the guide is
painted — `assets/pages/highlight.js`), one language per block.

A page that draws a few hosts asks the feed for less
(`?history=0&match=<hostname part>&view=live`, `http/feed_query.rs`):
only those hosts, no per-process history in `init`, and each frame cut
to the gauges a live chart draws — a tenth of a frame, and the other
nodes never leave the server. The generated clients send `history=0`
by default, the dashboard at `/b/<id>` included (it never read the
history); it asks for nothing else.

The feed replays what a reconnecting client missed (`Last-Event-ID`,
the last 4 096 events per bucket); a client that fell further behind is
told to `reset` and starts over from a fresh snapshot. Presence
timings are tunable (`--stale-after-ms`, `--offline-after-ms`,
`--history-points`).

**Where it comes from.** `http/sse.rs` over `hub/` (registry,
presence, per-process rings, per-bucket event log); `assets/` for the
pages — `app.js` discovers the route table (`GET /api/v1/routes`) and
hands the generated client to `pages/index.js` / `pages/bucket.js`,
which know route names and never a path. The key list is
`metricsCatalog` rendered by `catalogTable` (`pages/bucket.js`), whose
escaping `tests/it/dashboard_render.rs` holds.

## The telemetry, persisted

**What you see.** Start the backend with `--metrics-url
http://127.0.0.1:8428` (VictoriaMetrics) and `--events-url
http://127.0.0.1:9428` (VictoriaLogs) and every linked node's
telemetry outlives the process: one sample per series every 5 s
(`--store-interval-ms`) — `threepm_process_cpu_ratio` and
`threepm_process_memory_rss_bytes` (the app's whole process tree: the
spawned pid and every descendant, RSS summed — an upper bound when
members share pages — as 3pm reports them; the spawned pid alone from
a 3pm predating trees), `threepm_process_procs` and, per group of
the tree's members sharing an executable name, `threepm_subprocess_*`
(label `proc`, never a `pid`),
`threepm_process_up`,
`threepm_process_restarts_total`, `threepm_host_cpu_ratio`,
`threepm_host_memory_used_bytes`, `threepm_host_network_receive_bytes_total`,
`threepm_host_disk_used_bytes{mount}`, `threepm_node_up`, and what 3pm
itself costs the node as its host agent measured it —
`threepm_daemon_cpu_ratio`, `threepm_daemon_memory_rss_bytes`,
`threepm_host_agent_memory_rss_bytes`, `threepm_daemon_apps_online`,
`threepm_daemon_restarts_total`, `threepm_daemon_handoffs_total`…
(catalog ids `daemon.*`, `host_agent.*`) — labelled
`cluster` (the bucket), `node`, and per process `app`, `instance`,
`ns`. Process lifecycle events (`start`, `online`, `exit` with code and
signal, `restart`, `errored`…) land in the logs engine, one line each.
`GET /api/v1/info` → `store` says whether the engines are configured
and counts what was `sent`, `dropped`, `retries` and `errors`, with the
kind of the last failure. Without the flags nothing is persisted and
nothing changes.

Every number `3pm hostmetrics` collects is kept, not a chosen few: the
CPU per core and by state, the kernel counters, the memory
decomposition and the page-fault counters, PSI, processes and TCP
connections by state, the socket and conntrack counters, the daemon's
cgroup, then one series set per network interface, block device,
temperature sensor, GPU, RAID array and battery — with `core`, `iface`,
`blockdev`, `sensor`, `gpu`, `array` and `bat` as query labels — plus
when each sampling tier last landed, so a tier that stopped is visible.
A state is always part of the metric's name, never a `state` label that
would collide with an application's own. `GET …/metrics/catalog` lists
the 292 lines and the dashboard's **Metric keys** table shows them.

A node's **top processes** (`top_processes`: the heaviest processes of
the host by CPU, resident memory and network rate, at most 30 rows) are
live state, not series: `nodeHost` and the feed's host carry the table,
nothing is stored per process, and `system.sample.top` dates it.

A laptop's or a UPS-backed box's packs are twelve of those lines
(`system.battery.*`, one set per pack under `bat`): charge, energy,
what the pack holds today and held new, the health those two make, the
draw, the voltage, the cycle count, the temperature, whether the bay is
filled, and two codes — `system.battery.state` for what sysfs `status`
says (nominal: 0 unknown, 1 charging, 2 discharging, 3 not charging, 4
full, 5 a status this backend has no name for), `system.battery.level`
for the coarse level a gauge with no numeric capacity falls back to
(ordinal, 1 critical … 5 full). The direction the energy flows is the
state, never the sign of the draw, which is a magnitude. `system.on_ac`
is the host's own answer, not a pack's, so it carries no `bat`. Every
reading is optional and an absent one stays absent: no `status` file
writes no state, no sensor writes no temperature, and a full pack
drawing nothing writes a real 0.

**Extended Monitoring** (a 3pm node's `[hostmetrics] extended`, on by
default) adds the kernel's own counters on Linux, ~1 000 to 1 600
series per node, stored **one series per family**:
`threepm_hostx_<family>_total` for a counter family,
`threepm_hostx_<family>` for a gauge one — `vmstat` and
`vmstat_gauge`, `netstat`, `snmp`, `diskstats`, `cpuidle`, `hwmon`,
`thermal`… — with `field` the kernel's own spelling of the reading and
the row under the label that already means it (`core`, `iface`,
`blockdev`, `array`) or a generic `row` (a PSI resource, a protocol, a
chip, a zone, a C-state). A chart names the family and filters the
field: `{"metric":"host.ext.vmstat","filters":[{"label":"field","op":"eq","value":"pgsteal_kswapd"}]}`;
an unfiltered query of the line shows the fields that family holds,
while `metricLabelValues` on `field` lists every field of every family
in the bucket (and an application's own `field` label, if it has one);
the inventory lists neither `field` nor `row`. The live feed carries
none of it — ~40 KB a node at every tick, drawn by no dashboard — and
`nodeHost` serves the node's current `extended` map whole. A slow-tier
counter (`host.ext.cpuidle`, read every 5 minutes) rates over at least
10 minutes. The catalog has one hand-written line per family
(`host.ext.*`, 42 with the three eBPF short-lived-process families),
each saying where the fields'
meaning is written — the man page, the kernel doc, the RFC. `field` and
`row` are custom labels, like `bat`: an application's own `field` label
still works. Per group of a process tree come sixteen more
(`subprocess.faults.*`, `subprocess.context_switches.*`,
`subprocess.disk.*`, `subprocess.io.*`, `subprocess.syscalls.*`,
`subprocess.threads`, `subprocess.fds`, `subprocess.fds.usage_max`,
`subprocess.memory.{pss,footprint,swap}`), each present only where the
platform gives it — a gauge summed only when every member of the
group gives it, a counter accumulated by the backend (see "Process
trees") — and per app `process.memory.pss`, the tree's PSS —
what it really costs where `process.memory.rss` double-counts shared
pages, absent unless every member's could be read.

Network traffic per app, from a 3pm node's own meter (not Extended
Monitoring: on wherever the node has a source — Linux, macOS):
`process.net.received` / `process.net.sent` per app (its whole process
tree, each socket once; `threepm_process_net_{received,sent}_bytes_total`,
read with `rate`), and `subprocess.net.received` / `.sent` per group,
accumulated like the other group counters. The counter restarts with
the app and with a daemon update — a counter reset `rate` handles. On
Linux an outbound connection opened and closed between two of the
daemon's 5 s windows is not in it;
on macOS loopback is not. The live figure is on the record
(`RedactedRecord.net`: `r`/`t` per second, `rb`/`tb` totals) and on each
tree line (`TreeProc.net`).

**Where it comes from.** `src/store/`: the writer pulls the hub's last
frame per node on its own tick (nothing per frame, the live feed is
untouched), encodes off the runtime with the label blocks of every
series rendered once and kept (`prefix.rs`), gzips the bodies side by
side, posts Prometheus text and NDJSON with a short retry ladder; the schema and the sizing are in
OV `3pm-backend-docs-archived/backend` → Metrics store and
`3pm-metrics-plan.md` in `~/3pm-rd-doc-2026-09-09.tgz`.
The workspace `ecosystem.toml` starts the engine on loopback with a
dedup interval equal to the store interval.

## The node's whole state, received as diffs

**What you see.** Nothing on the pages, the feed or the API says how a
node's state travels. On the link there is one mechanism: the backend
asks its agents (`welcome.pace`, from `--state-every-s` /
`THREEPM_BACKEND_STATE_EVERY_S`, default `1`, `1` at least — `0` is
refused at start) for **one `state` frame a second** carrying only what
moved in the node — the host, the process table, each instance's
application metrics, each process tree. Measured on real traces,
everything included: 357 B/s for a dev Mac with 11 apps, 463 B/s for a
Linux server with 8. It is the only way in: the link is at **version
4**, and an agent that predates it is refused at the handshake with
`ERR_LINK_VERSION` (close 4426) — or at the upgrade with 400, if it
offers the older sub-protocol — until the node runs `3pm update`; the
frames of a stream this build no longer knows, should one ever come,
are unknown tags — skipped, never an error.

`GET /api/v1/info` → `link` is what says the fleet and this backend
agree on the wire: `frames_invalid` counts agent frames whose tag this
build knows and whose **body** it could not read (a field one side
spells differently: skipped, the session kept), `crc_full_mismatch` the
sessions in which a full state section failed its checksum. Both `0`
is the normal reading; either one moving names a bug in one of the two
builds, and the backend's log says which frame or which section. Only
an **established** session is counted or logged that way: before its
`hello` a peer has proved nothing — the `Authorization` header is a
claim over two public ids — so what it sends is skipped quietly, eight
frames at most, and the ninth closes the connection.

**Where it comes from.** `src/hub/node_state.rs`: `NodeState`, one per
node, kept across a session replacement, applies every patch of a frame
on its own with `threepm_link::NodeReader` — **the** patch reader, the
only one — a gap leaves its section untouched and the session sends
`state_resync {secs}` (at most once a second, the one `ws/resync.rs`
coalescer of a session), a **full** patch with a wrong checksum is kept
and turns the checks off until the node's next session (one `error!`
line naming the section, and `info.link.crc_full_mismatch`); a
section past 4 MiB, or the one that takes the node's sections past
32 MiB together (both tracked as bounds, measured only past them), is
dropped and counted — from its typed holder too, the host included —
and comes back whole at the agent's next patch of it; a 405th section
is refused and counted, with one warning a session, until one held
goes. A frame of **another incarnation** of the agent (`gen`: it
restarted) forgets every section that frame did not bring again: a new
agent owes all it has in full and never says `gone` for what went away
while it was down, so nothing of its former self is served, or piles
up restart after restart. What the reader refused so far is the
`dropped` of the feed's telemetry and the store's
`threepm_telemetry_dropped_total`. The process table is decoded when
its section moves, and read at every frame's clock.
`hub/registry_state.rs` hands what moved,
typed, to the holders the consumers read: the host section to
`NodeHost` (the view's addresses and accounts following it), an
instance's metrics to `NodeApps::set`, a tree to `NodeTrees::set`.
Both refuse what is over the daemon's caps or off the label grammar —
counted, and the state held for the instance dropped rather than left
behind its section — and an instance goes with its section: `gone`, or
the agent restarted without it, at once. Its row of the process table
is a safety net only (absent from two frames in a row: forgotten, its
section with it, so the next patch is a gap and not a patch onto what
nobody serves) — and never on the table of a **daemon that is down**,
which lists nothing: the agent says no `gone` then, and an instance
that never moves (one that only declares actions) is still there when
the daemon answers again. A node **revoked** keeps its view and none
of what it sent. A section that does not decode,
or names another instance than its own, counts as gone. The trees held
are current as of every frame, and their clock says so. Then
`Hub::state` publishes, for **every** frame, the `telemetry` the feed
always carried (`hub/feed_snapshot.rs`, the backend's own type now) —
numbered by the backend, the table as of the frame's clock (a process's
uptime is derived from it), the daemon, the host by the feed's rule —
so the feed, the rings and the store's freshness work as they did.
Tests: `hub::node_state` (typed hand-back, gaps, gone, the bounds and
the node's budget, the checksum rule, an agent restarted, the table
read at every clock, the refusal count), `ws::resync` (a section that
came in full is no longer asked), `hub::apps` and `hub::trees` (caps, grammar, the two
ways out), `hub::registry_tests`, `tests/it/state_e2e.rs` (the feed's
shape, `nodeHost`, the view's address, `nodeAppMetrics`, the trees, the
store's lines, a heartbeat, `state_resync`, gone, an older stream's
frame skipped), `dashboard_e2e` (link 1 refused) and every
`real_agent_*` test (a real daemon and agent).

## The host, received as a diff

**What you see.** The bucket page, the tablets, `nodeHost`, the charts
read a host that is whole and current, and a linked node costs little
on the link: the host is the `host` section of the node's state, so
only what moved in it travels. Measured on real traces against the
whole snapshot in every frame: 359 → 194 B/s for a dev Mac, 576 → 403
B/s for a Linux server, and static sections cost nothing after the
first frame. Numbers arrive at **3 significant digits**. `nodeHost`
answers the rebuilt host, identity included; its `identity` field is
always `"patched"` (it used to say `full`, `merged` or `tick` for a
host read off `telemetry` frames — that source is gone, the field stays
for the pages that read it), and a node that sends no host (its host
agent is down) is a 404 that says so.

**Where it comes from.** The section is rebuilt and bounded by
`NodeState` like any other (above). `src/hub/host.rs`: `NodeHost`, kept
across a session replacement, holds what the section stands for — the
typed `HostMetrics`, decoded once per revision when the section moves,
shared as a pointer — and the agent's clock on that frame. It is the
**only** host the backend reads: the store writes it as both the
figures and the identity (every host family, per interface and per
mount included); the view's private addresses, account and sessions
follow it (a patch touching them announces the node, a host gone leaves
the last answer); `nodeHost` serves it. The `telemetry` the feed
publishes gets it put in by the feed's rule — `host_full` on the first
frame, on an identity change and every minute, a stripped `host_tick`
otherwise (`hub/feed_snapshot.rs`: `FullPolicy`, `strip_static`) — so
dashboards and tablets see the shape they always did, at 5 KiB less a
second per node and viewer. Moving the feed itself to diffs is a later
piece of work. Tests: `hub::host` (the decode, a section that is not a
host, the feed's rule), `hub::feed_snapshot`, `tests/it/host_e2e.rs`
(the route whole while the feed's tick is stripped, the view, the clock,
the 404s), `tests/it/state_e2e.rs` and `real_agent_host` (a real daemon
and agent: the rebuilt host equals `3pm hostmetrics --json` at three
digits on everything that does not move within a second).

## Link traffic: what each node's link costs, by kind

**What you see.** A node card's "Link traffic" panel: per direction
(↑ agent → backend, ↓) and kind (`host`, `extended`, `top`, `procs`,
`app`, `trees`, `events`, `shell`, `calls`, `control`, `traffic`, `logs`,
`artifacts`, `blobs`, `api`), bytes/s and messages/s over the last
minute, the state kinds marked "est." (a pro-rata split of an exact
per-frame total, measured ±5–40 %: `rd/logs-bytes-counter/lot0-preview.md`),
and `rx_bps` — what this backend measured on the WebSocket — beside as
the cross-check. In charts: `link.bytes` / `link.messages`, filtered or
grouped by `link_dir` / `link_kind`.

**How.** The agent counts and ships the `traffic` section every 5 s
(`3pm/docs/backend-agent/link-protocol.md` §7). `hub/node_state.rs`
holds it like any section it has no typed view of (`traffic()`),
`Collecting::finish` reads it leniently (`traffic_view`), and
`store/link_samples.rs` writes `threepm_link_bytes_total` /
`threepm_link_messages_total` `{cluster, node, link_dir, link_kind}` at
every store tick of a collected node, `Traffic::known` only (≤ 30
pairs a node, ~60 series): a kind this build does not know is dropped
and said once per node. `link_dir` / `link_kind` are `Custom` query
labels carried by the `link.*` lines only, like `bat` and `field`: no
word reserved, an application's own `link_kind` coexists.

**Tests.** `store::tests_link` (the series and their labels, an unknown
kind dropped and warned once, the catalog naming exactly what is
written, the hub handing the section over), `query::tests_link` (the
labels filter and group on `link.*` only), `dashboard_traffic` (the
panel's query, rows, "est." and escaping, under `node`).

## Charts: the metrics query

**What you see.** `api.queryMetrics(clusterId, body)` — `POST
/api/buckets/{id}/metrics/query` — answers a chart in one call: a
range and a step, one or more queries naming a metric of the catalog
with filters, a time function and an aggregation, formulas over their
refs, and options. The answer is one `times[]` array and, per series,
numeric `values[]` aligned on it with `null` for a gap, plus a summary
(min, max, avg, last, count) — what uPlot or ECharts draw as is.

CPU per app in the `prod` namespace, last hour, 30 s:

```json
{ "range": { "from": "now-1h", "step": "30s" },
  "queries": [{ "ref": "A", "metric": "process.cpu.utilization",
    "filters": [{ "label": "ns", "op": "eq", "value": "prod" }],
    "aggregate": { "fn": "avg", "by": ["app"] } }] }
```

Peak memory per node over 6 h, the 20 largest by their last value:

```json
{ "range": { "from": "now-6h" },
  "queries": [{ "ref": "A", "metric": "process.memory.rss",
    "time": { "fn": "max_over_time" }, "aggregate": { "fn": "max", "by": ["node"] } }],
  "options": { "max_points": 360, "max_series": 20, "sort": "last" } }
```

Memory in use as a percentage of the host's, per node — two hidden
queries and a formula:

```json
{ "range": { "from": "now-1h" },
  "queries": [
    { "ref": "U", "metric": "system.memory.used", "aggregate": { "fn": "sum", "by": ["node"] }, "hidden": true },
    { "ref": "T", "metric": "system.memory.total", "aggregate": { "fn": "max", "by": ["node"] }, "hidden": true } ],
  "formulas": [{ "ref": "PCT", "expr": "U / T * 100", "unit": "percent" }] }
```

What a query may name is `api.metricsCatalog(clusterId)`: ids in
OpenTelemetry style (`process.cpu.utilization`, `system.network.receive`,
`node.up`…), each with its kind (gauge or counter), unit, labels
(`node`, `app`, `instance`, `ns`, `mount`, `window`, `src`, the
host dimensions `core`, `iface`, `blockdev`, `sensor`, `gpu`, `array`,
`bat`, and a process tree's groups, `proc`)
and the time function a chart wants by default — `rate` for a counter,
`avg_over_time` for a gauge — so `time` can be left out. The values a
label takes in the bucket are `api.metricLabelValues(clusterId, "app")`.
A dashboard or an LLM that starts from nothing asks
`api.metricsInventory(clusterId)` (`GET …/metrics/inventory`) once:
the catalog lines that actually hold data in the bucket over the last
two days, grouped by the labels they carry, with their descriptions
and, for an application's own line, the apps that emit it; what every
other label holds (`window`, `mount`, `iface`, an application's own
labels…, 50 values at most); and the hosts with their processes
(`{ app, instance, ns? }`, label values that go into `filters` as they
are) — every series carries `node` = a node_id, and `hosts` is where a
hostname becomes one. Its first field, `tokens`,
says what the answer costs an LLM before it reads the rest.
It answers TOON-style text by default, a header per list and a row per
entry (about a fifth fewer tokens than the JSON); the generated client
gets the JSON (it sends `Accept: application/json`), and so does
`?format=json`. `?detail=index` drops the descriptions (about 40 %
fewer tokens; more than half fewer as text). The bucket page links it
(**inventory**, next to the API links).
A question that starts from a host or an app asks
`api.metricsByServer(clusterId)` (`GET …/metrics/servers`) instead:
each server keyed by node_id with the ids of its host's own metrics,
then its apps keyed by name, each with the ids of the metrics its
series carry on that server — what can be queried with `node` (and
`app`) without drawing nothing; a Linux-only line shows under the
Linux host only. The metrics are described once, in `metric_groups`,
as the inventory describes them; present means a series over the last
24 hours. Same formats and `?detail=index` as the inventory. The
bucket page links it too (**by server**, next to **inventory**).
One that starts from an app asks `api.metricsByApp(clusterId)`
(`GET …/metrics/apps`): each app keyed by name with the servers it
runs on, keyed by node_id, and on each the ids of the metrics its
series carry there — `agent` on every host, a worker on two; `hosts`
turns a node_id into a hostname, and a host's own lines stay with
**by server**. Linked on the bucket page as **by app**.
A step left out is derived from `max_points` (600 by default) on a
ladder of round values; `max_series` (200 by default, 2 000 at most) keeps the top
series by average (or `last`); `mode: "summary"` answers without
values, for an assistant that wants numbers, not a chart.

A query that is wrong is refused before anything reaches the engine,
400 `ERR_BAD_REQUEST` with a message naming the field: an unknown
field (a typo would otherwise draw the wrong chart), an unknown
metric, a label the metric does not carry, `rate` on a gauge, a regex
that does not compile, a range past 31 days, more points than
`max_points`. Without `--metrics-url` the route answers 409
`ERR_STORE_DISABLED`; an engine that is down, slow or busy is 503
`ERR_STORE_UNAVAILABLE`.

The same chart refreshed within the same step, by however many
viewers, costs the engine one request: the answer is cached for a
step (`--store-query-cache-mb`, 64 by default) and identical requests
in flight share one; `GET /api/v1/info` → `store.cache` shows hits,
misses and coalesced requests. Answers are gzipped when the browser
asks, which every browser does.

**Where it comes from.** `src/query/` (the contract, the catalog,
validation into a plan, formulas, alignment and execution — engine
agnostic, a test holds it), `src/store/victoria/{metricsql,read}.rs`
(the plan as MetricsQL with the bucket first in every selector and
`extra_label` on the URL, the reads under `--store-query-concurrency`),
`src/store/cache.rs` (the answers kept a step, single-flight),
`src/http/api_metrics.rs`; the gzip layer sits on every JSON route in
`src/http/mod.rs`. The engine's language never crosses the
API in either direction: OV `3pm-backend-docs-archived/backend` → Metrics API.

## Logs: what the nodes ship, queried

**What you see.** With `--events-url` set, every linked node sends its
logs (3pm 0.55.0 and later): its apps' output, the system journal and the
log files of the other programs it runs, reduced on the node to templates
(a line with its variables masked), counts per template per 5 minutes and
the raw lines that carry news (a new template, a warning once an hour, a
spike, the 30 s before a new error). What the node sends and why is in
3pm's `FEATURES.md`, "Log shipping". `api.logsQuery(clusterId, body)` —
`POST /api/buckets/{id}/logs/query`, the owner's, like `eventsQuery` —
answers the bucket's logs in one call. What went wrong over the last six
hours, errors and worse only:

```json
{"range":{"from":"now-6h"},"filters":{"sev":"err+"},"limits":{"templates":50,"lines":100}}
```

`range` is the events query's (`from` / `to`, epoch ms or `now-<duration>`,
30 days at most). `filters` are exact and AND-ed: `node`, `src` (the
node's source id), `owner` (the writer's name), `tpl` (a template id,
unique per source of a node only: it needs `node` and `src`, else
`400`), and `sev` — a word
of `debug`, `info`, `notice`, `warning`, `err`, `crit`, a list, or
`<word>+` for that level or worse. `text` keeps the shipped lines that
contain it (case sensitive); counts have no text. `limits {templates ≤
500 (200), lines ≤ 1 000 (200)}`.

The answer is `{range, sources, templates, lines, truncated}`:

- `sources`: every source the nodes declared over the range, `{node,
  hostname, src, kind (app|journal|file), path?, app?, owner?, format?,
  lines, shipped, dropped, templates}`;
- `templates`: worst severity first, then by lines, `{node, src, tpl,
  sev, text, lines, first_seen, last_seen, params: [{var, distinct,
  top}]}` — `sev` is the worst counted over the range; the templates
  beyond a node's 2 000 busiest come back together per source and
  severity as `tpl: null`, `text: "(other templates)"`;
- `lines`: the shipped lines, newest first, `{ts, node, hostname, src,
  tpl?, sev, why, line, ident?, unit?}` (`why`: `new`, `error`, `back`,
  `spike`, `return`, `context`);
- `truncated`: what the limits cut, and `counts`: the nodes whose
  counts the engine would not give (their templates and source `lines`
  are missing; 0 when the answer is whole).

Counts come in windows of about 5 minutes stamped at their end: the
window still open at `to` is not counted yet, so the last 5 to 10
minutes lag behind the lines. Without a `node` filter they are asked
five nodes at a time, four queries at once, 20 s for all (VictoriaMetrics refuses a query touching more than
`-search.maxUniqueTimeseries`, 100 000 on the fleet); a chunk it refuses
is logged and counted in `truncated.counts`, the rest of the answer
stands. `params` come back empty rather than failing the answer;
`distinct` is the largest over the whole range (merged by the engine),
`top` the values most often in the tops of the latest 2 000 windows.
Bounds: `limits.logsQuery` of `GET /api/v1/routes`.

**Series budget.** At any moment a node writes at most 2 000 labelled
templates (`tpl` = its id, usually one `sev` each) plus, per source, one
`tpl="other"` series per severity (6 at most) and one
`threepm_log_dropped` series per drop reason: with the agent's 512 files
plus its apps and the journal, ~2 000 + a few hundred series in
practice (prod, 2026-10-06: 36–601 per node, 1 977 for 14 nodes). The
seats bound only the labelled templates: `other` and the drops are
outside them, so the worst case for one node is 2 000 + its 1 000
sources × (6 severities + 4 drop reasons), ~12 000 series. The
2 000 seats bound the series written *at once*, not over a range: a
rerank (every 10 min) that moves a seat starts a new series, so a range
counts every template that held a seat during it. Log series share
VictoriaMetrics' `-storage.maxHourlySeries=500000` with the host
metrics; a node at its seat cap adds at most one series per new seat
taken.

**How it arrives.** `pushLogs`, `POST /api/buckets/{id}/logs`, is the
agent's own route (`link-protocol.md` §6b in 3pm): a batch every 10 s,
gzip NDJSON, the node's fleet bearer, any active node of the bucket.
The `welcome` says `logs {apps, host, files}` only when `--events-url`
is set; without it the route answers `503 ERR_STORE_DISABLED` and the
agents ship nothing. Refused before the body is read: no
`content-length` or over 1 MiB (`413`), a batch of the node already in
flight, or the node over its budget — 128 KiB/s inflated, 4 MiB burst
(`429` + `retry-after`). A body inflating past 4 MiB is a `413`, one
that is not gzip or does not open with `batch` a `400`. Records that do
not read, are of an unknown kind or name an undeclared source are
skipped and counted. A retried batch (same `run` and `seq`) is stored
once, a line once per `(node, src, pos)` (`pos` is opaque to the backend:
hashed with `src`, never parsed). A store that does not take the
batch is a `503 ERR_STORE_UNAVAILABLE`; the node retries, and only the
half still owed is written. At most 32 batches of the whole fleet are
read and stored at once: past that, `503 ERR_STORE_UNAVAILABLE` +
`retry-after: 1` (the agent backs off and sends the same batch again).
A record whose identifiers or labels (`src`, `run`, `path`, `app`,
`format`, owner fields, `ident`, `unit`, `pos`, a parameter's `var`)
hold a control character is skipped and counted: they become series
labels, stream fields and listing cells (a line's own text may hold
any). New sources or templates past the node's catalog bound evict its
least recently seen rows — never one the batch names (a source a
template of the batch belongs to included); refused only when the batch
itself fills the bound, then said in the answer (`refused {sources: [ids],
templates: [[src, tpl]], reason: "catalog_full"}`, threepm-link
`LogsRefused`; each list sorted and cut at 200, the node re-sends the
rest) and logged. A body slower than 30 s is a `408`; the
fleet-wide slot (32) is taken only once the body is read. `GET /api/v1/info` → `store.logs` counts
batches, duplicates, records, lines stored, bytes, refusals, failures.

**Where it is kept.**

| data | store |
|---|---|
| sources | SQLite `log_sources` (1 000 per node, least recently seen evicted first) |
| templates (text, severity, first/last seen) | SQLite `log_templates` (50 000 per node, the same; a template counted is marked seen once a day, a re-declared one too) |
| counts | VictoriaMetrics `threepm_log_messages{cluster,node,src,tpl,sev}`, one point per window; per node the 2 000 busiest templates labelled, the rest `tpl="other"` (the seats survive a restart) |
| drops reported by the node | VictoriaMetrics `threepm_log_dropped{…,src,reason}` |
| shipped lines, parameter stats | VictoriaLogs tenant `0:1` (`kind=line\|param`), stream `cluster,node,src` |

Events stay in tenant `0:0`; logs share the events instance and its
retention (`-retentionPeriod=30d` in `ecosystem.toml`), so a range never
goes past 30 days. The catalog follows: once an hour the sources and
templates no node named for 30 days are removed (`src/logs/retention.rs`)
— a daily-dated file no longer holds a source seat for good.

**Where it comes from.** `src/http/api_logs.rs` and `src/logs/` (budget,
parse, dedup, tally, render, sink) for the ingest;
`src/http/api_logs_query.rs`, `src/query/logs.rs`,
`src/query/logs_assemble.rs`, `src/store/victoria/logs_read.rs`,
`src/db/logs_read.rs` for the query; `src/db/logs_upkeep.rs` (eviction,
retention, seats and last-seen writes); migrations V21–V22. Tests:
`tests/it/logs_e2e/`.

## Insights: what deserves a look

`GET /api/buckets/{id}/insights` answers the question a fleet keeps
asking: what is wrong, or about to be, and where. The incidents are
ranked by the insights apps, nine Python processes that read this
backend's store next to it (`insights/` in this repository): absolute rules (a
disk full, a reboot loop, a collector silent), resources running out
or leaking, a machine unlike its peers, a series unlike its own past,
dated level shifts, series profiles, crash loops. Their `triage` groups
what they find on one machine and metric family, or on the fleet,
into an incident scored 0 to 10, and says how to read it: `problem`
(bad in itself), `context` (only unusual, damped), `inventory` (a
standing fact, an old agent, a worn battery).

**What you see.** Text for an LLM by default, one row per incident:
`id, score, tier, hostname, metric, labels, sources, since, passes,
label, summary`. With `?format=json` (the generated client's
`insights(clusterId, query)`), each incident carries its catalog
`metric` and a `chart`, a `queryMetrics` body that draws its headline
series, so a page draws an incident without knowing the store.
`summary` is the detectors' sentence, in French, with the figures;
`lang` says so. `label` is a person's verdict (`true`, `false`,
`known`; a `known` incident is listed only with `known=true`).
Filters: `tier`, `node` (its incidents and the fleet-wide ones that
name it), `min_score` (on the score as served), `limit` (20 by default,
at most 200). A fleet-wide incident has no `node_id`: its machines are
in `hosts`, and its `chart` draws them.
`insight(clusterId, incident)` gives one incident with the findings it
groups (detector, series, score, weight, summary), the process events
of its node over 6 hours, and the incidents that started with it.
`stale: true` says the ranking stopped moving (no pass for two
intervals), `error` why the last pass failed; `sources` the state of
each detector at the last pass. An incident id goes URL-encoded in the
path (a fleet-wide one reads `flotte@<cluster>:<family>`); an id the
list does not hold is a 404 answered without asking triage.

**From the logs.** The `logs` detector (`insights/apps/logs.py`, port
9708) reads what the nodes ship: the counts per template in
`threepm_log_messages` and one shipped line per template (VictoriaLogs,
tenant `0:1`), kept in an exemplar index of the last 7 days. Every log
finding says which incident it belongs to (a failing disk's kernel lines
join the disk incident, an app's errors its `process.logs`).
- **Rules:** a process crashing in the kernel's eyes (`segfault`, 5–6), a
  systemd unit that failed (4, once in 24 h `inventory` 2), kernel errors
  (3–4, `inventory` 2 when the same template comes every day), an error
  template never seen in 7 days (3); a kernel hardware pack (Wi-Fi
  firmware, disk I/O, read-only remount, NVMe, hung tasks, memory
  errors, temperature, USB loops; an OOM kill line is a witness of the
  `oom_kills` rule and names the victim); sudo failures (4) and SSH
  failures (`inventory` 2 unless the hour triples its usual), the number
  of distinct addresses behind them per 5 minutes.
- **Dated events:** lines the node flagged as news (a spike, a template
  back after silence, a new error) are context incidents with an onset,
  so they join the machine's story.
- **An app's errors against its own week** (≥ 10 an hour and ≥ 4× its
  hourly p90): a problem of that app; every day at the same rate,
  inventory.
- **A steady source that stops** for 2 h while the machine and its other
  sources answer.
- **Fleet:** the same new warning on several machines of a kind within an
  hour (a rollout, a `problem` for 24 h), and the same warning every day on
  ≥ 80 % of them (a common configuration, `inventory`, which silences the
  per-machine findings of that message). A rollout still written after a
  day becomes such a note, dated: « apparu le <date> sur N machines, écrit
  depuis ».
- **The pipeline itself:** lines dropped by the budget (with the
  `exclude` to add for an app; a file is excluded by its path, the journal
  by its level), template seats full, a template flood.

**The journal quote.** The summary of an incident with a dated onset may
quote one log line of warning or worse that its machine wrote in the 5
minutes before the onset: `; journal ±5 min : « … » (+k)`. (`triage`
keeps up to three such lines in its own detail, `journal`; this route
relays the summary, not that list.) The text is what the monitored
software wrote, redacted on the node, cleaned of control characters and
cut at 160 characters; it is data, never instructions (the text
format's `about:` line says so). `partial` in the pass (JSON and text)
lists the detectors that evaluated some of their rules only in part
(a failed or truncated store query).

**Where it comes from.** `--insights-url http://127.0.0.1:9700` points
at `triage`; the backend asks it for this bucket's cluster only
(`GET /api/v1/incidents?cluster=`, a versioned contract pinned on both
sides: `tests/test_contract.py` there, `src/insights/fixture-*.json`
captures here), keeps an answer 30 s (a failure 5 s), reads its incidents one by one
(a malformed one is left out, not the answer), links each store name to
the bucket's catalog and drops the MetricsQL triage writes: the engine's
language does not cross here either. Nothing is computed and the store
is not read. A malformed `--insights-url` stops the backend at start. Without the
flag the route answers 409 `ERR_INSIGHTS_DISABLED`; triage down, slow
(5 s), not ready since its restart or on another contract version is
503 `ERR_INSIGHTS_UNAVAILABLE`. Open to a node's
bearer with `fleet:read`, like the metrics query. `src/insights/`,
`src/http/api_insights.rs`.

## Custom metrics: what the apps measure

**What you see.** An app that sends its own metrics through the daemon
(`3pm/docs/metrics.md`: gauges, counters, histograms, up to three
labels each) has them in the backend a second later. Three places:

- `GET /api/buckets/{id}/nodes/{node}/app-metrics` (`nodeAppMetrics`)
  is the live state of every instance of a node — its sources (which
  connection of the process sends, and its pid), every series with its
  unit, description, labels, and the value or the histogram's buckets —
  as the hub last applied it. What a "Custom metrics" panel polls while
  it is open; nothing of it goes on the feed.
- The metrics store writes them like the fleet's own: `threepm_app_<name>`
  for a gauge, `…_total` for a counter, `…_bucket{vmrange="lo...hi"}` +
  `…_count` + `…_sum` for a histogram, with `app`, `instance`, `ns`,
  `src` and the app's labels. `http.latency` becomes
  `threepm_app_http_latency`.
- The catalog (`GET …/metrics/catalog`) lists them as `app.<name>`
  (`source: "app"`) with their labels, so a chart query can name them:

```
{ "range": { "from": "now-1h" },
  "queries": [{ "ref": "A", "metric": "app.http.latency",
                "time": { "fn": "quantile", "q": 0.99 },
                "filters": [{ "label": "route", "op": "eq", "value": "/api/bucket/:id" }],
                "aggregate": { "fn": "sum", "by": ["app"] } }],
  "options": { "explain": true } }
```

A histogram takes `quantile` (with `q`; the median when `time` is
omitted), or `rate` / `increase` of its count; `raw` and the
`*_over_time` family are refused on it, as is a label the metric never
carried. `options.explain` returns `results[].expr`, the expression
exactly as compiled for the engine — to read, never to send back.

**Events.** An app's events (`io.emit(...)` and friends: a name, a
level, attributes, an exception, a fingerprint and a repeat count)
reach the feed as `emit` (fifty a second per bucket at most) and the
events store as `kind: "app"` lines dated by the app's clock, with the
attributes flat under `a.` and `exception.*`. `POST
/api/buckets/{id}/events/query` (`eventsQuery`) reads them back —
newest first, a range, exact filters (`kind`, `app`, `level`,
`event_name`, `fingerprint`…), a limit — without the engine's language
ever showing. A node may emit a hundred application events a second
(burst 500); the rest are dropped and counted.

**On the dashboard.** Each node card has a "Custom metrics" panel,
closed by default: open, it polls `nodeAppMetrics` every second and
shows each instance's sources and series (value, unit, description,
labels; a histogram's count, sum and approximate p50 / p90 / p99).
The "Events" list shows the live feed on top and the last day from
the store underneath.

**How it behaves.** An instance's metrics are the `app/<id>` section of
its node's state (see "The node's whole state, received as diffs"): the
agent sends every `--app-metrics-every-s` seconds (1) what moved in it,
the backend rebuilds the section and replaces the instance's state
whole each time — so an instance a restart gave a new incarnation
keeps nothing of the old one, and a lost frame is that section asked
again (`state_resync`), at most once a second per node. A state over
the caps (1 000 series, 128 buckets a histogram, 100 instances a node)
or whose name is off the label grammar is refused and counted
(`nodeAppMetrics.refused`), and nothing is held for the instance. It
goes with its section (`gone`) and with its row of the process table.
The catalog remembers a metric for 30 days after it was last seen and
holds 5 000 per bucket (`info.store.catalog_over_cap` counts the rest).

**Where it comes from.** `hub/node_state.rs` (the section), `hub/apps.rs`
(the state per instance, the caps), `store/app_samples.rs` (the lines),
`query/dynamic.rs` + `db/app_metrics.rs` (the catalog and its table),
`http/api_app_metrics.rs`; the wire is the `app/<id>` section of
`AgentFrame::State` in `threepm-link` (`3pm/docs/backend-agent/link-protocol.md`).

## Process trees: every process of an app

**What you see.** An app is every process it spawned — `npm run dev`,
the `node` it starts, Electron and its helpers — and 3pm reports it so
(`3pm ls --tree`). The backend has the same picture, in three places:

- The process table of each node card writes `saw +13` after an app
  with other processes (`proc_count` on the telemetry's record, the
  spawned pid included); the link opens the card's **Process trees**
  panel on that app. Open, the panel polls `nodeProcessTrees` every
  second and draws, per app with more than one process, the folded
  tree `3pm ls --tree` draws — a chain of single children on one row
  (`npm › bash › node › Electron`), twins grouped (`Electron Helper
  ×4`), `⚠` on a process that left the app's process group and that
  `3pm stop` will not reach — with pid, CPU, memory and uptime (no
  ports: they do not travel on the link, the process table's
  `listen_ports` is their union), then every process under a fold.
  The title says how many trees, and how many were refused.
- `GET /api/buckets/{id}/nodes/{node}/trees` (`nodeProcessTrees`) is
  what the panel reads: one entry per instance the hub holds a tree
  for, each with `procs` (every line: `procs[0]` the spawned pid, a
  parent before its children) and `compact` (the folded rows, computed
  by the same code as the CLI's), the totals of the same frame, and
  the `refused` count. `GET …/procs/{proc}/tree`
  (`nodeProcessTree`) is one instance, 404 while the backend holds no
  tree for it: an instance running alone, or before its first tree.
  Nothing of it goes on the feed: at 300 KB a second for two hundred
  apps of fifteen processes, a broadcast would cost every open
  dashboard the whole fleet.
- The metrics store writes one series set per **group** — the members
  of an instance's tree sharing an executable name — for as long as
  the agent sends the tree: `threepm_subprocess_procs` (the members),
  `threepm_subprocess_cpu_ratio` and `…_memory_rss_bytes` (summed),
  `threepm_subprocess_detached` (how many left the process group), on
  the app's labels plus `proc` (the executable's name), and the tree's
  count as `threepm_process_procs`. There is no `pid`: keyed by pid, a
  kiosk's shell loop forking a `sleep` every 3 s made 17 000 series
  sets a day per tablet (735 000 new series on 2026-10-01, the engine's
  daily cap within reach of 25 tablets, and past it every node loses
  its new series). By group, the count follows the programs. Extended
  Monitoring's gauges are summed only when every member gives them (a
  partial sum would read as a drop), the fd limit becomes
  `threepm_subprocess_fd_usage_ratio_max` (the highest open / limit
  among the members: the `EMFILE` question, which a sum of limits could
  not answer), and the ten counters are accumulated per group by the
  backend: a member is `(pid, start time)`, its first reading a
  baseline when it was already running at the group's first sight
  (a backend restart then reads as a counter reset, not a spike) and
  whole when it started after, and a member that exits keeps its share
  — the counters never go down while the group lives (a group unseen
  for a minute starts again from 0). What is lost: the history of one
  member among homonyms (which of eight `electron` grew), and what a
  process that lives and dies between two store ticks did; the member
  by member detail stays live (the tree panel, `nodeProcessTree`). The
  catalog lists them as `subprocess.*` and `process.procs`; a fleet
  query wants an `aggregate.by`, like the host rows. `proc` and `pid`
  stay reserved label keys for applications, in 3pm too, and a query
  naming `pid` is refused (400).

**How it behaves.** An instance's tree is the `tree/<id>` section of
its node's state (see "The node's whole state, received as diffs"): the
agent sends the *shape* whenever an instance's membership moved, and the
*figures* every `--trees-every-s` seconds (1 by default; `0` = the
shape only, when it changes) for every instance with more than one
process. The hub keeps the last tree per `(node, instance)`, replaced
whole each time its section moves — a restart's new incarnation
included — drops it when the section is gone (the app runs alone now),
and forgets it with the instance's row of the process table. Nothing is
asked by heuristics any more: a lost frame is a gap of that section,
answered `state_resync`. Guards: 100 trees per node, 256 lines a tree
and 64 bytes a name (the daemon's own caps, enforced again), and a tree
for an instance the process table does not list is refused; each is
counted in `refused`, and the tree held for the instance goes rather
than stay behind its section — never an empty tree, which takes no
place. Member names are cleaned again on the way in, as the daemon
does — control characters, bidi overrides and line separators blanked —
so no agent can bend the panel's lines. An older 3pm reports the
spawned pid alone. The history's resolution is the store's
(`--store-interval-ms`, 5 s by default); the panel is live at 1 s
whatever it is. A tree held is current as of the node's every frame (an
agent sends it only when it moved), so the store keeps writing its
members' figures while the node is heard — with `--trees-every-s 0`
those are the figures of the last shape frame — and stops a minute
after the node's last frame; the count (`threepm_process_procs`) comes
from the process table.

**Where it comes from.** `hub/node_state.rs` (the section),
`hub/trees.rs` (the state per node, the guards), `http/api_trees.rs`
(the two routes over `threepm_protocol::tree::compact`),
`store/tree_samples.rs` (the lines) + `store/tree_groups.rs` (the
groups, their sums and accumulators, carried by `prefix.rs`), `query/catalog/lines/{process,subprocess}.rs`,
`assets/pages/fleet.js` (`processTreesPanel`, `treeGuides`, whose
escaping `tests/it/dashboard_render.rs` holds); the wire is the
`tree/<id>` section of `AgentFrame::State` and `welcome.pace.trees` in
`threepm-link` (`3pm/docs/backend-agent/link-protocol.md`). Tests:
`hub/trees_tests.rs` (replacement, the guards, the two ways out),
`store/tests_tree.rs`, `tests/it/trees_e2e.rs` (a fake agent: served,
stored, refused, cleaned, gone, the row gone) and
`tests/it/real_agent_trees.rs` (a shell and two sleeps under a real 3pm
and its agent).

## Remote actions and profiles: run what the apps expose

**What you see.** An app declares actions through its SDK (`io.action`
in `@3pm/api`); the daemon publishes them with the sources of each
instance, the agent relays them, and this backend lets the operator run
one from the dashboard or the API and read the result — or the file it
left: the SDK's built-in `3pm:profile` takes a CPU or heap sampling
profile, stored here as an artifact with its hot frames.

```console
$ curl -sS -H "Authorization: Bearer $TOKEN" -X POST \
    http://127.0.0.1:9616/api/buckets/3pmc_…/nodes/3pmn_…/actions \
    -d '{"app": "api", "action": "cache.flush", "params": {"all": true}}'
{"call_id":"kq3Z…","node_id":"3pmn_…","request":{"op":"action","target":{"name":"api"},"action":"cache.flush","params":{"all":true},"timeout_ms":5000},"state":"sent","sent_at":1726300000000,"deadline_at":1726300020000,"artifacts":[]}
$ curl -sS -H "Authorization: Bearer $TOKEN" http://127.0.0.1:9616/api/buckets/3pmc_…/calls/kq3Z…
{"call_id":"kq3Z…","state":"replied","result":{"res":"triggered","results":[{"proc_id":3,"name":"api","instance":0,"source":"main","action":"cache.flush","duration_ms":12,"ok":true,"data":{"flushed":4}}]},…,"artifacts":[]}
$ curl -sS -H "Authorization: Bearer $TOKEN" -X POST …/nodes/3pmn_…/actions \
    -d '{"app": "api", "action": "3pm:profile", "params": {"kind": "cpu", "duration_ms": 10000}}'
$ curl -sS -H "Authorization: Bearer $TOKEN" …/api/buckets/3pmc_…/artifacts?node=3pmn_…
[{"artifact_id":"a9Xf…","app":"api","instance":0,"source":"main","action":"3pm:profile","ext":"cpuprofile","bytes":184322,"gz_bytes":41210,"sha256":"…","summary":{"kind":"cpu","samples":9500,"top":[{"function":"hot","location":"/srv/app/worker.js:6","self_value":6120000,"percent":62.4},…]},…}]
$ curl -sS --compressed -H "Authorization: Bearer $TOKEN" -o hot.cpuprofile …/artifacts/a9Xf…/file
```

In the dashboard, an app that declares actions has them right under
its row of the node card's process table — under its CPU and memory,
as `3pm ls` would list them: a button per action and per source (`CPU
profile 10 s` and `Heap profile 30 s` for `3pm:profile`, with a
countdown; a params field for the rest — the button's tooltip is the
declared description and timeout). Click one: the **Action results**
dialog opens on the call — `sent…`, "waiting for the node…", then
`replied` with the handler's JSON folded under it (or the error in
words), fetched with `getCall` as the feed reports each step; the last
10 calls stay in the dialog. The rows are fed by `GET
…/nodes/{node}/actions` (`nodeActions`) — the declarations alone,
nothing of the metrics — polled every 3 s for every node that is not
offline; a row is repainted only when the declarations move, so what
is typed in its params field survives the process table's one-second
refresh (the table is kept by app id, not rebuilt). The buttons run
as the logged-in account, which owns the bucket. The
"Artifacts" panel lists what the node's actions left — the top 5
frames of a profile, raw and gzip sizes, a download per file. The feed
carries every step (`call` events: `sent`, `acked`, `replied`, or
`unacked`, `refused`, `lost`) and every file (`artifact` events), but
only states, ids, timings, sizes and error codes: every open page of
the bucket reads it, so params, results, messages and summaries stay
with `getCall` and `listArtifacts`.

**How it works.** `nodeAction` builds the link's `action` command
(`nodeCall` sends `restart`, `stop` or `ping`; an `action` there is a
400), records the call (`calls`, at-most-once) and hands it to the
node's live session (offline: 409; its queue full: 503
`ERR_NODE_BUSY`, the call recorded `refused`); the agent acknowledges,
asks its daemon (`trigger`, always `remote: true`, so an action declared
`remote: false` answers `refused`) and replies with each instance's
outcome. A handler's `data` travels gzip + base64 **in the reply**,
while the call's results stay under 256 KiB together; a result past
that fails its own outcome (`too_large`) and is a file to return
instead. A **file** an action returns never travels on the link at
all: before its reply, the agent uploads it to `PUT
…/calls/{call}/artifacts` with the bearer of its `welcome` — raw
bytes, at most what the action declared it may answer (32 MiB when it
declares none), one per instance and sixteen per call — and the reply
names the artifact the upload minted, with the size and SHA-256 of the
raw bytes; a reference that is not the upload's fails that outcome, and
an upload no reply claims — or whose call settled while its bytes were
arriving — is removed. A bucket takes 2 GB of files in flight at most,
whatever its nodes declare; retention keeps it at 1 GB. The bytes
are gzipped and hashed as they arrive and stored under `--data-dir`
(`<db>.d/` by default) as
`artifacts/<bucket>/<node>/<id>.<ext>.gz` — never in SQLite. The call
is settled first; a profile (`cpuprofile`, `heapprofile`) is summarised
right after, two at a time (`threepm_core::profile_summary`, the same
code `3pm profile` prints with), so its `summary` reads `null` for a
moment. Retention: 30 days and 1 GB per bucket, oldest out; a call
never answered by its deadline (the caller's `timeout_ms`, else the
timeout the app declared, 30 s when none, plus `--call-margin-ms`,
15 s) is settled by the sweeper — `unacked` when the agent never saw
it, `lost` when it did; one a reply has started to settle gets 60 s
more.

**The locks.** The host must allow `proc:action` — its `link.toml`
`caps`, declared in every `hello` and shown on the node's row, the
agent itself refusing what it does not carry; the action must be
declared by the app,
and declared `remote`; the routes are `guarded` — from another site a
session cookie (or `Basic`) is refused 403, an `Authorization: Bearer`
passes (see "From another origin: CORS is open") — and answer only the
account that owns the bucket — another account gets the 404 of a
missing bucket, nobody a 401 — or, for `nodeAction`, `getCall` and the
artifacts, a node of that bucket whose bearer carries `fleet:invoke`
(see "Node calls"), since they run code on the nodes and serve what it
produced (a profile shows an app's code). The upload is the one route
no account may call: only the node the call was **placed on**, which
needs no scope for it — answering a call must not cost the right to
place one.
Every call is audited
on the node (`proc_action` on its bus) and here (`kind: action` in the
events store, the feed's `emit`).

**What there is not.** No roles, and no permission per action, per app
or per node for an account: its tokens (`3pmu_…`, see "Accounts") carry
no scope, and whoever holds one — or a session it opened — can run
every `remote` action of every app on every node of every bucket the
account owns, restart or stop any app, and read every profile ever
taken there. Without a credential the routes are closed, not open. The
finer grains are elsewhere: a node calling another is held to the
target's `callers` rules (by calling node, application and action, see
"Node calls"), and the app has its own — `remote: false` keeps an
action off this path altogether, and a handler is expected to validate
its `params`. Access finer than the account, if it comes, is a backend
feature of its own; nothing on the wire assumes it.

**What a profile is.** `3pm:profile` takes a *sampling* profile, never a
dump: `kind: cpu` samples the stack (a `.cpuprofile`, default 1 ms),
`kind: heap` samples allocations (a `.heapprofile`, `HeapProfiler`
sampling, default every 32 KiB allocated) and says which functions
allocate — not what the heap holds. A heap snapshot is deliberately
not offered from the fleet: it freezes the JavaScript thread while it
is taken, weighs hundreds of MB on a large renderer and carries the
app's data, secrets included.

## Node identity and fleet access

**What you see.** Each node's card has an **Access** panel: three boxes —
*List the fleet*, *Read the fleet*, *Place calls* — and a list of who may
place a call on that node. Nothing is granted when a node enrolls, so a
freshly linked host can read nothing of its bucket. The node table shows
a badge per scope a node holds. What you grant or take away is felt on
that node's very next request; nothing has to reconnect.

**Where it comes from.** Every `welcome` hands the node its own bearer
(`3pma_<10 hex>_<64 hex>`), so `3pm fleet …` on a server authenticates as
*that node* and the owner's account token never has to sit on it. The
bearer says which node is calling and nothing more: the node's row —
its state and its scopes — is read again on every single request, which
is what makes a grant and a revocation both take effect at once. The
bearer lives as long as its WebSocket session plus ten minutes (a short
cut does not break a command in flight), is dropped on the spot by
*Revoke*, and is never persisted: a backend restart re-mints at the next
handshake. Only `sha256(bearer)` is ever kept in memory, and the token
never reaches a log.

Serving a call needs no scope at all. `getBlob` and `putBlob` admit a
node that is **party to the blob** — the call's writer or its reader —
whether or not its row carries anything, because the blob is the
authorisation and always was: one that is not this node's reads as a
404, exactly like a blob that does not exist. Asking a target for
`fleet:invoke` before it could read the bytes sent to it would have
handed it, in exchange, the right to call every one of its neighbours.
`fleet:invoke` is what it takes to **place** a call, and nothing else
opens that way: a surface test names the two routes.

A node's bearer reaches fourteen read routes and nothing else — never
`approveNode`, never an enrollment token, never an account, and never
the panel that would grant it more. What those routes answer depends on
the scope: with `fleet:list` a node reads the fleet as a list — every
node's identity and its process table, as `3pm ls` shows it — and its
own grant, never a neighbour's (who may invoke what on whom is the map
you would need to move sideways, and it stays with the owner). With
`fleet:read` it also reads the hosts, the daemons, the per-process
history and the applications' own events. Of its calls it sees only the
ones it placed itself: `getCall` answers as if a neighbour's call did
not exist, the artifact list leaves them out, and the live feed is
filtered and projected the same way — one rule, whether a node asks a
route or reads the stream. `GET /api/v1/routes?route=<name>` names the
scope each route needs.

## Node calls: one host asking another

**What you see.** A node places a call on another node of the same
bucket with its own bearer, through the same `nodeAction` an operator
uses. The caller needs `fleet:invoke`, and that is enough: it may run
any action of any other node of the bucket, `3pm:profile` included. The
**target** can narrow it in its Access panel ("Who may call this node"):
empty, it accepts every node that may call; once it holds a rule, only
what a rule admits goes through, and an action of the `3pm:` prefix has
to be named in full there. A rule no call matches, `{"from": "*",
"app": "-"}`, closes the node to all of them. On the dashboard a call reads `by 3pmn_…
(edge-7)/edge` — the node, and the application its daemon said was
asking; `-` stands where no application was named, which is what `3pm
fleet call` from a terminal looks like. That application name is
declared by the **calling node's own daemon**: it is an operational
guard rail for reading a fleet's traffic, not a boundary — the boundary
is the node, and any node a rule admits can send any `from_app`.

A node may also call **itself**, under the same rules (`fleet:invoke`,
and a rule that admits it once its Access panel holds one): the bytes
loop through this backend and back into the agent that placed the call.
It is how two applications of one host share a session, since a daemon
never opens a `live` leg on its own — `io.open('<own node id or
hostname>', …)`.

**Linking two machines from a script.** Everything the Access panel
does is one route, `setNodeAccess`, sent with the owner's account token.
To let the node A call `kiosk/show` on the node B:

```bash
T=3pmu_…; U=https://fleet.example.net
curl -s -H "Authorization: Bearer $T" $U/api/buckets                          # the bucket: cluster_id
curl -s -H "Authorization: Bearer $T" $U/api/buckets/$C/nodes                 # node_id by hostname, scopes, callers
curl -s -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
  -X POST $U/api/buckets/$C/nodes/$A/access -d '{"scopes":["fleet:invoke"]}'  # A may place calls
curl -s -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
  -X POST $U/api/buckets/$C/nodes/$B/access \
  -d '{"callers":[{"from":"'$A'","app":"kiosk","action":"show"}]}'            # B accepts A for kiosk/show
```

Each list sent replaces the node's list whole (read it from `listNodes`
and send the old rules back with the new one); a list left out is
kept. Both answers are the node's new `NodeView`, and the change holds
from the node's next request. The same two bodies are the "Link two
machines" recipes of `GET /api/v1/routes`.

A call can carry bytes. `input` (how many, their SHA-256, their
extension) opens a blob the caller writes and the target reads; `output:
true` opens one the other way. Neither application ever speaks to the
network: the SDK writes to its own agent's socket, the agent relays, and
what comes back is a file on disk. The answer to `nodeAction` names both
blobs, so nothing has to be parsed out of the request. A call that
carries bytes either way has to name its `instance` — sixteen instances
would each pull the same hundred megabytes, or write over one another's
answer — and `deadline_at` allows for the transfer as well as for the
handler: half a gigabyte is given time to travel, a small file reserves
a minute, not ten.

Both blobs are charged to the node that **placed** the call, including
the one its target writes: a caller you accept spends its own allowance
and never yours. The output blob reserves what the target's action
declared it may answer — `io.action(name, {output: {max_bytes}})`, as
the node published it, never a size the caller names — and 256 MiB when
it declares nothing (an older agent, or an action without the bound):
a 2 MB screenshot costs its caller 2 MB of quota, so dozens can run at
once where eight did. `nodeAppMetrics` and `nodeActions` show it as
`output_max_bytes`. The blob is given back the moment the call is over
and nobody claimed it. A node has at most a hundred blobs in flight at
once — a blob stops counting the moment it has been relayed, not when its
retention ends, so a caller downloading output after output never runs
out of room — and opens at most two hundred a minute; a blob that **carried nothing** — no byte written,
no reader ever attached — gives its creation back when it is forgotten,
so a call refused by a rule, by a node that was not there or by a target
that said no costs the retry nothing. One byte, or one reader, and it is
spent for the minute like any other. An input bigger than `--blob-max-mb` is refused before
anything is reserved, because a blob holds what it declares against its
bucket's quota for as long as it is open.

**Where it comes from.** The target's answer is pushed into the output
blob and announced on the outcome. The backend keeps no copy: the
answer stays the call's blob, and `getBlob` reads it until the relay's
own retention takes it. The outcome must name the blob this call opened,
with a well-formed digest and extension, or it fails; so does a second
instance trying to claim the one blob the call opened. The backend never
reads those bytes — the relay never hashes either — so the digest the
target announced is its claim, relayed as is.

Once the call is over, `getCall` says `result.ok` (the action ran; each
instance still carries its own `ok`) and fills the output in: how many
bytes, which digest, which extension. That is what lets the side that
pulls the blob check it against something rather than trust the relay.

When the call settles, the node that placed it is told on its own link,
without asking — the backend re-reads its row first, so a node whose
`fleet:invoke` was taken away in the meantime hears nothing and reads
the result through `getCall` if the scope comes back. If it was not
connected at that moment, its next `welcome` hands it the calls it
placed that are over (the last twenty-four hours, at most 256 ids,
newest first); the agent ignores the ones it already knows and reads the
rest with `getCall`. Nothing is queued, and nothing is replayed: a call
happens at most once.

### Between LAN neighbours: the bytes skip the relay

**What you see.** A node calling another node that sits on the same LAN
(same public address, as this backend sees both links) gets `peer` in
the answer that placed the call — the target's node id, public key and
private addresses — and the target's ticket names the caller the same
way. The two agents move the call's file between themselves over TCP +
TLS (each presents its node key; only the key the ticket names is
accepted). The call is placed, audited and settled here exactly as
before, and its blobs stay open as the fallback: when the direct
connection does not come up, the bytes take the relay without the
caller noticing. Once the writer knows, `path` on the call (`getCall`,
the feed's `call` events) says `direct` or `relay`.

**Where it comes from.** `calls/direct.rs`: the rule (both ends are
nodes, not the same one, both settings allow it, both announced a port
in `hello.direct_port`, same public address, a private address each),
the peers built from the hub's view of each node (`Hub::direct_of`:
the session's port, the address the link came from, the host's private
interfaces — at most eight), and `path_report` (`blobs/path.rs`: a
`direct` report from the blob's writer takes it off the relay and gives
its creation back — a file is forgotten, a `live` leg stays in the
registry, served by nobody, reserving nothing, alive on its writer's
`blob_stat` — unless the relay carried a byte of it or has a reader). Per node:
`setNodeAccess.direct` — `off`, `files`, `all` (default: a stream's
live legs too, on both ends: the relay then has nothing to show a
viewer; set `files` on a node whose sessions must stay watchable). `--direct off` turns it off for the whole backend.

A session (`stream: true`) goes direct only when both nodes say `all`.
Its legs then never reach the relay; what `live` shows on the call is
what each writer reports every 5 s (`blob_stat`), marked `direct`, and
the dashboard has no Watch button for it — "direct · no preview". A
leg gone direct is over 15 s after its writer's last report — an end
and an abandon read the same — and it ends alone: the other leg, if it
went by the relay and is still writing, runs on (a short question, a
long answer); one that is idle or never started ends with it.

## Shells: `3pm fleet ssh`

**What you see.** `nodeShell` (`POST /api/buckets/{id}/nodes/{node}/shell`,
body `{ "term": "xterm-256color", "cols": 120, "rows": 40 }`) opens a
login shell on the node — the login shell of the unix user its agent
runs as, on a real pty of that size — for the bucket's account or for a
node holding `fleet:invoke` whose call the target's `callers` admits
for the action `3pm:shell` (`app` `-`; an empty list admits every node
that may call, a wildcard action never reaches it). A node the rules do
not admit reads 403 `ERR_SHELL_NOT_ALLOWED`, one without
`fleet:invoke` 403 `ERR_FORBIDDEN`, a bad `term` 422; a page on
another site is refused even with a `Bearer`, which `nodeAction` lets
through. The answer is the call: `input.blob`, the keyboard, which the
caller `PUT`s as one long streamed body, and `output.blob`, the screen,
which it `GET`s — two `pipe` blobs, lossless and with back pressure,
one reader each, carrying the `attach` messages of `3pm attach` (input,
resize, ping; output, pong, exited). The call stays in flight as long as
the shell runs, a day at most, with `live` counting the bytes each way;
a node's link that reconnects meanwhile does not end it. It settles
`replied` with `res: "exited"` and the shell's `code` or `signal` (`ok`
on 0), or with the agent's refusal — `forbidden` when the node took
`node:shell` out of its `link.toml` or runs an agent that does not know
the command. A leg quiet for a minute (the command line pings every 20
s, the agent answers), a `3pm fleet ssh` that went away, or a backend that stops, ends both legs and the
shell is hung up.

**One command instead** (`3pm fleet ssh <node> -- <cmd>`): the same
body with `"command": ["wc", "-c"]` — at most 256 words, 16 KiB in all,
no NUL, the first not empty, else 422 — and `"pty": true` for a pty.
The backend places `exec` rather than `shell` (an agent that does not
know it refuses it, and the legs end at once as for any refusal); the
node runs the command through the same login shell, on a pty or on
pipes, where the screen leg carries its stdout as `output`, its stderr
as `stderr` and the keyboard leg its stdin, closed by `eof`. Same
gate, rules, quotas, legs and end; `open_requested` (and `refused`)
carry `command`, the agent adds an `exec` line.

Every session is audited in the events store (`eventsQuery`, `kind:
shell`, `call: <call id>`): this backend writes `open_requested` (who,
from which host, the terminal) when it places the call and `ended`
(the status, the agent's refusal, or `lost` / `unacked` when the call
settled without the agent) when it settles — two lines a node cannot
leave out —, and `refused` (who, why, no `call`) for an attempt it
turned away once it knew the target (`callers`, the node offline or
busy, a quota); a session the agent refuses has its two legs ended at
once. The agent adds `open` (user and shell), `line` (each
command line as the screen showed it, `[hidden]` where the terminal did
not echo, the raw input under a full-screen program), `screen` and
`close` (duration, bytes each way, how many lines it sent: a gap in the
store shows against it). `line` and `screen` come only from a session
that is recorded: without `--record` (the command line's default,
bound in the handshake),
`record_shells` off or a node with `[shell] record = "never"`, the
agent sends `open`, `exec`/`attach` and `close` with `lines: 0` — no
gap. None of them is rate-limited, and none goes on
the bucket's feed, which nodes holding `fleet:read` follow too. What
the agent writes is as complete as the screen lets it be, and no more:
the backend's lines are the ones to trust. A `line` names the call its
agent says it belongs to; the backend does not check that the call is
that node's, so read a session's lines with `node` as well as `call`.

**Where it comes from.** `http/api_shell.rs` (the route),
`calls/shell.rs` (the legs, the backend's audit lines, the shell whose
legs are gone), `db/calls_shell.rs` and `db/calls_abandon.rs` (a shell
outlives its connection), `store/shell_line.rs` (the lines), `blobs/`
(`pipe`).

**Attach.** With `attach: {app, instance?, mode?, read_only?, peek?}`
(never with `command`, an app of 1 to 256 bytes without a control
character, `mode` `rendered` or `raw`, else 422), `nodeShell` places an
`attach` call (`3pm fleet attach <node> <app>`): the terminal of a
`tty = true` app of the node, relayed by its agent from its daemon's
own `attach`, on the same two legs, under the same rules. The app is on
the `open_requested` (and `refused`) line; the agent's `attach` line
follows its `open`. An attach call is a shell's everywhere a shell's is
found (`is_shell`, the sweeps of gone legs and of a reconnect).

**As another account.** With `user` (a unix login,
`[a-z_][a-z0-9_-]{0,31}`, never with `attach`, else 422), `nodeShell`
wraps the shell or command in `run_as {user, req}` (`3pm fleet ssh -l
<user>`): a new op, so an agent that predates it refuses it as unknown
rather than ignoring a field and running as itself. The target's agent
becomes that account only when it runs as root, and otherwise takes
only its own name. `user` is on the `open_requested` and `refused`
lines. Every reader of a stored request reads the wrapped one
(`calls::shell::unwrapped`: `is_shell`, the legs, the call's view, the
claimed output; the SQL of the shells in flight and of an abandoned
link lists `run_as` beside `shell`/`exec`/`attach`).

**Port forwarding** (`3pm fleet ssh -L [bind:]port:host:port <node>`).
`nodeForward` (`POST /api/buckets/{id}/nodes/{node}/forward`, body
`{host, port}`: a host of 1 to 255 bytes without space, a port 1 to
65535, else 422) places one `forward` call per TCP connection the
command line accepted: the node connects to `host:port` (any
destination, the owner's switch is the node's `node:forward`
capability, on by default) and pumps the bytes. The answer and the
legs are a shell's, carrying `live` frames of raw bytes; an empty
frame flagged `END` is a half-close of that way. The same guard as a
shell (account, or a node's `fleet:invoke` the target's `callers`
admits), but its own action, `3pm:forward` — a `3pm:shell` rule does
not open a tunnel — and its own refusal, 403 `ERR_FORWARD_NOT_ALLOWED`;
never a `Bearer` from another site. A connection the node cannot make
settles the call with the agent's `err` (`connect host:port: …`).
Audit, `kind: shell`: `open_requested` / `refused` carry `forward` =
`host:port`, `ended` when it settles, the agent's `close` the bytes each
way. A forward call is a shell's for every sweep (`is_shell`, gone legs,
a reconnect).

`-W host:port` (`3pm fleet ssh <node> -W host:port`, ssh's
`ProxyCommand`) is one such forward whose local side is the command
line's stdin and stdout: nothing of its own here.

**Reverse forwarding** (`3pm fleet ssh -R [bind:]port:host:port
<node>`). `nodeListen` (`POST /api/buckets/{id}/nodes/{node}/listen`,
body `{bind, port, e2e}`: `bind` empty for the node's loopback, else a
plain host of 255 bytes at most; `port` 0 for one the node picks; `e2e`
must be `true`, else 422; 409 `ERR_NODE_NO_E2E` for a node that cannot
encrypt) places one long `listen` call: the node binds, and its sealed
legs carry `bound {port}`, then one `accepted {conn, peer}` per
connection, the caller's keepalive every 20 s the other way — what
keeps the quiet legs under the relay's idle minute (no sweeper
change). Whether a non-loopback `bind` is allowed is the node's own
`[shell] gateway_ports`, which this backend neither reads nor sets.
Each accepted connection is taken over by a `nodeForward` with
`{accepted: {listen, conn}, e2e: true}` and neither `host` nor `port`
(422 otherwise): the node pumps that connection instead of dialing.
That forward is refused with **one 404** (`no running listen <id> of
yours on node …`) unless `listen` is a `listen` call on this node,
placed by this caller, still `sent`/`acked` — the same words whichever
failed, so a guessed call id says nothing of another caller's listen;
`3pm:forward` is checked first, so a caller it turns away learns
nothing of the listen either. Guarded exactly as `nodeForward`
(`3pm:forward`, `ERR_FORWARD_NOT_ALLOWED`, never a cross-site
`Bearer`). Never recorded. Audit, `kind: shell`: the listen's
`open_requested` / `refused` with `bind` and `port`, `ended`; each
connection taken over a forward's own line with `listen` (the call id)
and `conn` in place of `forward`. A
listen call is a shell's for every sweep (`is_shell`, the SQL of the
shells in flight and of an abandoned link). Code:
`src/http/api_listen.rs`, `api_forward.rs` (`listen_of`, `accepted`).

**Sessions are recorded when the caller asks.** Every shell, command
(`-- <cmd>`) and attach session placed with `--record` (`record: true`;
a current 3pm sends `false` without it, an older one that sends nothing
counts as asking) on a bucket whose `record_shells` setting is on — the
default, and every bucket that existed before the setting — is kept as an asciicast v2 file (`.cast`, gzipped on
disk) attached to its call: it is in the call's `artifacts`, in
`listArtifacts`, downloadable with `artifactFile`, and goes with the
artifacts' retention. What is recorded is **what the screen
displayed**: if a session runs `cat .env`, the secrets it prints are in
the recording. What is typed and not echoed — a password at a `sudo` or
`passwd` prompt — is not: the keyboard leg is never copied. The header
carries the terminal (`TERM`, columns, rows) and the title `<node>
<caller>`; events are output (`o`, stdout, stderr, an attach's screen
and diffs) and resizes (`r`), timed from the session's opening. The
backend's `ended` audit line names the recording's artifact
(`recording`). Bounded: 64 MiB of recording per session, then a marker
(`m`) and nothing more; the recording never slows a session — a
recorder that falls behind marks a gap and stops there. Not recorded:
forwards (`-L`, raw TCP) and copies (`3pm fleet cp`, whose screen is
the file: `threepm_link::cp::is_cp_transfer`, below). The other way round, a recorded session never
goes agent to agent when the backend is the one recording it (a
session in clear): only with `record_shells` off may such a session's
legs go LAN direct (`nodeShell` `direct: true`, both nodes `direct:
all`); an end-to-end session (below) goes direct recorded or not. The
owner turns it off on the settings page ("Record shell sessions"), or
`setBucketSettings {record_shells: false}`; recordings already kept
stay until retention.

**End to end: the backend relays what it cannot read.** A 3pm of this
version asks `e2e: true` on every `nodeShell` and `nodeForward` (shell,
command, `cp`, attach, `-L`): the session's bytes are sealed between
the command line and the target's agent (Noise XX), and the backend
relays opaque records on the same `pipe` legs — it holds no key, does
not tee them, and a byte it changed ends the session. A node whose
live session did not announce `shell_e2e` (an older agent) is refused
at once, 409 `ERR_NODE_NO_E2E` naming its agent's version; the caller
may retry in clear with `--no-e2e`, and that session is the old one,
teed and recorded here. An up-to-date agent refuses every session in
clear: the call settles with the agent's `err`, whose code the stored
answer keeps as it came (`result.code`: `e2e_required`, or
`e2e_broken` when the channel broke) under the action code
(`forbidden` / `failed`) — the command line reads that code.

The recording of a sealed session is made **by the node**: the request
says `record: true` when the bucket records shells, the agent writes
the asciicast from what it seals, uploads it with `putCallArtifact`
(cap `RECORDING_MAX_BYTES`, not an action's) and names it in `exited
{recording}`; the backend claims it as the call's artifact, and the
`ended` line names that artifact. An agent that died mid-session sent
none: `ended` says `recording: lost`. Audit: `open_requested` and the
agent's `open` carry `e2e` and `cli_key` (the caller's key
fingerprint — which machine opened the session, not just which token).

| A compromised backend can… | in clear (`--no-e2e`) | end to end |
|---|---|---|
| read a session live (keys, screen, `cp` files, `-L` bytes) | yes | **no** |
| inject keystrokes, change a file in flight | yes | **no** (the session ends) |
| pose as the node | yes | **no**, except at the first contact (TOFU) |
| read the stored recording and audit lines | yes | yes (by choice) |
| see metadata, sizes, timing; refuse to relay | yes | yes |
| **open** a session with the fleet token it holds | yes | **yes** |

What "readable" covers: the recording is what the node wrote on the
screen leg. A copy (`3pm fleet cp`, up or down) is never recorded —
a download's screen is the file — by the backend nor the agent: both
recognise its fixed scripts byte for byte
(`threepm_link::cp::is_cp_transfer`: `sh -c <script> 3pm-cp <paths…>`),
and an argv one byte off is an ordinary command, recorded. Nothing is
hidden by it: the scripts only move bytes between the legs and the
named paths, and the audit keeps the command line, paths included.
`-L` is never recorded either.

The last line is the known limit: the node does not authenticate the
caller's key (`cli_key` is audit metadata), so a backend that holds a
fleet token can still open a session of its own. Out of scope.


### Who may open a session: the node, not this backend

Holding the bucket's account, its token, or a node's bearer is not
enough to open a session any more. Every `nodeShell`, `nodeForward` and
`nodeListen` call is sealed end to end, and the node's agent admits it
only when the caller's key — proven in the handshake — is in the node's
own `authorized_keys`, and only for exactly the request the caller
placed (the handshake binds argv, `--user`, `host:port`, the `-R`
connection, `bind:port`, `term`/size and `record`). This backend places
and relays calls; it cannot forge a caller key, and a request it
changes on the way gets no session (the handshake breaks, nothing runs,
nothing is dialed). A refused key settles the call `replied` with
`err.code` `forbidden` and `result.code` `key_not_authorized`, the
message naming the refused fingerprint; the `ended` audit line carries
it. `record` in `nodeShell`'s body (default `true`) is the caller's: it only
lowers `record_shells`, and the value in the answer is the one the
handshake binds. Proven against the real agent by
`tests/it/real_agent_access.rs` and `real_agent_exec.rs`.

## Blobs

**What you see.** A call's bytes are relayed, not stored. `PUT
/api/buckets/{id}/blobs/{blob}` writes one once; `GET` reads it as it
arrives — the headers come back before the first byte, so the receiving
end knows it is connected before the sender has begun. A file keeps 2
MiB in memory and touches the disk only while a reader is further behind
than that, and only for as long as it is; a stream never touches the
disk at all, and a reader that falls behind jumps to the head. The `PUT`
stops while the reader does not read (`--blob-window-mb`), `Range` works
once the blob is complete, and `?k=` — sixty seconds, one blob, read
only — is what lets an `<img>` tag read one. A node's share of its
bucket's blob quota is **half** of it: past that its own `putBlob` and
`nodeAction` read `429 ERR_QUOTA` while its neighbours keep their room,
and there is no flag for it. Bounds: `--blob-max-mb` per
blob, `--blob-quota-mb` per bucket and **half of it per node**, so one
node's transfers never leave its neighbours nothing, `--blob-disk-mb`
for the whole backend, and sixty-four live `?k=` tokens per bucket.
Reading a blob is behind the cross-site guard like every other read of
what a call produced: a page on another site cannot spend the
operator's session on it, and a `?k=` — which had to be minted and put
in the URL — goes through from anywhere, which is what a browser tag
needs. A blob the relay gave up on ends its readers' responses **in
error** rather than cleanly, so a viewer can tell a cut stream from a
finished one.

## Streams

**What you see.** `nodeAction` with `stream: true` opens a **session**
rather than a call: two `live` blobs, one each way, that carry framed
messages instead of bytes with a length. A message is at most a
megabyte, carries a sequence number and a timestamp, and says whether a
decoder can start on it (`KEY`), whether it is the decoder's parameters
(`INIT`) and whether it is the last (`END`).

The relay never slows a camera down. It keeps the last
`--live-window-ms` of the stream in memory and nothing on disk; when it
has to forget something it forgets whole messages, and a viewer that has
fallen out of the window is sent on to the newest `KEY` rather than
handed a hole. A viewer that joins in the middle gets the stream's
`INIT` in front of it, so H.264 is joinable at all, and no reader is
ever handed half a message. Sixteen viewers may read one leg at once.

While a session runs, the call carries `live` — `bytes_in`,
`bytes_out`, `dropped_out` and `since`, counted in memory, on `getCall`
and on the feed's `call` events. `dropped_out` is what a viewer that
cannot keep up costs and nothing else: what the window forgets before
anyone has connected is not a dropped frame, and is not counted as one.

**What one stream carries over a distance.** A `live` leg crosses this
server between two agents, so a frame makes two TCP hops. Measured under
`tc netem delay 30ms` — 60 ms of round trip per hop, an ordinary distance
between two continents — one stream carries **5 MB/s at the latency
floor**: a hundred 50 KiB images a second arrive with a p50 age of
**61.0 ms** and a p90 of 61.3, against a floor of 60, and ten of 1 485
are lost. Thirty a second: p50 61.2, p90 61.5, **nothing lost**. The tail
is p99 92 ms at a hundred and 80 at thirty, which is the same tail the
link had before and is not the buffer's.

Above that, what the kernel's send buffer allows is **up to about
43 MB/s where the path permits it** — the buffer Linux sizes for that
round trip (2 626 560 bytes here) divided by it, read off the socket and
never measured end to end; the highest measured is the 5 MB/s above,
which is all the camera offered. A real intercontinental path, with loss,
keeps CUBIC well under buffer ÷ round trip.

Until this lot, 3pm pinned that buffer at 256 KiB to keep a frame's
latency down, and the same stream capped at **4 MB/s**: the same hundred
images a second delivered 4 of an offered 5, p50 216 ms, with 18 % thrown
away by the window behind a queue that stood at 200. `TCP_NOTSENT_LOWAT`
keeps the latency now (a frame's p99 is 1.5 ms on a bare loopback,
unchanged) and nothing keeps the ceiling. On a platform without that
knob (macOS, the BSDs, Windows) the buffer stands, and so does the
4 MB/s.

**What it costs the kernel.** Not resident memory — `sk_wmem_queued`,
which counts against `net.ipv4.tcp_mem`. Auto-tuning grows a socket's
send buffer to roughly twice the bandwidth-delay product, so a 5 MB/s
stream at 60 ms settles near 600 KiB and only a much faster path
approaches the `tcp_wmem` ceiling (4 MiB on a stock Linux). This server
tunes **every** accepted socket, so the count is three per linked node
(one link session and the agent's two warm relay connections) plus one
per dashboard, SSE or API client. Worst case at the ceiling that is
12 MiB per agent, about 1.2 GiB for a hundred nodes; `tcp_mem` is what
stops it, by putting the whole stack under pressure rather than any one
socket. Size for the real figure, not the ceiling, and watch `tcp_mem`
rather than the process.

A session's leg is charged to its bucket's quota for what it holds — a
256 KiB floor plus the window, the `INIT` kept outside it and the
message being read in, in 256 KiB steps — not ten megabytes up front;
a bucket with no room left forgets the window's oldest messages first,
then refuses the write (429). A call that fails (an error, or no
outcome that went well) gives both legs back at once, short of a
finished answer its caller is still reading. It ends when either leg has been quiet
for a minute (a leg that has not started is excused while the other one
is alive: a handler loading a model for thirty seconds has not gone
away), or after twenty-four hours, whichever comes first. Once both
legs are over and nobody is still reading them, the session's blobs are
forgotten on the sweeper's next tick: their reservation is given back at
once (a `file` blob, an answer somebody may still fetch, keeps its 15
minutes; neither holds one of the node's open slots once settled), and the call's
`live` counters go with them.

A browser watches one with two requests and no JavaScript: `viewBlob`
mints a sixty-second URL for the leg, and `getBlob?as=mjpeg` serves it
as `multipart/x-mixed-replace` — one `image/jpeg` part per message,
with its length, behind a fresh sixteen-byte boundary, `nosniff` and
`Content-Security-Policy: sandbox; default-src 'none'`. A message that
is not a whole JPEG is dropped without the connection going with it, and
the type on every part is the relay's, never the application's. The
dashboard's call card shows the counters and puts that URL in an `<img>`.

**Where it comes from.** A blob exists because a call opened it, never
because a request asked for one, and membership of that call is the
whole of the authorisation: the side that placed the call writes the
input and reads the output, the side that runs it does the opposite, the
account that owns the bucket may do either, and everything else reads as
a blob that is not there. No second bearer, no signed ticket, nothing
with a lifetime of its own.

### Behind nginx

A relay proxied by nginx needs its buffering turned off, in both
directions, or a `PUT` of a gigabyte is held in a temporary file and a
`GET` arrives in bursts. The whole backend goes behind **one**
`location /`: the agent's link is a WebSocket under `/api/v1/`, the
blobs are under `/api/buckets/`, and the dashboard's feed is
`/api/buckets/{id}/events` — splitting them is what leaves one of the
three on nginx's defaults.

```nginx
# Outside `server`, once per configuration.
map $http_upgrade $connection_upgrade { default upgrade; "" ""; }

location / {
  proxy_pass http://127.0.0.1:9616; proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;
  proxy_set_header Host $http_host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_request_buffering off; proxy_buffering off;
  proxy_max_temp_file_size 0; client_max_body_size 0;
  client_body_timeout 1h; proxy_read_timeout 1h; proxy_send_timeout 1h;
  gzip on; gzip_types text/event-stream; gzip_proxied any;
  gzip_comp_level 5; gzip_min_length 0; gzip_vary on;
  access_log off;
}

# `server` context, not `location`: nginx refuses to start if this sits
# inside the block above. Four kilobytes is for the first byte out.
ssl_buffer_size 4k;
```

Four of those lines are not negotiable.

**`gzip` on `text/event-stream` only.** The feed is JSON repeated
every second — seven times smaller gzipped — and with
`proxy_buffering off` every event the backend writes is a flush, so
each still leaves at once (the window kept across events). Nothing
else is added to `gzip_types`: the JSON routes are gzipped by the
backend itself (nginx leaves a response with a `Content-Encoding`
alone), and a blob or an MJPEG stream must go out byte for byte.

**`Upgrade` / `Connection`.** `/api/v1/link` is a WebSocket, and it is
under the same `location` as everything else. The `proxy_set_header
Connection "";` that an ordinary upstream wants strips its
`Connection: Upgrade`, axum's upgrade extractor refuses the request,
and **no agent links at all**. The `map` above
sends `Connection: upgrade` on a request that asked to upgrade and
nothing on one that did not, which is both answers at once.

**`proxy_http_version 1.1`.** Without it nginx buffers every chunked
`PUT` whole, whatever `proxy_request_buffering off` says, because an
HTTP/1.0 upstream has no chunked encoding to forward.

**`client_body_timeout 1h`.** This is the timer the relay's back
pressure runs into, not `proxy_send_timeout`. With
`proxy_request_buffering off` nginx reads the client only as fast as the
upstream takes it, and the timer armed on the **client** side while the
upstream is not taking any is `client_body_timeout` (60 s by default).
The relay stops reading a `PUT` while the reader is more than
`--blob-window-mb` behind; it resumes the moment that reader comes back
within a chunk of the window, so an ordinary slow reader never holds it
for a minute. The one long stop is a blob **nobody has attached to
yet** — the writer runs one window ahead and waits — which is exactly
where the default would cut the upload. `proxy_send_timeout 1h` is
harmless and worth keeping for the `GET` side of a stream nobody feeds
for a while.

**`Host` and `X-Forwarded-For`.** `--trust-proxy` reads what nginx puts
there and nothing else: without these two the backend logs and rate
limits every node under the proxy's own address. `$http_host` and not
`$host`: the cross-site guard compares the authority of an `Origin`
against the `Host` it received, and `$host` is normalised **without the
port** — on a vhost anywhere but 443 that turns `Origin: https://h:8443`
against `Host: h` into a 403 on every guarded request. On 443 the two
agree, which is what makes it a trap rather than an outage.

A note on HTTP/2 **from the client**: the backend is HTTP/1.1 only, so
nginx always talks h1 upstream, but `proxy_request_buffering off` on an
h2 request from the client was ignored by nginx's v2 module for a long
time — the body went to a temporary file first, which is the failure
this block exists to prevent. It was fixed in **nginx 1.25.1**; below
that, a browser `fetch` PUT of a gigabyte is buffered whatever this
block says, and the agents' relays are h1 and never take that path. Ask
`nginx -v` before believing a measurement. `proxy_max_temp_file_size 0;`
does **not** guard against it — it governs only the buffering of an
upstream *response* to disk, and is already moot under `proxy_buffering
off` — but it is free, and it keeps a large `GET` out of the temporary
directory if `proxy_buffering` is ever turned back on.

**A shell needs the whole block.** `3pm fleet ssh` is two requests
that last as long as the session: a `PUT` that sends a keystroke now
and then and a `GET` that may say nothing for a while. Without
`proxy_http_version 1.1` and `proxy_request_buffering off` the
keystrokes never reach the node (nginx waits for the end of the body);
with `proxy_buffering on` the screen arrives in bursts. The command
line's twenty-second pings (and the agent's pongs) keep a quiet shell
inside nginx's default minute (`client_body_timeout`,
`proxy_read_timeout`, `proxy_send_timeout`), with one missed ping of
margin and no more: keep the block's hour, or a stalled ping cuts the
session (the relay then ends both legs). Check with `nginx -T` before
opening the route on a proxied backend.

`access_log off` is not optional either: a `viewBlob` token travels in
the URL (`?k=`), and an access log is a file of live credentials.

One node relaying a wall of cameras holds a socket per stream in each
direction, so give the host the file descriptors and the sockets to
match: `ulimit -n 65535` for the backend's unit (`LimitNOFILE=65535`),
and `sysctl -w net.core.somaxconn=1024`.

## The JavaScript client, generated from the route table

**What you see.** The backend serves the client for its own API:

```js
import { Client } from "/api/v1/client.js";

const api = new Client();          // same origin
const buckets = await api.listBuckets();
const feed = api.events(buckets[0].cluster_id, {
  init: (detail) => render(detail),
  node: (n) => upsert(n),
  telemetry: (t) => plot(t),
});
feed.close();
```

One ES module, no dependency, no build step: a method per endpoint, a
`Subscription` that reconnects on its own (and turns a `reset` into a
fresh `init`), and `ApiError {status, code, detail}` carrying the
backend's own `ERR_*`. `fetch` and `EventSource` are taken from the
global scope, or passed in. `docs/examples/fleet.mjs` is a whole
interface — print the fleet, then follow it — in fifty lines.

**This client targets browsers.** Nothing in it touches the DOM when
the module is imported, but `connect()` below draws, and the feed
prefers the browser's own `EventSource`. A Node program uses the SDK's
fleet client (`require("@3pm/api/fleet")`), generated from the same
route table.

Live data is the SSE feed, two ways: handlers (`api.events`) or a loop
— `for await (const [kind, payload] of api.follow(clusterId))` closes
the feed when you leave it (`docs/examples/follow.mjs`). The two routes
the API also lists for the agent, `enroll` (a `POST`) and `link` (the
WebSocket), are its protocol (`3pm link`), not a feed: `api.enroll()`
and `api.link()` throw an `AgentOnlyError` that says so and names
`events`, and the `LINK` constant carries the sub-protocol
(`3pm.link`) and where the protocol is written down.

`GET /api/v1/routes` is the API describing itself: every route with its
method, path, argument names, kind (`json`, `sse`, `agent-ws`,
`agent-http`, `static`), who may call it (`auth`: `public` or `user`, an account;
`node`: the scope a node's bearer needs, when one may), whether the
cross-site guard stands in front of it (`guarded`), its body and response
types, and a line of documentation — plus a `realtime` block that says
how live data is had (the SSE feed and its event names, the agent's
WebSockets and their sub-protocol, each pointing at its route by name),
where the client lives and a runnable snippet.

A route whose body is a small language of its own — `queryMetrics`,
`eventsQuery` — carries more than that line: `details` spells out every
field, what comes back, and a runnable example body, and the `limits`
block gives the numeric bounds by route name (range, points, series,
queries, formulas, filters, value lengths, body size). Both are
generated from the code that enforces them — the limits are the
validator's own constants, and a test parses each example through the
very validation the handler applies — so reading the descriptor is
enough to write a valid query, without discovering the bounds one 400
at a time. The one bound not there is the step floor, which depends on
how the backend was started: `store.interval_ms` of `GET /api/v1/info`,
and the block says so.

The document opens with its own price, `tokens: { claude, o200k }` —
`o200k` exact (OpenAI's `o200k_base`), `claude` estimated from the
bytes with a ratio measured per shape through Anthropic's
`count_tokens` API — counted over the exact bytes the route sends and
put first on purpose: a consumer with a context budget reads what the
document costs before reading the document. For an LLM the route
answers text by default: an index, one TOON-style row per route it can
use, recipes (a body for each common question — the value now, a
total over a period, the top N, a percentage, a rate, one host by
name, one label value, the error events — and how to read its answer),
then `metricsInventory`, `queryMetrics` and `eventsQuery` given whole
with their types — what a dashboard starts from (10 741 Claude tokens,
against 8 497 for the JSON table) — and
`?route=<name>` for one route in full — its details, its bounds, and
the TypeScript of its body and answer with every type they name. The
JSON table is `?format=json`, or what an `Accept: application/json`
gets, which is what the generated client sends. `?version=` defaults to `latest`; a backend serves
its own table or refuses, never a guess. `client.discover()` reads it
and follows that backend's paths from then on, so a page built against
one version keeps working against another — and says so when the
versions differ (`onVersionMismatch`, a console warning by default;
the client's own `VERSION` is exported). `client.call(name, args,
body, options)` drives any route by name, and `options.signal` on it
or on any generated method abandons a request a page no longer needs.

### Asking the viewer for a token

**What you see.** A page of your own, served from anywhere, that wants
to read a backend it has no session on:

```js
import { connect } from "https://fleet.example.com/api/v1/client.js";

const api = await connect({ baseUrl: "https://fleet.example.com" });
const buckets = await api.listBuckets();      // ready to call
```

`connect()` asks only when it has to. With a token already remembered
for this page it checks it against the backend and goes straight
through; otherwise a small card asks for one — "Log in", an
`3pmu_…` field, the same wording as the dashboard's own login. Enter
submits, Escape (or a click outside) dismisses and rejects with a
`TokenPromptCancelled` error.

The card lives in a **shadow root** of its own, so the page's CSS
cannot deform it and its own cannot leak out, and it follows the
viewer's light/dark preference.

**Nothing is stored until the backend has accepted it.** A token that
does not start with `3pmu_` is refused on the spot, with a line saying
so rather than a flat 401 later; anything else is spent on one
`listBuckets` call — the cheapest authenticated read, and one the route
table leaves unguarded, so it answers from another origin. Only a token
that came back 200 is written to `localStorage`, under
`3pm.token:<location.pathname>` (`connect({ namespace })` overrides the
key — one for a whole app, or one per environment). Every read and
write is wrapped: in a private window or with site data blocked,
nothing is stored and the viewer is simply asked again.

Revoked or rotated under the page? The 401 clears the stored token,
reopens the card, and — once a new one is accepted — **retries the call
that met the 401**, so the page recovers instead of freezing.
`api.forgetToken()` clears it on purpose.

**Where it comes from.** `connect()`, `promptForToken()`, `tokenKey()`
and `Client.forgetToken()` in `src/api/client.tmpl.js`, generated into
`assets/api.js` by `just api` like the rest. `new Client(…)` is
unchanged: the prompt is opt-in and nothing draws until it is called.

### From another origin: CORS is open

**What you see.** A page served from anywhere can call this backend —
there is no proxy to put in front of it any more. Every origin, every
method, and the request headers a preflight asks for, so `Authorization`
really is allowed (`Access-Control-Allow-Headers: *` would *not* have
covered it: the Fetch spec excludes it from the wildcard on purpose).
A preflight is answered for ten minutes, so a burst of calls does not
preflight twice per call. Response headers are exposed with `*`, so a
cross-origin reader sees more than the six safelisted ones.

```console
$ curl -si -X OPTIONS https://fleet.example.com/api/buckets/$B/metrics/catalog \
    -H 'Origin: https://somewhere.example' \
    -H 'Access-Control-Request-Method: GET' \
    -H 'Access-Control-Request-Headers: authorization, content-type'
HTTP/1.1 200 OK
vary: access-control-request-headers
access-control-allow-origin: *
access-control-allow-methods: *
access-control-allow-headers: authorization, content-type
access-control-max-age: 600
```

**Credentials do not travel, and that is the point.** There is no
`Access-Control-Allow-Credentials`, because a wildcard origin forbids
one — and the session cookie is `SameSite=Strict; HttpOnly` anyway, so
it never leaves this origin whatever we answer. A cross-origin caller
authenticates with a token it puts on the request on purpose:
`new Client({ baseUrl, token })`, or `Authorization: Bearer 3pmu_…`.

**What CORS does not open is the cross-site guard — and what opens the
guard is a bearer.** The guard is about a page elsewhere driving an
operator's session, a credential the browser attaches by itself. So a
request carrying another origin is still answered **403
`ERR_FORBIDDEN`** when it rides the session cookie or `Basic`, and on
`register` / `login` whatever it carries. A request that carries
`Authorization: Bearer 3pmu_…` is the opposite — the page holds the
token and put it there on purpose — and passes from any origin on every
route behind an account: a dashboard served elsewhere can run a custom
action (`nodeAction`), follow it (`getCall`), read its artifacts, and
make every other write its token allows; a bad token is a 401. The 403,
when it comes, is *readable*: it carries the CORS header, so the page
sees the status and the `ERR_FORBIDDEN` body instead of an opaque
network error it cannot tell from the backend being down.

**Where it comes from.** `src/api/surface/` declares the surface once
as data: `mod.rs` says what an entry is (`Endpoint`), `table.rs`
assembles the table from one file per resource under `table/`
(`meta`, `buckets`, `metrics`, `calls`, `blobs`, `pages`). The axum router mounts exactly that table, and `just api`
generates `assets/api.js`, `assets/api.d.ts` and `assets/api.json` from
it — `just check` fails when what is committed no longer matches.
`tests/it/api_surface.rs` holds the table to the running server: every
entry mounted on the method it declares, every guarded entry really
guarded, `/api/v1/routes` equal to the generated file, and every field
the API serializes declared in the types. Adding an endpoint is one
entry, one handler arm, and `just api`. There is deliberately no
OpenAPI: half the surface (the SSE feed, the two agent WebSockets) is
not describable in it.

## Accounts: keeping the API to yourself

**What you see.** Every bucket belongs to an account, and everything
under `/api/buckets` wants one. An account is a name and its tokens —
no password, no email. `/` without a session shows a login card: paste
a token, or create an account, whose token is shown once with a Copy
button (it is the only way back in). Logged in, the header shows the
name and **Log out**, and `/` lists the account's tokens under
"Account tokens": one per laptop or CI job (the secret shown once),
each revocable — the sessions it opened close with it, and the last one
stays (409 `ERR_LAST_TOKEN`).

- **Registration.** `--registration closed` (the default) lets the
  first account in and no other (403 `ERR_REGISTRATION_CLOSED`); that
  first account adopts every bucket created before accounts existed.
  `--registration open`, or `--dev`, lets anyone who reaches the port
  register.
- **A token** reads `3pmu_<id>_<secret>`: only its SHA-256 is stored
  (compared in constant time), the id is what lists and logs show. A
  script sends it on every request — `Authorization: Bearer 3pmu_…`, or
  `curl -u :3pmu_…` (Basic, the token as the password) — and the
  generated client takes it as `token`.
- **The dashboard** trades it for a session (`login`): a `3pm_session`
  cookie, `HttpOnly`, `SameSite=Strict`, 30 days sliding, `Secure` with
  `--cookie-secure` (set it behind HTTPS). The feed (`EventSource`)
  runs on it. Log out closes the session, not the token.
- **Another account's bucket** answers 404, like one that does not
  exist, so ids cannot be probed; no credential answers 401
  `ERR_UNAUTHORIZED` with the challenge. The pages, `/api/v1/info`,
  `/api/v1/routes`, `register`, `login` and the agent's own routes stay
  open — an agent authenticates with its node key, over the enrollment
  token at `enroll` and over the server's nonce at `link`.
- **Cross-site refusal, always on**, in front of every mutating route,
  `login` included. A request whose `Origin` (or `Sec-Fetch-Site`) names
  another site than its `Host` is refused 403 `ERR_FORBIDDEN` before
  anything else, so a page elsewhere cannot drive a logged-in browser.
  Only a credential the page had to hold opens it: a minted `?k=` on
  `getBlob`, an `Authorization: Bearer` on a route behind an account.

Beyond that: enrollments are budgeted per client address
(`--enroll-pending-per-ip`, then 429 `ERR_RATE_LIMITED`), and so are
link handshakes until they are over (`--link-pending-per-ip`, 64 by
default, judged before the database is read; then 503 `ERR_LINK_BUSY`,
also the answer when the handshake nonce store is full) — an IPv6
client counted by its /64. 503 and not 429: a busy backend is a retry
within the agent's 1–60 s transient backoff, never its 15-minute
rate-limit park, so a flood of upgrades without a key no longer holds
the honest agents away. A /48 can still fill the shared nonce store;
the agents then retry within a minute. Slow header
senders are dropped (`--header-timeout-ms`), agent frames are capped at
1 MiB before inflation, and a silent link is closed (`--silence-ms`).

**Where it comes from.** The table's `auth` column (`public` / `user`)
decides which routes sit behind the account layer. `auth/`: the token,
the cookie and the layer (`require_user`: who is calling, and whether a
route's `{id}` is a bucket that account owns). `db/users.rs`,
`http/api_auth.rs`, `http/guard.rs` (pure `cross_site`, table-tested),
`ws/budget.rs`, `server.rs`.

## What is redacted

**What you see.** The process table the backend stores and shows never
contains environment variable *values*, script paths or home paths —
the agent strips them before sending, and an exit event carries its
code and signal but not the stderr tail. Token secrets are never
stored, and `TokenView` / `NodeView` carry no key material. Log lines
arrive already redacted: the node replaces the secrets it recognises
before anything leaves (a pattern list; 3pm's `FEATURES.md`, "Log
shipping"), and the backend stores what it receives.

**Where it comes from.** `threepm-link`'s `redact` (in 3pm), and the
views in `http/views.rs`, whose tests assert no secret survives
serialization.

</document>

<document url="https://3pm.edgecr4ft.net/docs/backend/readme.md" title="Backend README">

# 3pm-backend

The fleet backend of [3pm](https://github.com/Unitech/3pm): the server
side of `3pm link`. It enrolls hosts into buckets, admits their agents
over one WebSocket each, ingests the redacted process table and host
metrics at 1 Hz, and serves a JSON API, an SSE feed and a small
dashboard. axum 0.8 + SQLite, one binary, no other service.

```
$ just dev          # --dev on 127.0.0.1:9616, clear text, local advertise allowed
```

Open `http://127.0.0.1:9616/`, create a bucket, mint a token, and paste
the printed line on a host running 3pm:

```
$ 3pm link 3pmc_<cluster> 3pmk_<token> ws://127.0.0.1:9616
```

## Building

This repository builds next to a checkout of 3pm: the shared crates
(`threepm-link`, the wire contract; `threepm-core`, `threepm-protocol`;
`threepm-testsupport` for the real-agent test) are path dependencies on
`../3pm/crates/…`.

```
3pm-rd/
  3pm/            git clone https://github.com/Unitech/3pm
  3pm-backend/    this repository
```

| Recipe | What it runs |
|---|---|
| `just check` | `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings` |
| `just test [filter]` | unit + integration tests under `cargo nextest`; the integration tests are one binary, `tests/it`, a module per file (`just test 'test(/^auth_e2e::/)'` runs one) |
| `just e2e` | builds a debug `3pm` and its `agent` module from `../3pm`, then the `real_agent*` modules of `tests/it` |
| `just dev` | the backend in `--dev` mode on `127.0.0.1:9616` |
| `just msrv` | `cargo +1.88 check` |
| `just ci` | everything CI gates |

`THREEPM_SRC=/path/to/3pm just e2e` uses another checkout for the
binaries (the path dependencies stay on `../3pm`).

## Documentation

- [FEATURES.md](FEATURES.md): what the backend does, one section per
  user-facing feature.
- The wire contract, the shared crate and the agent module are
  documented in 3pm under `docs/backend-agent/`.

`docs/backend.md` — configuration, HTTP API and SSE feed, enrollment and
link endpoints, the hub, database, crypto, file by file, tests, caveats —
left the tree on 2026-09-09 and lives in OpenViking at
`viking://resources/3pm-backend-docs-archived/backend/`. The section names
quoted across this repository ("→ Metrics store", "→ Metrics API") are its.

</document>

<document url="https://3pm.edgecr4ft.net/docs/fleet/overview.md" title="Fleet link overview">

# Fleet link: `3pm agent` ↔ `3pm-backend`

The fleet link connects a 3pm daemon to a central backend: the daemon's
host enrolls once, then a supervised agent app streams the redacted
process table and host metrics over one WebSocket, and (later) accepts a
small whitelist of remote commands. This directory documents what
exists after phases A, B and C of the plan — the wire contract, the
shared crate, the agent module, and the backend up to ingestion, the
first dashboard and the node API. It is the reference for phase D;
`FEATURES.md` carries the user-facing summary, `CHANGELOG.md` the
history.

| Document | What it covers |
|---|---|
| [link-protocol.md](link-protocol.md) | The wire contract: identifiers, enrollment, link handshake, the two frame vocabularies and how they decode, commands and calls, events, the node's state (`state`) and what never travels, framing/keepalive/retry classes, error codes, fixtures |
| `viking://resources/3pm-docs-archived/backend-agent/threepm-link/` (OpenViking; left the tree 2026-09-09) | `crates/threepm-link`: the pure shared crate, file by file, with its public API |
| [agent.md](agent.md) | `modules/agent` (git submodule, repository `3pm-agent`): CLI, environment, state directory, the supervised process, file by file, tests, caveats |
| `viking://resources/3pm-backend-docs-archived/backend/` (OpenViking; left the 3pm-backend tree 2026-09-09) | The backend: configuration, HTTP API and SSE feed, enrollment and link endpoints, the hub, database, crypto, file by file, tests, caveats |

## Shape of the system

```
3pm daemon ── rpc.sock / pub.sock ── 3pm-agent (__run, supervised app, wait_ready)
                                          │  WebSocket, sub-protocol 3pm.link
                                          │  agent → server: Binary, raw DEFLATE (one stream per connection)
                                          │  server → agent: Text JSON
                                     3pm-backend (axum, SQLite)
                                          │  HTTP JSON API (buckets, tokens, nodes) + SSE feed
                                     dashboard (`/`, `/b/{id}`; calls next)
```

Layering is one-way: `threepm-core ← threepm-protocol ← threepm-link ←
{modules/agent, 3pm-backend}`. `threepm-link` has no I/O, no clock and no
randomness; both endpoints share its vocabulary and its table-driven
tests. The local NDJSON protocol is nested, never re-modelled: a remote
`restart` becomes exactly one local `Request::Restart`.

## Status by step

| Step | Deliverable | Commit |
|---|---|---|
| 1–3 | `crates/threepm-link` (ids, envelope, errors, records, telemetry, commands, events, certs, proofs, pins, backoff, server URLs, frozen fixtures) | `cd894e4` `b0c0344` `daec6ec` |
| 4–6 | `3pm-backend` skeleton, SQLite layer, bucket CA, tokens, nonces, `GET /api/v1/enroll`, buckets/tokens API with the pasted line | `de8a6d3` `6e08f8b` `e63c8bd` |
| 7 | `modules/agent` skeleton: CLI, inputs, options, state directory, `status`, `unlink`, registration everywhere the bundle is assembled | `68472d5` |
| 8 | Agent enrollment: identity, transport, enroll dialogue, `link` end to end against a mock backend | `1f98191` |
| 9 | Agent session: handshake, single writer/reader, reconnection schedule, polite close | `9162b10` |
| 10 | Agent telemetry: 1 Hz redacted process table + host snapshot, lifecycle events as `emit` | `9879080` |
| 11 | Backend ingestion: `GET /api/v1/link` admission, handshake machine, session task, hub (registry, presence `online → stale → offline`, rings, per-bucket event log, sweeper), SSE feed, `GET /api/buckets/{id}` = `init` | `82d9401` |
| 12 | First dashboard: embedded `assets/` (buckets, bucket page with the pasted line, nodes, per-node process table and raw `telemetry` JSON), `http/pages.rs`; `tests/it/real_agent.rs` — a real 3pm links to the backend and the feed shows its processes | `c693b5c` |

| 13 | Node API: `GET /nodes`, `POST /approve`, `POST /revoke` (cuts the live session 4403), `NodeView` complete (`cert_expired`, `last_auth_error`), approve / revoke buttons on the bucket page | `21f8722` |
| 16 | `3pm link` / `3pm unlink` built-in sugar over the `agent` module (arguments passed through, the agent's help and exit code, `--json` via `THREEPM_JSON`, reinstall hint without the module); the pasted line runs as printed; `tests/real_agent.rs` goes through it | `eeb7c56` |
| review | Adversarial review of phases A–C (security, protocol, concurrency, tests) and its fixes: backend silence timer, close drain, fresh view after welcome + post-attach re-check, shutdown that cannot hang, own enrollment nonce store + per-IP budget, one-transaction enrollment, 1 MiB caps before inflation, header timeout, asset path guard, cross-site guard + `--admin-token`; link mirror enums with `Unknown`, frame budget, handshake tolerance, bare 4403; agent shutdown during connect, bus backoff, bounded ping, enrollment skew check; the missing tests on every side | `92654ce` `fad300c` `fd3505b` |

Phase C is complete and, from phase D, the `3pm link` / `3pm unlink`
sugar (step 16, pulled forward). Not built yet: remote commands (14),
TLS on the
backend (17), the final docs pass (18). Since then the backend moved to its own
repository (`3pm-backend`, next to this one); its `just e2e` runs the
real-agent test against a debug 3pm built from `../3pm`. The backend
also meters what each node sends (`NodeView.rx_bps` / `rx_bytes`,
counted in payload bytes before inflation, the rate averaged over the
last 5 complete seconds and shown as `Rx/s` on the bucket page); what a
node weighs on the link (≈ 350–460 B/s, everything included) is in
[link-protocol.md § 7 Volume](link-protocol.md#volume). A linked host now
also sees its link above the process table (`3pm ls` opens with `⇆ 3pm
link | node … | dash: …`, red with the reason when it is down), and the
bucket page's address is printed wherever the link is — after `3pm link`,
in `3pm agent status`, in that line.

Since link version 2 everything a node says about its state — the
host, the process table, each instance's application metrics and
process tree — travels as **one stream**: a `state` frame a tick
carrying the sections that moved (`host`, `procs`, `app/<id>`,
`tree/<id>`, each with its own revision), cut as JSON merge patches by
the agent against what the backend holds, rebuilt by one reader;
`state_resync {secs}` asks sections again in full. The `welcome` paces
it with one `pace {state_s, metrics_s, trees}` object. The contract is
[link-protocol.md § 7](link-protocol.md#7-the-nodes-state-state); the
agent side is the one task of [agent.md](agent.md#the-supervised-app)
(`node_state.rs`, `node_state/paces.rs`, `stream.rs`). The `telemetry`,
`host_patch`, `app_metrics` and `trees` streams of link 1 — with
`resync`, `host_resync`, `trees_request`, `welcome.full_every_s` and
`host_patch_every_s` — are gone, and a peer that does not speak the
current version is refused at the handshake (`ERR_LINK_VERSION`).

Link version 3 dropped the node certificate and the bucket CA: a node is
its row on the backend — id, public key, state, re-read at every
handshake — and enrollment became one signed `POST /api/v1/enroll`
instead of a three-frame WebSocket dialogue. The link upgrade names the
node in the clear (`Authorization: 3pm-node <cluster-id>:<node-id>`) and
proves it with the same nonce and signature as before
([link-protocol.md §2–§3](link-protocol.md)).

Link version 4 gave each direction its own frame enum (`ServerFrame`,
`AgentFrame`), cut the handshake to three frames (the proof moved into
`hello.sig`), typed every identifier the two sides exchange (`BlobId`,
`CallId`, `Sha256`, and the link's own `LinkInput` / `OutputBlob` in
place of the local wire's `ActionInput` / `OutputRef`), folded the three
pace knobs of `welcome` into one `pace` object, and made decoding two
steps: a tag this build does not know is skipped, a body it cannot read
is **counted and answered**, and neither ends a session. The
sub-protocol lost its number (`3pm.link`).

Link version 5 left the link with **one** way to move bytes. The
`blob_chunk` frame is gone: the file an action returns is uploaded by
the agent to the backend's artifact store over HTTP (`PUT
/api/buckets/{cluster}/calls/{call}/artifacts`) before the reply, and
the reply names it (`Outcome.file = {artifact_id, bytes, ext, sha256}`,
fetched with `artifactFile` by either end). A `data` too big for the
reply's inline budget is now an error telling the author to return a
file instead of silently becoming bytes to fetch. With no frame only an
agent could send, the two outcome types collapse into one `Outcome`
carried by both `reply` and `call_result`
([link-protocol.md §5](link-protocol.md#5-commands-and-calls)).

## Security model in one paragraph

Each host has an Ed25519 **node key**, generated at `link`, stored
owner-only in `<home>/agent/node.key`, never leaving the host. A node is
its row in the backend's database — cluster, id, public key, state — and
nothing else: no certificate, no CA, nothing to renew, and the row is
re-read at every handshake. Enrollment is authorised by a one-time
**token** `3pmk_<id>.<secret>`; the backend stores its SHA-256 only and
compares in constant time. A single-use token valid for at most one hour
yields an *active* node; anything reusable or longer-lived yields a
*pending* node that a human approves. Enrollment itself is one signed
`POST /api/v1/enroll`; every connection afterwards proves possession of
the node key against a server nonce (domain-separated messages for the
two). Wire growth is additive only: unknown message tags, error codes and
enum values decode to `Unknown` and are skipped.

## Caveats carried over from the plan

- Link handshake nonces live in memory: one backend instance.
- Environment variable **names** travel by default (`--no-env-keys`
  turns them off); values never do.
- Windows: the agent talks to the daemon over `UnixStream` like `serve`;
  the 0600 mode on `node.key` is Unix-only (temp + rename still holds).
- The backend's mutating endpoints have no authentication yet (an
  `--admin-token` is an open decision).

</document>

<document url="https://3pm.edgecr4ft.net/docs/fleet/agent.md" title="The agent module">

# `modules/agent` — the `3pm agent` module

The second bundled module (`modules/agent/`, its own workspace and
lockfile, bin `3pm-agent`, `module.toml` name `agent`). Two roles in one
binary: the CLI 3pm dispatches to (`3pm agent link|unlink|status`; `3pm
link` / `3pm unlink` are built-in sugar over the first two, arguments
passed through untouched), and
`__run`, the supervised app that holds the link. Lib + bin: the binary,
the unit tests and the test backend share one implementation.

## Usage

```
$ 3pm link 3pmc_<cluster> 3pmk_<token> wss://fleet.example.com [flags]   # = 3pm agent link …
$ 3pm agent link 3pmc_<cluster> 3pmk_<token> wss://fleet.example.com [flags]
$ 3pm link 3pmc_<cluster> 3pmk_<token>   # no server: wss://fleet.edgecr4ft.net (DEFAULT_SERVER)
$ 3pm link [--json]               # the link; exit 0 only when established (= 3pm agent link, alone)
$ 3pm link stop [--json]          # pause: stop the app, keep the enrollment (= 3pm agent link stop)
$ 3pm link start [--json]         # resume; registers the app again if it was deleted
$ 3pm agent status [--json]       # also `3pm link status`
$ 3pm link delete [--json]        # = 3pm unlink (also `rm`, `remove`)
$ 3pm agent unlink [--json]
```

`link` flags: `--insecure` (accept `ws://` to a non-loopback host; over
`wss://` it is refused until the TLS hardening step), `--name` (app
name, default `agent`), `--cap` (repeatable, comma-friendly:
`metrics:read`, `proc:restart`, `proc:stop`, `proc:action`; default
all — a `link.toml` written before `proc:action` existed lists the
three of its day and is read as all), `--no-env-keys`
(do not ship environment variable names), `--force` (enroll again over an
existing link: fresh key, hence a new node id; the app is restarted in
place), `--json`.

`3pm link` alone — no positional and no `THREEPM_LINK_*` in the
environment — is a check: the `status` view plus a `link` line,
`connected for 3m 2s (session 2, server 0.1.0)` or `NOT established —
<why>`, and exit 0 only when the link is up (1 with `link not
established: <why>` on stderr: not linked, app stopped / not registered,
daemon unreachable, no session reported yet, `link down for … (<last
disconnect>), retrying`, `parked: the node was revoked`). The running
agent writes `session.json` in the state directory on every change of
its session (pid, connected, since, sessions, retries, parked, last
disconnect, server version) and removes it on a clean exit; readers
trust it only when its pid is the app's current one.

The same two files feed the line `3pm ls` prints above its table on a
linked host — `⇆ 3pm link | node 3pmn_… | dash: <bucket page>`, green,
or red with the reason (`crates/threepm-cli/src/link_state.rs` +
`render/linkline.rs`; the verdict itself is
`threepm_link::local::status`, shared with `status_cmd`). The CLI reads
the files directly: no module is spawned for a table. The bucket page —
`http(s)://<server>/b/<cluster-id>`, `ServerUrl::bucket_url` — is also
the `dashboard` line of `3pm agent status` and the last line of a
successful `3pm link` (`"dashboard"` in `--json`).

`link stop` stops the supervised app and saves the process list, so
the pause survives a reboot; the key and `link.toml` stay.
`link start` restarts the app or, when it was deleted by hand,
registers it again from `link.toml` (same spec as `link`). Both refuse
a host that is not linked, before the daemon is asked anything.

The three positionals can come from the environment, which **wins**
(a provisioning script over a pasted line): `THREEPM_LINK_CLUSTER`,
`THREEPM_LINK_TOKEN`, `THREEPM_LINK_SERVER`, `THREEPM_LINK_INSECURE`.
Empty variables do not shadow the positionals. The server is optional:
given nowhere (a blank value counts as none), it is
`inputs::DEFAULT_SERVER`, `wss://fleet.edgecr4ft.net`. A missing input names its
position and its variable; a malformed token is reported **without
echoing it**; clear text to a host that is not loopback is refused with
`ERR_INSECURE_SERVER` — the agent's own code string, not one of the
link's: the wire has no such failure to name.

`--json` contracts (also `THREEPM_JSON=1`, which 3pm's dispatcher sets
when it sees the module's `--json`):

| Command | Document |
|---|---|
| `link` | `{node_id, cluster_id, state: active\|pending, affected: [ProcessRecord]}` |
| `status`, bare `link` | `{linked, app, cluster_id?, node_id?, server?, process?: RedactedRecord, session?: {connected, for_ms, sessions, retries, parked, frames_invalid, last_disconnect?, server_version?}, daemon_error?}` — `session` only when the app runs and has reported |
| `unlink`, `link delete` | `{affected: [ProcessRecord], removed: bool}` |
| `link stop` | `{stopped: true, app, affected: [ProcessRecord]}` |
| `link start` | `{started: true, app, registered: bool, affected: [ProcessRecord]}` — `registered` when the app had to be created again |

Exit code 1 with `[3PM][ERROR] …` on stderr for every refusal
(`ERR_TOKEN_EXHAUSTED`, `ERR_TOKEN_INVALID`, a dead port, "already
linked — pass --force"…), and **nothing is written** before the backend
said yes.

## The supervised app

`link` starts this binary as `<name>` with `args = ["__run"]`, `cwd` =
the state directory, `wait_ready = true`, tagged `spec.module = {agent,
version}` (so it lists under **Modules** and `stop|restart|delete all`
leave it alone), then `Save`s the process list. Its environment is only
the `THREEPM_AGENT_*` options below — identity and server live in
`link.toml`, never in env (`3pm show` prints env).

| Variable | Default | Meaning |
|---|---|---|
| `THREEPM_AGENT_STATE_DIR` | `<home>/agent` | Holds `node.key`, `link.toml`, and `session.json` while the agent runs |
| `THREEPM_AGENT_NAME` | `agent` | App name (what `unlink` deletes) |
| `THREEPM_AGENT_LOG` | `info` | `tracing` filter for the `__run` role (stdout, so it lands in the app's out log). Read by the app, **not** injected by `link` — set it with `3pm env`/`--env` on the `agent` app if needed |

The pace of the link is the backend's (`welcome.pace`), not an option: the
`THREEPM_AGENT_INTERVAL_MS` / `THREEPM_AGENT_FULL_EVERY_S` an older
`3pm link` left on the app are no longer read, whatever they say.

Readiness is announced on the control channel **as soon as the identity
is loaded**: ready means "the agent runs", not "the backend answers" —
otherwise a backend outage would put the app in `errored` and 3pm would
restart it in a loop.

Then the session and its three workers (`run.rs::spawn_workers`: the
node's state, the events, and `publish_session` keeping `session.json`)
until a shutdown (control-channel `shutdown`, SIGINT — 3pm's default
kill signal — or SIGTERM, or the daemon's own `daemon_shutdown` event):

- **session**: connect with the node's two ids in `Authorization` →
  handshake → writer + reader → classify the disconnect → wait the
  schedule says → again. The agent speaks link 5 only: a server whose
  challenge offers an older version alone is refused before anything is
  signed (`ERR_LINK_VERSION`, retried on the Auth schedule — link
  protocol §8). Revoked parks with an hourly probe (logged once).
  Shutdown sends `Close(Normal)` and waits 2 s for the echo.
  A call is acked by whatever accepts it, as its first act — so one the
  reader could not hand over is never acked and the backend's view says
  so. Calls run in `commands.rs`, one task each: the capability against
  what this host allows (`link.toml` `caps`), four in flight at most (`ping` excepted), the
  daemon asked with a bounded budget (`DaemonLink`, a connection per
  call), an action's result gzip inline within the reply's budget and
  the file it returned uploaded to the backend's artifact store over
  HTTP before the reply names it (link protocol §5); a call whose
  connection dropped withholds its reply (at-most-once).
- **the node's state** (`node_state.rs`): the one stream that carries
  the node — link protocol §7 is the reference, not repeated here. One
  task, one daemon connection (`DaemonLink`, each call bounded to 3 s,
  reconnection 1 → 30 s), one `NodeWriter`. Every `welcome.pace.state_s`
  seconds (1 s floor) it asks the daemon for complete states — `List`
  and `HostMetrics`; `AppMetrics` (`id` alone) for the instances
  whose `(gen, app_metrics_rev)` moved, every `pace.metrics_s`;
  `ProcessTrees` for the instances with more than one process whose
  `(gen, tree_rev)` moved, plus the figures on `pace.trees`
  (`shape` = the shape only, `off` = no tree) —, shapes each into its
  section (`threepm_link::node_state`), cuts it against what the backend
  holds and sends what moved in **one** tracked `state` frame, empty
  when nothing did: the heartbeat. A section is committed only once the
  writer confirmed the bytes on the socket (a frame not written is cut
  again next tick, larger, at the same revisions); owed in full on the
  first tick of a session, on `state_resync {secs}` and when the daemon
  was replaced (another `boot_id`); said `gone` when its instance left
  the table, its tree is back to one process, or the host agent is down
  — never while the daemon itself is down. A section alone over the
  1 MiB cap is not sent (warned once) and commits nothing; a daemon
  answering `unknown_request` to `ProcessTrees` is asked again 60 s
  later. While the daemon is down the frame still goes: an empty table,
  no daemon identity.
- **events**: subscription to the daemon bus filtered to `process`,
  `daemon_shutdown` and `app`, forwarded as `emit`; lifecycle events
  wait for room on the control channel (never dropped), `app` events
  are `try_send` (a burst never holds the channel full and starves the
  `state` frames); `daemon_shutdown` then closes the link politely;
  resubscription 1 → 30 s.

## State directory

`<home>/agent/` (0700 on Unix), written only after a successful
enrollment, in this order: `node.key` first (base64url seed, **created
0600 before any byte lands**, temp file in the same directory + fsync +
rename), then `link.toml`:

```toml
# Written by `3pm agent link`. No secret here: the node key is node.key.
version = 4
cluster_id = "3pmc_…"
node_id = "3pmn_…"
server = "wss://fleet.example.com:443"
insecure = false
caps = ["metrics:read", "proc:restart", "proc:stop", "proc:action", "node:shell", "node:forward"]
env_keys = true
app = "agent"
linked_at_ms = 1788523845154

# Never written by `3pm agent link`: the operator's, by hand.
[shell]
gateway_ports = true
```

`version` (`threepm_link::local::LINK_FILE_VERSION`, currently 4) says how
`caps` reads: since 4 the list is literal; a file with `version = 3`
listing exactly the five capabilities of its day reads as all of them
(so a default link gains `node:forward`), any other v3 list is literal;
a file with `version = 2`
(written before `node:shell` existed) has exactly the four capabilities
of its day read as all — any other v2 list is literal —, a file with
`version = 1` (written before `proc:action` existed) has its three
capabilities of the day read as all, and a file with no `version` key —
never written by any build — is read literally, like today's. So a node
linked with the defaults offers `node:shell` (a root shell where the
agent runs as root) after its update, without a re-link: take it out of
`caps` to refuse `3pm fleet ssh`. `[shell] gateway_ports`
(`threepm_link::local::ShellSettings`, off when absent) lets `3pm fleet
ssh -R` bind an address other than the loopback, as ssh's
`GatewayPorts`: the agent reads it, the backend can neither read nor
change it, and a re-link keeps it. Unknown keys are ignored,
missing ones default, unknown capabilities are dropped at parse time —
except in `[logs]`, where an unknown key (a misspelt `file = false`)
makes the section malformed: it then ships nothing
(`LogsSettings::closed`), and `LinkConfig::logs_problem` says why.
The shape of `link.toml` and of `session.json` lives in
`threepm_link::local`, not here: the 3pm CLI reads both to print
its link line above `3pm ls` (`crates/threepm-cli/src/link_state.rs`),
without spawning this module. `shell.key` (X25519, 0600) appears
beside them at the first end-to-end `3pm fleet ssh` session the node
serves: its static key for that channel, shown as `shell key SHA256:…`
by `3pm link status`; the caller's own is `cli.key` beside its
`known_nodes` (`$THREEPM_HOME/cli.key`, moved from `~/.3pm/agent/` on
first use). `unlink` deletes the app (a missing one is not an
error), saves, then removes the directory except `authorized_keys` and
`shell.key`; not linked means the daemon is not even asked. `logs/`
holds the log pipeline's positions, source ids and miner state ("Logs"
below), so an unlink forgets them too.

## Fleet ssh access: `authorized_keys`

A node opens a `3pm fleet ssh` session (every mode: shell, exec, `cp`,
attach, `-L`, `-W`, `-R`, `--user`) only for a caller key listed in
`<state dir>/authorized_keys` — `$THREEPM_HOME/agent/` or
`THREEPM_AGENT_STATE_DIR`, `/root/.3pm/agent/` on a root node. Lines
`SHA256:<fingerprint> [comment]`, `#` comments, a bad line skipped with
a warning. sshd's `StrictModes`: owned by the agent's euid and not
group/other-writable (0644 accepted), else refused. **Absent or empty:
every session refused.** Re-read per session (64 KiB cap); written
atomically under `flock` of a sibling lock file by `3pm link
authorize|deauthorize`, `3pm link … --authorize` or
`THREEPM_LINK_AUTHORIZE`; listed by `3pm link status` (JSON
`authorized_keys {path, keys[], closed?}`). A file posed by hand by an
agent ≤ 0.48 is honoured as is once the agent is updated.

**Where the check sits.** `shell/host.rs`: the session takes one of 16
handshake slots (5 s for the caller's first message, 60 s for its
answer: the first-contact prompt), at most 8 of them per caller as the
backend stamped it (`from`, `shell/gate.rs`: the key itself is only
known at the handshake's last message, so this is what keeps one
stolen `3pmu_` token from refusing everyone else `unavailable`; the
same account's own token stolen still can, until it is revoked), the Noise handshake runs on the legs
(relay or LAN direct), the initiator's static key is checked against
the file, and only then is the session's permit taken, `--user`
resolved, a socket dialed or bound, or a process spawned. The proof is
a type: `shell::run` takes an `Authorized` that only the check builds.
Refusal: `AgentErrCode::KeyNotAuthorized` (`key_not_authorized`, the
backend's `forbidden`), the message naming the fingerprint and the
`sudo 3pm link authorize …` to type; the caller exits 255.

**The request is bound (prologue v2).** Prologue = `3pm fleet ssh e2e
v2` ‖ call id ‖ node id ‖ `Bound::encode` (`e2e/bind.rs`): tagged,
length-prefixed fields for the operation, argv and `pty`, the `RunAs`
user, the attach target/instance/mode/`read_only`/peek, `host:port`, an
`-R` takeover's listen and connection, `bind:port`, `term` and first
size, `record`. The command line encodes what it asked for (its
`nodeShell`/`nodeForward`/`nodeListen` body and the `record` the
backend answered), the node what it received in the `AgentRequest`;
any difference fails the handshake before anything starts. An `-R`
connection (`Forward{accepted}`) is taken only by the key that opened
the listen. A 3pm ≤ 0.48 cannot talk to a v2 agent: update callers and
nodes together.

**Recording.** `record` is the caller's (`--record`; without it the
command line sends `false`), lowered only
by the backend (bucket `record_shells` off, `cp`, forwards), bound by
the handshake; `[shell] record = "never"` in `link.toml` wins locally
and the node says so in its answer. The audit's command lines follow
the effective value (`shell/host.rs`: asked and the node records): an
unrecorded session builds no `vt100` audit (`shell/pump.rs`) nor typed
lines (`shell/attach.rs`), and emits no `line`/`screen`, only `open`,
`exec`/`attach` and `close {lines: 0}`.

**Recovering from a lockout** (lost key, file badly posed, node
closed): with local access to the machine (real ssh, the console,
`sshpass` as `envision@` on a tablet), `sudo 3pm link authorize
SHA256:…`; or, from a machine holding the offline rescue key made by
`3pm fleet key new --out`, `THREEPM_CLI_KEY=/Volumes/USB/secours.key 3pm
fleet key push <node> SHA256:…`. That rescue key is a fleet **caller**
key, not the release signing rescue key of `~/.3pm-release/`. Before an
agent that enforces the file reaches a node, pose the file with the
running agent (`3pm-rd/rd/fleet-ssh-authorized-keys/scripts/fleet-authorize.sh`, then
`fleet-authorize-check.sh`).

## Fleet identity, data socket and gate

Full behaviour and the CLI surface: `FEATURES.md` → *Fleet RPC* and
*Fleet link agent*. This is only the map of where it lives.

`session.json` carries the fleet half next to the link status
(`state::fleet::FleetSession`, `#[serde(flatten)]` over `SessionFile`):
`fleet_token` (the node's bearer, `3pma_<id>_<secret>`, redacted in
`Debug` and so safe to log) and `fleet_token_at_ms`, both replaced in
block at every `welcome` (a `welcome` with no bearer erases the
previous one) in the same file `publish_session` (`run.rs`) already
rewrites on every session change, 0600. `3pm agent fleet …` is a
*different process*: it reads the file rather than asking the running
agent, and reports `TokenProblem::Abandoned(pid)` when the file says
connected but its writer's `pid` is gone (`kill(pid, 0)`, `EPERM`
counting as alive): the file is rewritten only when the session
changes, so its age says nothing about a healthy agent — and judging
by it warned on every node linked for more than ten minutes.
`staleness()` speaks of the token's age only when that writer is gone
or the link is down.

`agent/data.sock` (`$THREEPM_DATA_SOCK` overrides) is opened by
`run_fleet::open` alongside the gate, never on its own — a platform without Unix sockets (`#[cfg(not(unix))]`) opens
neither, and a call that needs either reads `node_offline`. It serves
the SDK's `PUT`/`GET` contract (`packages/node/README.md` → *`data.sock`*)
and relays to the backend on the agent's own warm connections, without
ever handing the SDK a network address or a backend token.

The **`fleet_ready` gate** is a second control connection the module
opens to its own daemon line: `run_fleet::gate_config` reads the
32-byte secret the daemon drew for this launch of the `agent` app from
`$THREEPM_FLEET_SECRET` (`threepm_protocol::control::FLEET_SECRET_ENV`,
cleared from the process's own environment right after) and the pid
identity (`ctl::Identity::from_env`), then sends `hello{source:
"fleet"}` followed by `fleet_ready{node_id, cluster_id, data_sock,
secret}`. A root daemon that hosts users answers an opened gate with
`hosted_socks{socks: [{uid, gid, path}]}` — `<hosted home>/agent/data-<uid>.sock`,
the path the SDK of a hosted app looks for first — and `data::hosted`
binds each one on the same server as `data.sock` (same `handler::Ctx`,
so the same tokens), `lchown`s it to the user, 0600. The list is whole
(empty when the daemon hosts nobody) and the agent serves exactly it: a
reopened gate keeps what it already serves and closes what the list no
longer names — a daemon handoff re-reads `[users]` and cuts the gate. A path it
cannot take (a file or a link there, no directory) costs that user
alone. No secret in this launch's environment — an old daemon, or a
process that is not the daemon's own `agent` row — means no gate and no
route for a node-placed call; `fleet_gate::run` is what the daemon then
believes.

`3pm agent fleet nodes | apps | inspect | call | top` (`3pm fleet …` is
sugar over it, like `3pm link`) live in `src/fleet/` (`nodes`, `apps`,
`inspect` and `call` over `http::FleetHttp` with the `session.json`
bearer, `top` in `src/fleet/top/` fed by the backend's SSE) and
`src/fleet_gate/` (the server side that receives a `Forward` for this
node and places it). All five need only this host's own link — no
admin token, no separate credential.

## Logs

The agent reads this host's logs and posts them to the backend over HTTP,
never on the link (`link-protocol.md` §6b; user view: `FEATURES.md`, "Log
shipping"). It runs while the last `welcome` carried `logs` and
`link.toml` `[logs]` agrees on at least one switch
(`LogsSettings::effective`); a `welcome` that changes what is wanted
restarts it, one without `logs` stops it. A disconnect does not: the
sender retries, the sources keep their positions.

```text
apps (bus) ─┐
journalctl ─┼─▶ fold → redact → prep → mine → policy → budget → ring ─▶ cut every 10 s ─▶ POST /logs
files ──────┘                                └─▶ catalog, counts ──┘
```

| File (`src/logs/`) | Role |
|---|---|
| `mod.rs` | The tree and its map; `[logs] exclude` globs (`*`, `?` match `/` too) |
| `raw.rs` | `Item` / `Raw` / `Mark`: a line cleaned (ANSI stripped, cut at 8 KiB), not yet redacted, with the position its batch commits |
| `ids.rs` | Short source ids (`a1`, `j`, `f3`), kept in `logs/ids`, never handed to another source |
| `commit.rs` | Read vs committed positions: `offsets` and `journal.cursor` advance only on `200` |
| `sources.rs` | The entry: spawns the producers `LogsWanted` enables, one bounded stream (`ITEMS_CAPACITY` 1 024) |
| `apps.rs` | The 3pm apps' output: a second bus subscription (`log`), its own cursor; `lagged` counted as `lag`; the agent's own app skipped |
| `journal.rs` | One `journalctl --follow --output=json --priority=<n>` child (Linux; `THREEPM_JOURNALCTL` in tests), respawned 1 s → 60 s, resumes after the committed cursor, first start `--since=now`, entries older than 24 h dropped (`stale`) |
| `merge.rs` | Discovery every 60 s: the host agent's `LogSources` (through the daemon) merged with automonit's by (dev, inode), `exclude` applied, sticky set |
| `automonit.rs` | The automonit `logs` action (version 1) every 5 min, 15 s timeout; asks only the root-installed module row, never an app a user named `automonit` |
| `vet.rs` | What a file automonit adds alone must look like: a log by its path, never `/etc`, `/root` or a private name, normalised, parent not a symlink, the declared owner uid |
| `files.rs` | The poll tailer (1 s): first open at the declared inode from its end, rotation and `copytruncate`, 256 KiB per tick, 512 files, parked after 10 min idle, `EACCES` skipped until the next pass |
| `tail.rs` | One file: `O_NOFOLLOW \| O_NONBLOCK \| O_CLOEXEC` + `fstat` against the expected (dev, inode, uid; a successor: same owner, mode, one link), 64-byte fingerprint, lines with a stable `pos` |
| `redact.rs` | Secrets replaced by `<REDACTED>` before anything else reads the line; a pattern list, linear (`regex`), one line at a time (a YAML env `value:` on the line after its `name:` is not caught) |
| `prep.rs` | The PoC's prep V2: process tag kept, timestamp and level stripped, exception heads read as errors, logfmt/JSON as fields, ids masked; continuation lines folded |
| `drain.rs` | Drain (similarity 0.5 on informative tokens, level in the leaf key, bounded) |
| `mine.rs` | The miners of every source and their state (`logs/miner`, atomic, bounded); template ids never reused (an epoch in every wire source id, `f3.k2x9`) |
| `policy.rs` | What leaves line by line: counts per 5-min window always, a raw line only for `new`, `error`, `back`, `spike`, `return`, `context` |
| `policy_counts.rs` | The count windows and parameter stats (distinct + top 3), closed into `cnt` records within the wire's bounds |
| `budget.rs` | 4 KiB/s per source (256 KiB burst) and the 4 MiB RAM ring while the backend is away; over either, `dropped {budget\|ring}` |
| `gather.rs` | What the pipeline holds between two cuts: catalog, declarations, drops, positions; empty lines neither mined nor counted |
| `pipeline.rs` | The loop: items in, a cut every 10 s to the sender; `logs/status.json` for `3pm agent status`; the stop (`STOP_BUDGET`) |
| `ship.rs` | Cuts into parts under 1 MiB gzip / 4 MiB inflated, each opening with `batch {run, seq, ts}`; one in flight; `200` commit, `429` wait, `413` split, `400` drop, `401` next bearer, `503`/5xx/timeout retry 1 s → 60 s |
| `start.rs` | When the pipeline runs (`desired`), `HttpPoster` (the session's current bearer at each post) |
| `../http_logs.rs` | `FleetHttp::push_logs`: one `POST /api/buckets/<cluster>/logs`, the answer as a status code |

State, under `<home>/agent/logs/` (written atomically, `write_private`):

| File | Holds |
|---|---|
| `offsets` | per file `<dev> <ino> <offset> <fingerprint> <path>`, committed on `200` |
| `journal.cursor` | the journal cursor of the last accepted batch |
| `ids` | the source ids handed out, and the counter |
| `miner` | every source's templates and id counter, with its epoch; saved before a cut that carries a new template |
| `status.json` | the counters of the `logs` row in `3pm agent status` (not state: rewritten while running) |

A line read but not accepted before a stop is read again at the next
start and stored once by the backend (`(node, src, pos)`); app lines
carry no `pos` (the bus is never re-read).

**Stop.** On `shutdown` the session ends first, then `stop_logs`: the
pipeline closes every open window and tries a last batch within
`STOP_BUDGET` (5 s; `run` gives it 9 s at most before aborting), then
the other workers stop. The agent's row has `kill_timeout_ms = 10000`
(`start.rs` `KILL_TIMEOUT_MS`, and `[module] kill_timeout_ms` in
`module.toml` raises rows linked before at the next module refresh), so
the daemon does not SIGKILL it at the default 1.6 s. What cannot go out
in time is lost as counts (they are deltas) and read again as lines.

## Link traffic

`src/link_traffic.rs`: process-wide counters per (direction, kind) —
the closed list in `threepm_link::traffic`. Bytes and messages are
added where they pass: the writer and `send_frame` (a frame's compressed
payload; a `state` frame split pro rata of its patches' JSON length,
`Deflater::frame_sized` giving the JSON length), `next_frame` (a Text
frame's payload, by what it decoded to; pings 0 B), `FleetHttp::send_bytes`
(the one counted path of `get`/`post`/`put_artifact`/`put_blob`, the
caller's kind), `push_logs` (every attempt), and the relay's blob legs
(`link_traffic::Counted`, per chunk). `node_state/traffic_cut.rs` cuts
the `traffic` section every `max(1, 5 / state_s)` ticks through the
same engine; `link_traffic/file.rs` writes `<state>/traffic.json`
(counters + the last minute's rates) once a minute for the `traffic`
row of `3pm agent status`.

## Files

| File | Role |
|---|---|
| `main.rs` | Role detection (`argv[1] == "__run"`), clap dispatch, `[3PM][ERROR]` on failure |
| `lib.rs` | Module tree, `VERSION`, `AGENT_NAME`, `MODULE_NAME`, `now_ms` |
| `cli.rs` | `Cli { Link(LinkArgs) \| Unlink {json} \| Status {json} }`, `json_wanted(flag, lookup)` |
| `inputs.rs` | Pure `LinkInputs::resolve(&LinkArgs, lookup)`: env > positionals, parsing, the insecure guard, caps sorted/unique/known, `InputError` |
| `options.rs` | `AgentOptions {state_dir, name}`: `from_env(lookup, home)` (infallible: an empty value is no value) / `to_env()` round trip, `truthy` |
| `state/mod.rs` | `StateDir`: paths, `is_linked` (`link.toml` **and** `node.key`), `create`, `remove`, `remove_stale_files` (`node.crt` / `ca.pub`, unused since link 3), typed `read_*` / `write_*`; re-exports `LinkConfig` / `SessionFile` from `threepm_link::local`, whose shape the CLI reads too |
| `state/fs.rs` | `write_private` (0600 before any byte) / `write_public`: temp + fsync + rename |
| `identity.rs` | `generate_key`, `pubkey_b64`, `sign_enroll_proof(key, token_id)`; `NodeIdentity {key}`: `save`, `load` (which drops what an older link left beside the key), `pubkey_b64`, `sign_link_proof`, `direct_identity`. The key only: the `Authorization` header is two public ids, built where it is sent (`threepm_link::LinkAuth` in `session.rs::open`) |
| `tls.rs` | `connector(&ServerUrl, insecure) -> Option<Connector>`: Plain → `None`; Tls → rustls (ring) + `webpki-roots`; `--insecure` over TLS → refused loudly until the hardening step |
| `codec.rs` | `Deflater` (one stream per connection, `Sync` flush per frame), `Inflater` (for the test backend; cap inclusive, exactly 1 MiB passes like the real backend), `decode_server` (capped; a tag this build does not know is `Unknown`, a body it cannot read is `Invalid`) |
| `transport/mod.rs` | `ws_config()` (1 MiB messages/frames, 32 KiB write buffer → 1 MiB max), `connect(url, headers, connector)` (sub-protocol, `User-Agent`, 10 s, HTTP rejection → `ConnectError::Rejected {status, code, message}`), `Frame {Link, Close, Eof, Other}`, `next_frame` / `read_frame`, `send_link`, `close_reason_code` |
| `transport/handshake.rs` | `HelloFacts {boot_id, caps}`, `Welcomed` (`pace`, `fleet_token`, `pending_calls`), `check_challenge` (version overlap → the negotiated version, then skew; the agent speaks link 5 only, so an older server is `HandshakeError::LinkVersion` — `ERR_LINK_VERSION`, `Disconnect::LinkVersion` — before anything is signed), `hello_frame` (the signature over the challenge, the facts and the negotiated version: three frames, no separate `proof`), `read_link` (skips `unknown` frames within the timeout), `handshake(…)`, `HandshakeError::disconnect()` |
| `transport/writer.rs` | `Outbound {Link, Tracked(Link, oneshot::Sender<bool>), Scoped(Link, ConnId, Option<oneshot::Sender<bool>>), Close}` (`Tracked` answers `true` once the bytes went out, `false` when the codec dropped the frame, nothing when the connection died first; `Scoped` is a frame of one connection — a call's ack, its reply — skipped once that connection ended or its sender gave up, answered like `Tracked` when someone waits; a `Tracked`/`Scoped` frame nobody waits for any more is skipped, never written stale on the next connection), `run(sink, deflater, ctrl_rx) -> WriterEnd {Closed, Dead}` (every frame rides the one control channel), `encode_or_drop` (a frame over the cap is dropped and its tracked sender told `false` — never `Dead`; the node's state is packed under the cap before it gets here); `CTRL_CAPACITY` 256, `FLUSH_EVERY` 5 s, `PING_EVERY` 20 s |
| `transport/reader.rs` | the frames handed to the session (a call, a `state_resync`, a call result). The reader sends no `ack` of its own; a tag this build does not know is skipped and counted, and a body it cannot read is counted too and, for a `call`, answered `err malformed`, `run(stream, ctrl_tx, inbound_tx) -> ReaderEnd {Silence, Disconnect}`; `SILENCE` 60 s |
| `session.rs` | `SessionConfig {…, stop_probe, stable_session}` (defaults = the constants, injectable for real-time tests), `SessionStatus` (watch: connected, sessions, retries, attempt, last_disconnect, parked, server_version, pace, fleet_token, frames_invalid; in `session_types.rs`), `SessionHandle {ctrl_tx, status_rx}` + `take_state_resync()` (the `state_resync` orders for the node-state task, once; `RESYNC_CAPACITY` 16, `try_send`: a task not keeping up loses the order rather than stalling the link), `Session::new` / `run(shutdown)` (connect and handshake abort on shutdown; no backoff sleep after a shutdown), `random_boot_id`, `STOP_PROBE` 1 h, `CLOSE_GRACE` 2 s, `STABLE_SESSION` 60 s |
| `daemon.rs` | `DaemonLink::new(paths, role)`: `request(&Request)` (connects on the schedule with an identity `Ping`, bounded, a failure drops the client), `info()`, `boot_id()`, `replaced()` (another `boot_id` since the last call, said once: the node-state task then owes every section), `is_connected()`, `lost(why)`, `with_timeout`; `DAEMON_TIMEOUT` 3 s, reconnect 1 → 30 s — one for the node-state task, a pool of them for the commands; `daemon/testing.rs`: `FakeDaemon` (scripted daemon for the in-crate tests: `list`, `host`, `revs`, `trees`, `trees_unsupported`, `answers_per_connection`) and `fake_writer` (confirms every tracked frame with a verdict) |
| `node_state.rs` | `NodeStateTask::new(paths, redact).run(status_rx, resync_rx, ctrl_tx, shutdown)` (`gen` = the process's start in ms; ends with the session's `state_resync` sender; a period change from `welcome` is taken at once). A tick is `ask` (the daemon's complete states, a `Snapshot`; `plan(procs, paces, now)` says which `AppMetrics` / `ProcessTrees` to ask from the table's counters) then `round` (`cut` every section and the `gone` ones, `pack` them in frames under the cap — a section never split —, `stream::write`, `commit` on the writer's `true` only, which also records the `(gen, rev)` the section was shaped from). `CRC_EVERY` 60 s per section, each at its own moment of the minute (`paces::crc_minute`: a hash of the section's name shifts where its minute turns, and a patch carries a CRC when that minute is not the one of the section's last CRC: the sections that move every tick are not all serialized and checked on the same one); a patch's `p` moves into the frame, measured without being written (`json::json_len`); `TREES_RETRY` 60 s |
| `node_state/paces.rs` | What the `welcome`'s `Pace` asks of that task: the tick (1 s floor — the stream is always on), `Mode {Silent, ShapeOnly, Figures(n)}` from `pace.trees`, `Paces::of(status)` (`metrics_every` and the figures counted in ticks) |
| `stream.rs` | What a streaming task is made of: `ticker`, `Wake<T>` / `wake` (what woke the loop: the tick, a status change, what the backend asked, a stop), `write` (a tracked frame, confirmed by the writer; `CONFIRM_TIMEOUT` 5 s) |
| `events.rs` | `filter()` (`process`, `daemon_shutdown`, `app`), `emit_of(line)`, `run(paths, ctrl_tx, shutdown_tx, shutdown)`; lifecycle events `send().await`, `app` events `try_send` (dropped with a debug line when the channel is full); backoff after an EOF too, reset only after a `STABLE_BUS` (10 s) subscription, "bus closed" warned once per outage |
| `enroll.rs` | `HostFacts::detect`, `Enrollment {cluster, token, key, facts}`, pure `enroll_body(req) -> threepm_link::EnrollRequest` and `check_answer(answer, now_ms)` (the version picked must be one this build speaks, then the 5 min skew ⇒ `EnrollError::ClockSkew {delta_ms}`), `enroll(&reqwest::Client, &ServerUrl, &Enrollment) -> EnrollResponse` (one `POST` on `ServerUrl::enroll_url()`, the client being the agent's own — `http::client`, which refuses `--insecure` over TLS), `EnrollError {Transport, Timeout, Refused {code, message}, Status, LinkVersion, ClockSkew, Shape}`; `ENROLL_TIMEOUT` 10 s |
| `start.rs` | `spec(&AgentOptions, exe)` via `SpecPatch`, `register(daemon, spec)` = `Start {if_changed: false}` then `Save`, `affected_of` |
| `link_cmd.rs` | The `link` command; `refuse_existing_link`, pure `link_config(inputs, enrolled, app, now_ms, direct)` |
| `unlink_cmd.rs` | The `unlink` command; `delete_app` (NotFound tolerant), `save` |
| `status_cmd.rs` | Pure `view(cfg, app, process, session, now)` / `render` (with its `dashboard` line), `dashboard_url(server, cluster)`; the verdict is `threepm_link::local::status`; `Describe` answers `ProcessList` |
| `run.rs` | The `__run` role: options, state, identity, readiness, signals, tracing, the session and its three workers (`spawn_workers`: `publish_session` keeping `session.json`, the node-state task, the event forwarder), the fleet side from `run_fleet.rs` |
| `logs/`, `http_logs.rs` | Log shipping: sources, pipeline, sender, one row per file in "Logs" above |
| `e2e/` | The end-to-end channel of `fleet ssh`: `Noise_XX` handshake bound to call + node (`mod.rs`), records `[u16][Noise message]` and their `Sealer`/`Opener` (`record.rs`), compressed frames of a command on pipes, agreed in the handshake's payloads (`compress.rs`), legs as streams (`stream.rs`), `shell.key` / `cli.key` and their `SHA256:` fingerprints (`key.rs`) |
| `shell/forward.rs` | The target of `forward` (`-L`, `-W`, a `-R` takeover): connect (5 s) or take an `accepted` connection, `pump` both ways, then `pump_then_send` — the caller's leg held until the output leg's `PUT` ended, or the relay drops the answer's tail |
| `shell/listen.rs` | The target of `listen` (`-R`): `bind_addr` (loopback unless `gateway_ports`), the accept loop, the parked connections (`LISTEN_CLAIM_MS` 10 s, taken once by the same `from` and caller key), `bound` / `accepted` / `keepalive` on the legs, the end on `END`, a cut leg, `LISTEN_IDLE_MS` 60 s of silence or the agent's stop |
| `fleet/ssh/forward.rs` | The command line's `-L` (`serve`, `tunnel`), `-W` (`proxy`: stdin/stdout, stdout closed for ssh's EOF, exit 0 / 255) and `carry` (one `nodeForward`, `To::Dial` or `To::Accepted`, the output leg held until the input leg's `PUT` ended) |
| `fleet/ssh/reverse.rs` | The command line's `-R`: one sealed `nodeListen` per spec, keepalive every 20 s, each `accepted` taken over by a `nodeForward {accepted}` and `carry`, at most `TAKEOVERS_MAX` at once per `-R`, `END` on close |
| `shell/record.rs` | The node's recording of a sealed session: asciicast before sealing, bounded queue, upload before `exited {recording}`, stale files swept at start |
| `fleet/known_nodes.rs` | `~/.3pm/known_nodes` (TOFU): lookup, pin under a lock, the `/dev/tty` question, `--accept-new`, the changed-key refusal, `known-nodes ls` / `rm` |
| _(control channel)_ | `threepm_protocol::module::ctl` — `hello` then `ready`; the `shutdown` callback flips the agent's `watch` instead of exiting the process |

Registration sites (to duplicate for any new module): `justfile`
(`fmt`, `msrv`, `modules`, `e2e-modules`, `bundle`),
`.github/workflows/ci.yml` (`modules`, `msrv`), `release.yml`,
`distribution/nfpm.yml`, `docs/modules.md`, `FEATURES.md`,
`CHANGELOG.md`, `CLAUDE.md`. `install.sh` copies `modules/` as a whole.

## Tests

| Suite | Command | What it proves |
|---|---|---|
| unit | `cargo test --manifest-path modules/agent/Cargo.toml --lib --bins` | Per file; writer and reader under paused tokio time (ping at 20 s, flushes at 5/10/15/20 s, frames in order, a tracked frame confirmed and a stale one skipped, an oversized frame dropped and its sender told so, silence at exactly 60 s, a pong resets it; `state_resync` handed over, link 1's `resync` / `trees_request` / `host_resync` skipped); the handshake (a server one version behind refused before anything is signed, with `ERR_LINK_VERSION`); event mapping; `DaemonLink` schedule and bounded calls; the node-state task from the daemon's answers on (`node_state_tests.rs`, against the writer stand-in: every section whole in one frame first, then what moved, then a heartbeat, the mock `NodeReader` rebuilding what the daemon said; a frame not written commits nothing and is cut again at the same revisions; a resync, a new session and a replaced daemon owe full sections; what the counters say to ask — metrics every n ticks, shape only, silent; what is no longer there said `gone`, nothing while the daemon is down; a CRC once a minute per section, each at its own moment of it; frames under the cap, a section over it alone left out, warned once and asked again) and asking a scripted daemon (`node_state_ask_tests.rs`: whole states, no host agent, a daemon predating trees asked again only after the retry, a daemon down an empty tick, a replaced one owing everything); the paces (`period` never silent, the modes, ticks); `stream.rs` (what woke the loop, a closed writer channel is not a write) |
| session | `… --test session` (`tests/session/{main,shutdown,enrolling,reconnect,state,…}.rs`) | The session in process against the mock: handshake facts, reconnect ≤ 3 s after a cut, 4403 revoked ⇒ parked with no retry, 403 at the upgrade ⇒ probe on the stop schedule then relink, ack before reply, unknown frame skipped (`sessions == 1`, `retries == 0`), polite close 1000; shutdown ends a session stuck in connect / TCP-accepted-but-silent / held handshake within 1 s; enrollment refused on clock skew (nothing written) and accepted within tolerance; `attempt` back to 1 after a stable session; `error` instead of `challenge` ⇒ `Closed {4401, ProofInvalid}`; backend up late ⇒ retries grow then connected; the `welcome`'s `pace` in the status; the node's state (`state.rs`), the real task reading a scripted daemon (`tests/support/fake_daemon.rs`): every section whole with its CRC in the session's first frame, the mock's `NodeReader` rebuilding the table, the host, `app/1` and `tree/1` from the frames alone with never a gap; patches when the host and a registry move; heartbeats and no other frame of the agent's own when nothing does; one section whole again on `state_resync`, every one on a new session; an instance deleted said `gone`; a `pace` asking for less than a second still gets a frame a second |
| e2e | `THREEPM_BIN=target/debug/3pm cargo test --manifest-path modules/agent/Cargo.toml --test e2e --test e2e_trees` (or `just e2e-modules`; `tests/support/e2e.rs` stages the bundle for both) | A real 3pm dispatching to the staged module against the mock: status/unlink before any link, bad inputs refused without state, enroll ⇒ app online and in `dump.json`, state files and modes, `status`, refusals write nothing, pending + `--force`, unlink; the running app: hello with its signature, reconnect with the same boot id, Close 1000 on `stop` (no SIGKILL), new boot id on `restart`, parked when revoked; the node's state with a real sleeper: ≥ 3 `state` frames of one `gen` with a growing `ts`, every section whole with its CRC the first time, the host and the table rebuilt through `NodeReader` (the daemon's pid and protocol, the sleeper with its env key names), no env value / home path / script path in any frame, `emit {process}` on restart, `emit {daemon_shutdown}` then Close 1000 on `3pm kill`; the Node SDK under the real daemon (skipped without `node`): the instance's `app/<id>` section whole, then chained patches naming the counter's value and defining nothing again, the `deploy` event as `emit {app}`; a shell with two sleeps: a `tree/<id>` section with `sh` and its two `sleep`s, the same `id`/`gen` as the table's row, whose `proc_count` says 3 (the agent's own row says 1: no tree section is ever sent for it), held while the link goes on with heartbeats |

`tests/support/` is the mock backend: `mock_backend.rs` (its own thread
and runtime on `ws://127.0.0.1:0`, per-instance cluster, tokens
with uses and pre-approval, every agent frame recorded, `Order::{Cut,
Close(code, reason), Push(Link)}` to every live link, node state
switch, the `pace` it welcomes with (a tick a second and no trees by
default), `state_frames()` and `node()`
(the frames applied in order to a `NodeReader`, a gap failing the test), `hold_handshake`,
`refuse_handshake`, `enroll_skew_ms`, `link_range` (a backend of another link generation),
`unbound()` / `listen()` / `stop_listening()` (the same nodes across a
restart), `wait_for` / `wait_for_async`), `mock_serve.rs` (the accept
loop, and the `Authorization` ids looked up **before** the upgrade with
the same 401/403 answers), `mock_enroll.rs` (the `POST` handler, the
real backend's check order in miniature; its requests are kept apart
from the bucket ones — `enroll_requests()` / `enroll_seen()`),
`mock_link.rs` (challenge/hello/welcome, pings and agent close
codes counted).

## Notes and caveats

- Paused-time tests never involve the network: tokio auto-advances the
  clock while a task waits on I/O, so a paused test against the mock
  would skip its own timeouts. Session-level tests run in real time with
  short schedules (`stop_probe` is configurable for that).
- In a current-thread test, block only with `wait_for_async`: a
  `std::thread::sleep` starves the session task.
- The dispatcher fix that came with step 8: `3pm <module> … --json` is
  honored by `crates/threepm-cli/src/modules/dispatch.rs` (spawn notice
  to stderr, `THREEPM_JSON` set), so a first run's stdout stays one
  JSON document.
- `--insecure` over `wss://` is
  refused with a clear message until the TLS hardening step; `ws://` to
  loopback needs no flag.
- The `daemon_shutdown` path: the agent is itself a child of the daemon,
  so it forwards the event, closes the link and exits before the daemon
  reaps it.

</document>

<document url="https://3pm.edgecr4ft.net/docs/fleet/link-protocol.md" title="Link protocol 3pm.link.v1">

# Link protocol (`3pm.link`)

The contract between an agent and a fleet backend. Everything here is
typed in `crates/threepm-link` and frozen by the fixtures under
`crates/threepm-link/tests/fixtures/link/v5/`; a change that is not
additive bumps `LINK_VERSION`.

The vocabulary's generation is `link_min..=link_max` alone. The
WebSocket sub-protocol carries no number: it names the framing, a
mismatch is a 400 at the upgrade, and a version outside the accepted
range is `ERR_LINK_VERSION` one frame in ("update 3pm"). No build speaks
two versions, so a fleet crosses one **nodes first, backend last**: an
updated agent in front of an older backend is refused and links again on
the `Auth` schedule once the backend is deployed.

## 1. Identifiers

| Thing | Format |
|---|---|
| Cluster id | `3pmc_` + 16 lowercase base32 characters (80 bits). Public partition key. |
| Node id | `3pmn_` + 16 |
| Enrollment token | `3pmk_<token_id8>.<secret32>`. The backend stores `token_id`, `sha256(full token)`, expiry, `uses_left`, `pre_approved`. Its `Debug` is redacted. |
| Fleet token | `3pma_<id10>_<secret64>`, lowercase hex (40 + 256 bits). The node's bearer for the backend's HTTP routes, minted in every `welcome` and never stored on disk by the backend: it keeps `sha256(token) → node row` in memory. Its `Debug` shows `3pma_<id10>_<redacted>`. |
| Blob id, call id, artifact id | `^[A-Za-z0-9_-]{1,64}$` (`BlobId`, `CallId`, `ArtifactId`). The backend mints blob ids from 16 random bytes, call ids from 12 and artifact ids from 16, base64url; the agent derives a call's blobs as `<call>-d<n>` and `<call>-f<n>`, and never mints an artifact id — it echoes the one the upload answered. |
| Digest | 64 lowercase hex characters (`Sha256`) |
| Keys, signatures, nonces | base64url without padding; Ed25519 keys 32 bytes, signatures 64; nonces ≥ 16 bytes, single use |

Base32 is RFC 4648 lowercase without padding (`a–z`, `2–7`), chosen so a
line can be pasted and read aloud without ambiguity.

Blob, call and artifact ids become **path segments** (`PUT
/blobs/<blob>` on the agent's data socket, `GET …/calls/<call>` and `GET
…/artifacts/<artifact>` on the backend), so their alphabet is
base64url's and nothing more — no `.`, hence no `..`. Every
identifier above, and every digest, is enforced by its **type**, at the
field, when the frame is decoded: a frame that breaks one never becomes
a frame (§4), and nothing downstream has to remember to check.

A node **is** `(cluster_id, node_id, Ed25519 public key)`: one row on the
backend, `node.key` + `link.toml` on the host. There is no certificate
and no CA — the row is the authority, and it is re-read at every
handshake, so a revocation or an approval takes effect on the next
connection rather than when a document expires.

## 2. Enrollment — `POST /api/v1/enroll`

One request and one answer over HTTPS: a host trades an enrollment token
for a node id, which needs no session and no nonce of its own.

Request body (`EnrollRequest`):

```json
{"cluster_id": "3pmc_…", "token": "3pmk_….…", "node_pubkey": "<base64url>",
 "hostname": "web-01", "os": "linux", "arch": "x86_64",
 "agent_version": "0.19.0", "protocol": 1, "link_min": 5, "link_max": 5,
 "proof": "<base64url>"}
```

`proof` = Ed25519(node key, `enroll_message(token_id, node_pubkey)`) =
`"3pm-enroll-v2\0" token_id "\0" node_pubkey`. No server nonce: what a
replay has to beat is the token's use count, and the proof binds the
token to the key it enrolls.

Answer 200 (`EnrollResponse`): `{node_id, state: active|pending, link,
server_time_ms}`, `link` being the highest version the request's
`link_min..=link_max` shares with the server's. A refusal is an HTTP
status with the body `{"error":{"code","message"}}`: 400
`ERR_BAD_REQUEST`, 401 `ERR_TOKEN_INVALID` / `ERR_TOKEN_EXPIRED` /
`ERR_TOKEN_EXHAUSTED` / `ERR_PROOF_INVALID`, 403 `ERR_NODE_REVOKED`,
426 `ERR_LINK_VERSION`, 429 `ERR_RATE_LIMITED`.

Server checks, in this order: body well formed → the two version ranges
overlap → cluster known → `token_id` known and hash matches (constant
time) → token not revoked / expired / exhausted → `node_pubkey` is 32
bytes → proof verifies → an existing node with that key is not revoked →
**atomic decrement**
`UPDATE tokens SET uses_left = uses_left - 1 WHERE id = ? AND uses_left > 0`
(0 rows ⇒ exhausted) → upsert node (same `node_id` for a known key, else
random; `active` when the token was pre-approved, else `pending`). The
use and the node land in **one transaction**.

The agent writes nothing before the answer, and nothing at all when
`|server_time_ms − now| > 5 min` (`ERR_CLOCK_SKEW`, the delta shown).
The request is bounded to 10 s, and the server budgets concurrent
enrollments per client address (16 by default; beyond it 429
`ERR_RATE_LIMITED`).

## 3. Link — `GET /api/v1/link`

Headers: `Authorization: 3pm-node <cluster-id>:<node-id>` (`LinkAuth`:
the two ids as they are pasted, prefixes included, one colon),
`Sec-WebSocket-Protocol: 3pm.link`, `User-Agent: 3pm-agent/<version>`.

That header is a **claim**, never a credential — both ids are public.
What proves the node is `hello.sig`, verified against the public key
**on the row**; the row also carries the state, and both are read again
at every handshake.

HTTP rejections before any frame, body `{"error":{"code","message"}}`:
400 `ERR_BAD_REQUEST` (header missing or not that shape, wrong
sub-protocol), 401 `ERR_NODE_UNKNOWN` (no such node in that cluster),
403 `ERR_NODE_REVOKED`, 403 `ERR_NODE_PENDING`, 429
`ERR_RATE_LIMITED`, 503 `ERR_LINK_BUSY` (the backend holds as many
pending handshakes as it takes — from that client address, or in all:
retry soon; an agent that predates the code reads it as `ERR_UNKNOWN`,
and the 503 alone makes it Transient). Not `ERR_LINK_VERSION`: the
version is judged on `hello`, one frame in, and closes 4426.

| # | Direction | Message |
|---|---|---|
| 1 | S→A | `challenge {nonce, server_time_ms, link_min, link_max}` |
| 2 | A→S | `hello {sig, boot_id, protocol, link, version, caps, direct_port?}` |
| 3 | S→A | `welcome {link, server_version, pace, fleet_token?, pending_calls?}` |

`sig` = Ed25519(node key, `link_message(nonce, node_id)`) =
`"3pm-link-v1\0" node_id "\0" nonce`, `node_id` being the one of the
`Authorization` header: the signature is what makes that claim true, so
the agent's name (a constant) and the node id (already in the header)
are not fields of their own.

The agent checks the challenge before signing: the version overlap (the
highest common version goes into `hello.link`), then clock skew
(`|server_time_ms − now| ≤ 5 min`, else `ERR_CLOCK_SKEW` with the
delta). Unknown frames before `challenge` or between `hello` and
`welcome` are skipped on both sides, within the same 10 s.

A frame the peer cannot **decode** is read differently here than on an
established session (§4), because there is no session to protect yet.
The agent fails the attempt on one: a `challenge` or a `welcome` it
cannot read is a disagreement it will not resolve by waiting, so it
retries on the Transient schedule rather than reporting a handshake
timeout. The backend skips one and reads on within the same 10 s budget:
the peer has not proved anything, so its frame is worth no more than an
unknown tag. Failures:
Close 4401 `ERR_PROOF_INVALID`, 4408 `ERR_HANDSHAKE_TIMEOUT` (10 s),
4426; the server reads until the agent's close (2 s cap) before dropping
the socket, so the close code is never lost to an RST. A second
connection of the same `node_id` closes the previous one with 4409
`ERR_SESSION_REPLACED`. The server re-reads the node's state right after
registering the session: a revoke that raced the handshake closes it
4403 at once.

`boot_id` is random per agent process, so the server tells a restart
from a reconnect. `hello.caps` is what the **host owner** allows
(`link.toml` `caps`), declared so the backend can show it and enforced
by the agent: the backend grants nothing. Capabilities: `metrics:read`,
`proc:restart`, `proc:stop`, `proc:action`, `node:shell`; one this build
does not know decodes to `unknown` and is dropped. All are on by
default; a `link.toml` of version 2 that lists exactly the four before
`node:shell` reads as all of them (`agent.md`, *State directory*).
`direct_port` is the TCP port of the agent's LAN listener when it has
one (`link.toml` `direct`, on by default), announced at every handshake
(§5, *Bytes between LAN neighbours*).

`welcome.pace` is how fast the backend wants the node's state, and it is
mandatory in full:

```
pace { state_s: u32,            // the state tick
       metrics_s: u32,          // the app/* sections
       trees: {mode: "off"} | {mode: "shape"} | {mode: "figures", every_s: u32} }
```

Every period is floored at 1 s where it is decoded. `trees` is tagged on
`mode`, and a mode a build does not know reads as `off` — fewer sections
is the safe way to be wrong, and the backend learns from the sections it
never gets.

`welcome.pending_calls` lists the calls this node placed that settled
(or are still open) while it was away; the agent fetches each result
over HTTP (`getCall`). Entries that are not legal call ids are dropped
on decode, repeats are deduplicated, and the list is cut at
`PENDING_CALLS_MAX` (256) distinct ids — a receiver pages through
`getCall` for the rest rather than refusing the frame.

## 4. Frames

Every message is one JSON object tagged by `t`. One enum per direction,
so each side's `match` is exhaustive over what can actually arrive and a
frame it could never send is not in its vocabulary.

Server → agent (`ServerFrame`):

```
challenge {nonce, server_time_ms, link_min, link_max}
welcome {link, server_version, pace, fleet_token?, pending_calls?}
call {id, req: AgentRequest, deadline_ms?}
call_result {call, state, results, err?, output?}    a call THIS node placed has settled
state_resync {secs}                                  these sections did not apply: send them in full next; none named = all
error {code, message, server_time_ms?}
```

Agent → server (`AgentFrame`):

```
hello {sig, boot_id, protocol, link, version, caps, direct_port?}
ack {id}                                             sent by whoever accepts the call, as its first act
reply {id, res: AgentResponse}
emit {ev: AgentEvent}
state {ts, gen, secs}                                every pace.state_s; none = the heartbeat (§7)
path_report {blob, path, bytes}                      which way a blob this agent wrote to a peer went (direct | relay)
blob_stat {blob, bytes, last_ms}                     every 5 s: progress of a live blob gone direct, latest figure only
error {code, message, server_time_ms?}
```

**Decoding is two steps**: the tag, then the body.

| Outcome | Meaning |
|---|---|
| `Frame(f)` | a frame this build knows |
| `Unknown {tag}` | a tag it does not: skip it, count it |
| `Invalid {tag, id?, why}` | a known tag whose body did not decode: skip it, count it, answer it when `id` says which call it was about |

`id` is the frame's `id` (`call`, `ack`, `reply`) or `call`
(`call_result`) when the raw object carried one **as a string** that is a
legal call id; anything else names no call to answer. `why` says what
failed and is cut at `WHY_MAX` (256 bytes) where it is built: a serde
message quotes the value it choked on, and receivers repeat `why` into a
refusal, a log line and a call row before the sender has authenticated.

**A frame body never ends a session** on either side: the backend
counts it (`GET /api/v1/info` → `link.frames_invalid`) and reads on; the
agent counts it (`3pm link status --json` → `frames_invalid`) and, for a
`call`, answers `reply {id, res: err {code: "malformed"}}` rather than
letting the call die on its deadline. Only a message that is not a JSON
object with a string `t`, or one over 1 MiB, is an error — a broken
codec, not a disagreement about a field.

Every new numeric field is an `Option`. Enum vocabularies (`ErrorCode`,
`Capability`, `NodeState`, `AgentRequest.op`, `AgentResponse.res`,
`AgentEvent.ev`, `AgentErrCode`, `Path`, and the link-local `ProcStatus`
/ `ProcEventKind` carried by `RedactedRecord.status` and
`AgentEvent::Process.kind`) carry an `Unknown` catch-all — an unknown
process status or event kind costs one field, never the frame. The one
field read leniently rather than typed is `action.peer`: a peer this
build cannot read is dropped and the call runs through the relay.

## 5. Commands and calls

`AgentRequest` (tag `op`): `ping`, `restart {target}`, `stop {target}`,
`action {target, instance?, source?, action, params, timeout_ms?,
input?, output?, stream?, peer?}`, `shell {term, cols, rows, input,
output, from, from_hostname?}`, `exec {argv, pty?, term, cols, rows,
input, output, from, from_hostname?}`, `attach {target, instance?,
mode?, read_only?, peek?, cols, rows, input, output, from,
from_hostname?}`, where `target` is `{id}` or
`{name}`. Never `toggle_watch`, never `all`, and there is no variant for
`start`, `delete`, `save`, `scale` or `kill` — the enum **is** the
allow-list. Required capability: `proc:restart` / `proc:stop` /
`proc:action` / `node:shell` (`shell`, `exec`, `attach` and `run_as`); `ping` needs none. `proc:action` covers
the **bytes** of a call as well as its trigger; the fine-grained
refusal is the backend's (`callers` on the target, and the declared
`input.max_bytes`).

`AgentResponse` (tag `res`): `pong {version, protocol, pid}`, `ok
{affected: [RedactedRecord]}`, `triggered {results: [Outcome]}`,
`exited {code?, signal?}` (the end of a `shell`, an `exec` or an `attach`, *Shell sessions*),
`err {code, message}` with `AgentErrCode` ∈ `not_found`, `forbidden`,
`unavailable`, `timeout`, `daemon_error`, `malformed`.

Commands are **at-most-once**: a `call` in flight at a disconnect is not
replayed and its reply is withheld; the UI shows "not acknowledged —
retry?" with a new id. The `ack` is sent by whatever accepts the call, as
its first act, so a call that could not be taken (the inbound queue was
full) is never acked and that view is the truth. Each call runs in its
own task, at most `CALLS_IN_FLIGHT_MAX` (4) at a time (`ping` excepted;
the fifth **waits** for a permit, in order, and is `unavailable` only
when none came before its deadline — less the margin its refusal needs
to go out; a session — `shell`, `exec`, `attach`, `forward` — is counted
apart and refused at once when its own budget is full), and asks the
daemon with a bounded budget:
an action's `timeout_ms` (30 s when absent) plus 5 s, 30 s for the rest,
never past the call's `deadline_ms`. A daemon predating the op answers
`err {unknown_request}`, forwarded as `daemon_error`; a daemon down is
`unavailable`.

`action` becomes the daemon's `dispatch` with `remote: true` (an action
declared `remote: false` answers `refused`), and each instance it
reached answers with one `Outcome`: the daemon's `ActionOutcome`
(`proc_id`, `name`, `instance`, `source`, `action`, `duration_ms`, `ok`,
`err? {code, message}`; codes `failed`, `timeout`, `busy`, `gone`,
`refused`, `too_large`, unknown → `unknown`) with its result in link
form:

| Key | What it is | How the bytes travel |
|---|---|---|
| `data? {p: "inline", gz_b64}` | the handler's JSON, gzip + base64url | in the frame, within the reply's budget |
| `file? {artifact_id, bytes, ext, sha256}` | the file the action returned | uploaded to the artifact store before the reply |
| `output? {blob, bytes, ext, sha256}` | the file it pushed to the blob the caller opened (`action.output`) | the blob relay, the caller pulls it |

`Outcome` is **one type in both directions**: the same object travels up
in a `reply` and back out in a `call_result`, because nothing in it is
bytes only an agent could send. Until link 5 there were two, and the
asymmetry was `blob_chunk`.

Inline is a **per-reply** budget, not a per-outcome one: every inline
`data` of one reply together holds at most 256 KiB of gzip
(`INLINE_MAX_GZ_BYTES`), spent in outcome order through `InlineBudget`
— which is what keeps the largest reply (16 outcomes, every field at its
bound) under the 1 MiB frame cap. A `data` that does not fit **fails its
own outcome** with `too_large` ("the result is N gzip bytes; a result
this size is a file"), and the rest of the reply travels: a result too
big for the frame is not quietly turned into bytes to fetch, it is an
action to rewrite. `err.message` is the handler's text (cut to 1 KiB by
the daemon) and travels in clear: no secret and no third-party input
belongs in an action's error message.

The file an action returns is an **artifact**, not a payload of this
link. Before it sends the reply, the agent uploads the file, hashing the
raw bytes as it streams them; the outcome's `file` then names what came
back. The upload is the one HTTP route of this contract that is not on
the link, so it is spelled out here:

```
PUT /api/buckets/{clusterId}/calls/{callId}/artifacts
Authorization: Bearer <the welcome's fleet_token, `3pma_…`>
content-length: <raw bytes>        required
x-3pm-ext: <ext>                   the extension the daemon gave the file (`is_file_ext`)
x-3pm-instance: <n>                which instance of the target app returned it
<body: the file's raw bytes, not compressed>

201 {"artifact_id": "…", "bytes": 184322, "sha256": "…"}
```

`bytes` and `sha256` are over the **raw** bytes, and the agent compares
the answer's digest with its own before naming the artifact: the store
compresses what it keeps, so a digest over gzip would depend on which
end compressed it.

| Refusal | When |
|---|---|
| 400 | a header missing or malformed (`content-length`, `x-3pm-ext` not a file extension, `x-3pm-instance` not a number) |
| 403 | the bearer is not the call's **target** node |
| 404 | no such call in this bucket, or it has already settled |
| 408 | the body was still arriving at the call's deadline, or sent no byte for 60 s |
| 409 | this instance already uploaded one, or is uploading one, or the call already holds `ARTIFACTS_PER_CALL_MAX` (16 — the action fan-out) |
| 413 | over the cap: the action's declared `output.max_bytes` (up to 256 MiB), `FILE_DEFAULT_MAX_BYTES` (**32 MiB**) when it declares none — or over the bucket's upload budget, `UPLOAD_BYTES_PER_BUCKET_MAX` (**2 GiB**, twice the artifacts retention keeps, `content-length` against the stored bytes and the uploads still arriving) |
| 422 | the body ended short of `content-length` |

Nothing is kept on a refusal. An upload refused, or a digest that
disagrees, fails **that outcome** (`failed` / `too_large`, with the
backend's reason) and nothing else of the reply; a backend that sent no
`fleet_token` fails it before a byte is read. The artifact is
**pending** until the reply claims it — the claim checks that it belongs
to this call and this instance and that its `bytes`/`sha256` match the
outcome's — and one still pending when the call settles is removed.

Every end that holds the outcome fetches the artifact the same way, with
its own bearer (`artifactFile`) — which is what lets one type serve both
directions.

### Calls with bytes attached

An `action` may carry bytes, and then they are **blobs on the backend**,
never frames on this link: `input? {blob, bytes?, sha256?, ext?, live}`
is what the handler must `GET`, `output? "<blob>"` the blob it must
`PUT` its file to, and `stream?` says both legs are open-ended instead
of sized files. A declared `bytes` is what makes a blob a file (verified
by the receiver); nothing declared is a live stream.

The two things a blob needs locally — a filesystem `path` and the token
that opens it — never cross, and that is **structural**: the link
declares its own input, which has no field for either, and the local
request is *built* from it with both left empty. An input that names no
blob does not exist here either: one without would read as a file of the
receiving host, which a link never describes.

`output` and `file` are not the same thing and do not replace each
other: `output` is a transfer the **caller** set up and consumes,
`file` is what the action returned without being asked, kept with the
call for whoever looks at it later.

### Shell sessions

`shell` (`3pm fleet ssh`) opens an interactive login shell on a pty of
the target, as the agent's own unix user (root where the agent runs as
root). The agent runs it itself; it never reaches the daemon. `term`,
`cols` and `rows` are the caller's terminal (`TERM`, size); `input` is
the keyboard leg the agent reads and `output` the screen leg it writes:
two blobs of the lossless `pipe` mode (framed like a live stream, with
back-pressure and one reader), carrying the messages of the local
`attach` protocol (`threepm_protocol::shared::attach`: input, resize,
ping up; output, exited, pong down). `from` is who placed the call — the
backend's `placed_by`, `node:<id>/-` or the account — and
`from_hostname` that node's hostname: the agent stamps them on every
audit line without asking the backend.

The call ends with `exited {code?, signal?}`: `code` when the shell
exited, `signal` when a signal killed it, neither when the agent could
not tell. Either leg ending ends the session (the shell gets `SIGHUP`,
then `SIGKILL` 5 s later). The agent runs the shell **beside** the
connection that received the call, not in it: a reconnect of the link
leaves the shell (and its legs, HTTP of their own) running, and its
`exited` and audit events go out on whichever connection is live when
they are sent — so a `reply` may arrive on a later connection than the
`call`.
A build that predates the op decodes it as `unknown` and answers `err
{forbidden, "unknown command"}`; a host whose `caps` leave out
`node:shell` answers `err {forbidden, "missing capability
node:shell"}`. Who may place it is the backend's call: the target's
`callers` rules must name the builtin action `3pm:shell` (app `-`) when
they are not empty — `*` never covers a `3pm:` action.

`exec` (`3pm fleet ssh <node> -- <cmd>`) runs one command instead, on
the same two legs, under the same capability, permits and hang-up
rules: the agent runs `<login shell> -c "<argv>"`, each word of `argv`
quoted for the shell when it needs to be and the words joined by spaces
— what ssh does with its own. With `pty` the command gets a pty of
`term`, `cols` × `rows` and the legs carry what a `shell`'s do. Without
(`pty` absent or false; `term`, `cols`, `rows` then unused), the command
gets three pipes: its stdout comes back as `output` messages, its stderr
as `stderr` (`0x1d`), and the caller's `eof` (`0x05`) closes its stdin —
bytes both ways, unchanged, under the legs' back-pressure. It ends like a
shell, with `exited`, after `exited` and `END` on the screen leg. A
separate op, not a field of `shell`: a build that predates it answers
`err {forbidden, "unknown command"}`, where one reading a `shell` with a
field it ignores would start an interactive shell and take a script's
stdin for keystrokes.

`attach` (`3pm fleet attach <node> <app>`) joins the terminal of a
`tty = true` app instead, on the same two legs, under the same
capability, permits and stop rules. The agent opens a **fresh**
connection to its daemon's `rpc.sock` (never one of its pooled
ones: the connection leaves NDJSON), sends the daemon's own `attach
{target, instance, read_only, mode, peek, cols, rows}` and, once it is
answered `attached`, relays the messages as they are: the keyboard
leg's to the daemon, the daemon's (`screen`, `diff`, `output`, `title`,
`bell`, `restarted`, `exited`, `lagged`, `detached`, …) to the screen
leg. The daemon's refusal (`not_found`, `not_tty`, `unavailable`,
`invalid_spec`) is the call's `err` (`not_found` stays `not_found`, the
rest `daemon_error`). The app's own `exited` ends nothing: the app may
restart. The session ends when the daemon says `detached` (forwarded,
then `END`), or when a leg ends or the agent stops: the agent sends the
daemon `detach` — the app runs on — and the caller gets `detached
{client}` and `END` when its screen leg still listens. Either way the
reply is `exited {}` (neither code nor signal). Audit: `open`, `attach`,
the raw input lines (`line {alt: true}`; the agent relays the screen
without reading it), `close`.

`run_as {user, req}` (`3pm fleet ssh -l <user>`) wraps a `shell` or an
`exec` (anything else: `malformed`): the same session, run as the unix
account `user` (`[a-z_][a-z0-9_-]{0,31}`, `threepm_link::valid_user_name`).
An envelope rather than a field, so an agent that predates it answers
`unknown command` instead of running the session as itself. An agent
running as root becomes the account (its groups, home, shell; the
daemon's hosted-user drop); one that is not root takes only its own
name (`forbidden` otherwise); an unknown account is `not_found`. Both
are the call's reply, before any leg opens. The agent's `open` line
names the account it ran as.

`forward {host, port, input, output, from, from_hostname?}` (`3pm
fleet ssh -L`, one call per connection the caller accepted; `-W`, one
call whose local side is the command line's stdin and stdout) opens one
TCP connection from the target to `host:port` — any destination, the
name resolved by the target — under its own capability `node:forward`
and the shells' cap of sessions in flight. The two `pipe` legs carry
**raw bytes** as `live` messages, no `attach` kind byte: `input` is
what the caller's peer writes, `output` what the destination writes
back, and a message flagged `END` (empty) half-closes its way — the
caller's `END` is a `shutdown(WR)` toward the destination, the
destination's EOF is an `END` on `output`. The agent connects first
(5 s): a failure is the call's `err {unavailable, "connect host:port: …"}`
and the legs never open. Then the reply is `exited {code}` once both
ways are closed, `0` when both ended by an `END`, `1` when a leg or the
connection broke; the agent stopping ends it with an `err`. Audit: the
backend's `open_requested` / `refused` (`forward = host:port`) and
`ended`, the agent's `close` with `bytes_in` / `bytes_out`. Who may
place it: the target's `callers`, when not empty, must name the builtin
action `3pm:forward` (app `-`); `3pm:shell` does not cover it.
`link.toml` version 4 reads a version-3 `caps` listing exactly the five
capabilities of that day as all of them, so a default link gains
`node:forward` on update; any other list is kept as written.

`-W host:port` is that same call, nothing added on the wire: stdin is
the `input` leg, the `output` leg goes to stdout (data only, every
message on stderr), exit 0 when both ways closed, 255 otherwise. It is
what ssh's `ProxyCommand` wants, so the real `ssh` reaches a node's
sshd through the relay:

```
Host *.3pm
  ProxyCommand 3pm fleet ssh $(echo %h | sed 's/\.3pm$//') -W localhost:%p
```

(`ssh envision@envision-surface-7caf.3pm`, `rsync -e ssh … web-1.3pm:`.)

`listen {bind, port, input, output, from, from_hostname?, e2e}` (`3pm
fleet ssh -R`, always sealed) binds `bind:port` on the target — `bind`
empty is its loopback, `port` 0 one it picks — under `node:forward`
and one of the sessions in flight. A non-loopback `bind` is refused
(`err {forbidden}`) unless the target's own `link.toml` has `[shell]
gateway_ports = true`, a setting the backend can neither read nor
change. The legs carry `threepm_link::listen` messages, one per live
message inside the sealed records: on `output` `bound {port}`, then
`accepted {conn, peer}` per connection (numbered from 1); on `input`
the caller's `keepalive` every 20 s, each answered, and `END` to close.
Each accepted connection is parked for 10 s, closed unless a `forward
{accepted: {listen, conn}, e2e}` (no `host`/`port`) placed by the same
`from` with the same sealed caller key takes it — once — and pumps it
as a dial. The listen ends `exited {code}` (0 on the caller's `END`, 1
on a cut leg or 60 s without a word) and closes the port and every
connection still parked; the agent stopping ends it with an `err`.
Audit: the backend's `open_requested` (`bind`, `port`) and `ended`,
each connection a forward's own lines naming the listen; the agent's
`close`.

#### The end-to-end channel (`e2e`)

`shell`, `exec`, `attach`, `forward` and `run_as` carry `e2e: bool`
(default `false`) and `record: bool` (default `false`); an agent that
can seal announces `shell_e2e: true` in its `hello`. With `e2e`, the
two legs carry, in place of their bytes, records `[u16 BE
length][Noise message]` of `Noise_XX_25519_ChaChaPoly_BLAKE2s` between
the caller's 3pm and the target's agent (at most 65 519 clear bytes per
record), whatever those bytes are (`attach` messages, or the raw bytes
of `cp` and `forward`). The prologue binds the handshake to the call id,
the target's node id **and the canonical request** — every field that
decides what runs (the operation and its argv, terminal and size, the
attach target and mode, the forward's destination or listen, the
`run_as` account, `record`), encoded the same way by both ends: a
backend that rewrote the clear `AgentRequest` it relays fails the
handshake before anything starts. Three messages open it, before any
session byte: `e` on `input`, `e, ee, s, es` on `output`, `s, se` on
`input` once the caller checked `s` against its pin (`known_nodes`,
TOFU). The agent then checks the caller's static key against **its own
`authorized_keys`** (no file = nobody): a key not listed is
`key_not_authorized` and nothing runs — this, not the backend, is what
admits a caller. It starts the shell, the command or the connection
only then. The agent's static key is `shell.key` in its state directory;
the caller's fingerprint comes back in `ShellEvent::Open {cli_key?}`
for the audit.

The handshake's payloads negotiate **compression** of the records: the
caller's `e` carries the words it asks, separated by spaces (today
`zstd`, or nothing), the agent's `e, ee, s, es` the words it grants (a
part of those, or nothing), `s, se` nothing. The agent grants `zstd`
only to an `exec` without `pty` (`fleet cp`, `fleet ssh -C -T`), never
to a pty, an `attach` or a `forward`/`listen`. Once granted, both legs'
records carry one frame each: a flag byte, `0` the bytes as they are,
`1` a zstd frame, of at most 65 518 clear bytes; any other flag, or a
frame that opens to more, is `e2e_broken`. Nothing of it is in
`AgentRequest` or the canonical request: the payloads are hashed into
the handshake, so a backend that strips or adds the ask breaks it like
any change to the prologue. An agent that predates it reads the ask and
ignores it, and answers no payload; a caller that predates it asks
nothing — either way the records are as before.

Refusals, as the call's `err`: a request without `e2e` to an agent of
this version is `e2e_required` (no setting relaxes it); a handshake or
a record that does not open is `e2e_broken`, and the session ends. The
backend keeps the agent's code in the stored answer (`result.code`, its
action code being `forbidden` / `failed`); the command line reads that
code, never the message. The backend itself refuses `e2e` to a node
whose live session did not announce `shell_e2e` (`409
ERR_NODE_NO_E2E`), and never tees a sealed session.

`record: true` asks the agent to record the session (the bucket's
`record_shells`): it writes the asciicast v2 of what it puts on
`output` **before** sealing (`threepm_protocol::cast`, the backend's own
rules: screen only, 64 MiB, never waiting), uploads it with
`putCallArtifact` before replying, and names it in `exited {code?,
signal?, recording?}`. A forward is never recorded, nor a copy: an
`exec` whose argv is `sh -c <one of 3pm fleet cp's four scripts, byte
for byte> 3pm-cp <args…>` (`threepm_link::cp::is_cp_transfer`, shared
by both sides) — the backend asks `record: false` for it, and the agent
records none even asked. An agent that dies
mid-session sends no recording; the backend's `ended` line says
`recording: lost`. A sealed session may take the LAN direct way even
when recorded.

### Calls this node places

`call_result {call, state, results, err?, output?}` reports that a call
**this** node placed on another node of the bucket (`io.call`, `3pm
fleet call`) has settled. `call` is the backend's call id, `state` its
word for the row (`replied`, `lost`, `unacked`, …) — free text, so a
state this build never heard of still reaches the app that is waiting —,
`results` one `Outcome` per instance the target reached — the same
objects the target's `reply` carried, `file` included —, `err` what the
backend sends instead of outcomes (refused by the target's caller list,
node offline, deadline passed), and `output` the file the call produced.

An artifact named here is fetched by **this** node: `artifactFile` lets
a node that is party to a call read it with its own bearer, which is why
the two directions share one type since link 5.

The backend sends `call_result` on a non-lossy path and **drops** it when
the session is gone rather than queueing it; `welcome.pending_calls` is
how the agent picks those up again.

### Bytes between LAN neighbours

When the backend judges the two ends of a call to be neighbours on one
LAN, it names the other end in **both** tickets: `action.peer` for the
target (on this link), the call view's `peer` for the caller (over
HTTP). `Peer {node_id, pubkey, addrs}`: the peer's node id, its
base64url Ed25519 public key — the only key its TLS handshake may
present (raw public keys, RFC 7250, mutual) — and at most
`PEER_ADDRS_MAX` (8) private socket addresses to try, in order. The
bytes may then go agent to agent over TCP + TLS; the blobs of the ticket
stay open on the backend as the fallback, and nothing about the call
itself changes.

The agent announces a port only, in `hello.direct_port`; the backend
picks the addresses from the node's `host` section (its private
interfaces), and only for two nodes behind the same public IP. That
section is the node's own report, so a hostile node can steer its
neighbours' connections at any private `ip:port` they can reach: a TCP
connect and a TLS hello go out, nothing more. What bounds it: a receiver
dials `Peer::dialable()` only (private addresses with a port,
deduplicated, at most 8, whatever arrived), and the dialled end must
present `peer.pubkey` before a byte of the blob is sent. A consequence
to know: the public key of a node becomes known to the nodes it calls or
is called by on its LAN. The peer never reaches the daemon.

The agent that **writes** such a blob sends `path_report {blob, path,
bytes}`, `path` ∈ `direct` | `relay` (unknown values decode to
`unknown`): `direct` **when the two meet**, before the transfer, `relay`
once the relay has taken it. `direct` lets the backend forget the blob it
had opened, unless its relay already carried a byte of it or has a
reader on it, or the call named no peer (then the report is refused);
`relay` only records the path. Only a call's **input** goes direct
today: its output is read after the reply, too late for any rendezvous.
While a live blob goes direct, `blob_stat {blob, bytes, last_ms}` every
5 s keeps the call's view moving.

## 6. Events

`AgentEvent` (tag `ev`): `process {id, name, kind, exit_code?,
exit_signal?}` — **no `stderr_tail`** (attacker-influenced plaintext in a
compressed stream) —, `daemon_shutdown {reason}` and `app {id, name,
instance, ts, source?, event_name, level, severity_number, attrs, exc?,
fingerprint?, repeat}`: an application event (`io.event`,
`notify_error`) exactly as the daemon admitted it — rate-limited per
fingerprint (`repeat` = how many it suppressed since the previous one
through), `cwd` and `$HOME` already stripped from `exc.stacktrace` by
the daemon (`docs/metrics.md`). `attrs`, `exc.message` and label values
are the app's own text: the SDK docs say no secret belongs there. Logs
never travel **on the link**: they have their own HTTP channel (§6b).

`action {id, name, instance, ts, source?, call, action, origin, ok,
err_code?, duration_ms}` is the audit line of one custom action on one
instance, whoever called it: `origin` is `local` (`3pm trigger` on the
machine) or `remote` (a `call` from this backend, `call` being the
daemon's own call id, not the link's). Neither the params nor the result
ride it, and neither do the daemon's local `placed_by` / `input_bytes`:
the backend wrote the call row itself and already holds both.

`shell {call, from, ts, event}` is the audit of a `shell` (or `exec`)
session this node serves: `call` is the link's call id of the command, `from`
the caller it named, `ts` the agent's clock (Unix ms), and `event`
(tag `what`) one of:

| `what` | Fields | When |
|---|---|---|
| `open` | `user`, `shell` | the shell started: its unix user and program |
| `line` | `line`, `alt` | a command line as the screen showed it, prompt included (`alt: false`), or the raw input line typed under a full-screen program (`alt: true`); `[hidden]` when the terminal did not echo; at most 4 KiB, marked when cut |
| `screen` | `alt`, `line` | a full-screen program took (`alt: true`) or gave back the terminal; `line` is the last line typed before |
| `close` | `code?`, `signal?`, `duration_ms`, `bytes_in`, `bytes_out`, `lines` | the session is over; `lines` counts the audit lines it sent, so a loss shows |
| `exec` | `command`, `pty` | an `exec` session, right after `open`: the command line the shell was given (at most 4 KiB, marked when cut). Without a pty nothing else is written but `close` — never its stdin; with one, `line` and `screen` follow as for a shell |
| `attach` | `app`, `instance`, `mode`, `read_only` | an `attach` session, right after `open`: the app as the daemon resolved it; then only raw input lines (`line {alt: true}`) and `close` |

A kind this build does not know decodes to `unknown` and the event
survives. The audit is as complete as the screen lets it be, no more
(`tmux`, an escape sequence, typing blind): what the agent cannot hide
is written by the backend itself — the opening and the end of the call.

## 6b. Logs — `POST /api/buckets/{cluster}/logs`

Not a frame: the WebSocket keeps one DEFLATE context for every frame, and
a log line is attacker-influenced plaintext (`sshd` writes the user names
a stranger types); compressed next to a secret, its size would leak the
secret. A node posts its logs as batches, `application/x-ndjson` gzip,
`Authorization: Bearer <fleet token>`, every
`LOG_PUSH_EVERY_MS` (10 s), one request in flight, at most
`LOG_BODY_MAX_GZ_BYTES` (1 MiB) compressed and `LOG_BODY_MAX_BYTES`
(4 MiB) inflated. Types: `threepm_link::logs`.

`welcome.logs {apps, host, files}` says which sources the backend takes;
absent, the agent ships and probes nothing. The host's `link.toml`
`[logs] {apps, host, files, host_level, exclude}` is the other half: a
source flows only when both say yes (`LogsSettings::effective`).

One record per line, tagged `k`, in this order: `batch {run, seq, ts}`
first (a retried batch repeats `run` and `seq`, and is stored once), then
every `src` and `tpl` before a record that names them.

| `k` | Fields | What |
|---|---|---|
| `batch` | `run`, `seq`, `ts` | who sends, which batch |
| `src` | `src`, `kind` (`app`\|`journal`\|`file`), `path?`, `app?`, `instance?`, `stream?`, `owner? {kind, name, target?, unit?, container?}`, `format?` | a source, before anything names it; again when it changes and hourly |
| `tpl` | `src`, `tpl`, `sev`, `text` | a template (variables masked), when it appears or its text changes |
| `cnt` | `src`, `from`, `to`, `n: [[tpl, lines, sev], …]`, `params? [{tpl, var, distinct, top}]` | counts per template over a window |
| `line` | `src`, `tpl?`, `ts`, `sev`, `why`, `line`, `ident?`, `unit?`, `pos?` | a raw line, shipped because it is news: `why` = `new`, `error`, `back`, `spike`, `return`, `context`; `pos` = its stable place in the source (`dev:ino:gen:offset` — `gen` bumps when the file is truncated or its inode reused; `dev:ino:offset` from older nodes, read as generation 0 —, the journal cursor, the bus sequence; opaque to the backend) |
| `dropped` | `src`, `lines`, `reason` (`budget`\|`lag`\|`ring`\|`stale`) | lines read and not shipped |

`sev`: `debug`, `info`, `notice`, `warning`, `err`, `crit` (syslog 0–2);
`warn`, `error`, `critical`, `alert`, `emerg` read as their canonical word.
The `n` triple reads its first three elements and skips the rest, so it
may grow at its tail. A `welcome.logs` switch added after the first three
defaults **off**.

Exactly once, as far as it is cheap: a batch retried within a run repeats
`(run, seq)` and is stored once; a `line` with a `pos` is stored once per
`(node, src, pos)`, so lines re-read after an agent restart are not
duplicated. Counts are deltas: the lines of the one batch in flight when
an agent stopped may be counted twice (at most one push period). A node
counts a line in the window of the moment it **read** it (a backlog read
after a stop lands in the current window, not its own). On a clean stop
the node closes its open windows early (`to` = the stop, at least 5 s
before the window's end) and tries one last batch for a few seconds; a
crash, or a backend away at the stop, loses the open windows' counts —
the lines themselves are read again, their positions not committed.
Every word a build does not know — a record kind, a source or owner kind,
a severity, a `why`, a reason — decodes to `unknown`, and the record
survives (an unknown record kind is skipped and counted). Bounds
(`LogRecord::check`, applied by the backend; the agent cuts first with
`cap_text`): a line 8 KiB, a template 2 KiB, any other string 1 KiB, a
`cnt` 4 096 triples and 1 024 params, `top` 3 values.

Answers: `200 {accepted, skipped, duplicate?, refused? {sources, templates, reason}}` (`refused`: new sources or templates the node's catalog did not take, `reason` `catalog_full`; also counted in `skipped`, the node counts them and does not send them again in a loop) (the node then saves its
offsets and journal cursor), `429` + `retry-after`, `413`, `400` (a body
that is not gzip or does not open with a valid `batch`: the batch is
dropped, never retried), `401` (wait for the next `welcome`), `503
ERR_STORE_DISABLED` / `ERR_STORE_UNAVAILABLE` (with `retry-after: 1` when
the backend has 32 batches in flight); a timeout or a 5xx retries the same
batch, 1 s doubling to 60 s; a `retry-after` only ever lengthens that
pause (the longer of the two, at most 60 s), never shortens it.

What the node guarantees, because VictoriaMetrics keeps one point per
series per 5 s (`-dedup.minScrapeInterval`): a window's counts for one
`(src, tpl)` travel in **one** batch (a source with more templates than a
`cnt` holds may use several `cnt` records, in the same batch: the backend
sums them, `tpl="other"` included, before writing), and two windows of
one source end at least 5 s apart. Template and source ids are never
reused for something else: a node whose miner state is lost starts new
source ids (its state epoch is part of them), so `(src, tpl)` always names
one template.

## 7. The node's state (`state`)

**The one diff engine of the link** (`threepm-link/src/node_state/`): the
agent gets **complete states** from its daemon — which cuts no diff —,
keeps what the backend holds, and sends what moved; the backend rebuilds
the state with one reader.

```
state {
  ts,            // agent clock, Unix ms: the reader derives uptimes from it
  gen,           // the agent's incarnation (its start time, ms)
  secs: [{       // every section that moved this tick; none = the heartbeat
    sec,         // "host" · "procs" · "app/<id>" · "tree/<id>" · "traffic"
    rev,         // the section's revision after p; p was cut from rev - 1
    full?,       // p is the whole section
    crc?,        // CRC-32 of serde_json::to_vec(section after p): every full one, and one a minute, each section at its own moment of it
    gone?,       // the section is gone: forget it, no p
    p            // JSON Merge Patch (RFC 7386) over the section
  }]
}
state_resync { secs }   // S→A: send these in full next; none named = every section
```

One frame every `pace.state_s`, empty when nothing moved: the heartbeat,
whose clock the uptimes are read on. `pace.metrics_s` paces the `app/*`
sections, `pace.trees` the `tree/*` ones.

A node's state is a set of **sections**, each an independent JSON
document with its own revision — not one document: a gap resends one
section, a section's full state fits one frame (nothing is ever
chunked), and sections do not move at the same pace. One frame a tick
carries them all; several frames a tick only when the 1 MiB cap forces
it, a section never split; one over the cap alone is not sent, warned
once, and cut again next tick. A name the reader does not know is held
all the same: it needs no schema.

| Section | Holds | Shape |
|---|---|---|
| `host` | `HostMetrics`; `gone` while the host agent is down | `null` members dropped; arrays of rows as tables `{"~r": {key: row}, "~o"?: [keys]}` keyed by a fixed path table (`networks`→`name`, `disks`→`mount_point`, `block_devices`→`name`, `sensors`→`label`, `raid`→`name`, `batteries`→`name`, `top_processes`→`pid`, `3pmDaemon.processes`→`role`, anything else by index; a key that is a non-negative whole number is written zero-padded to 10 digits, so rows sorted by it are sorted as text and need no order; `#2` for a key taken twice; the order kept only when the rows are not sorted). 3 significant digits |
| `procs` | the table of `RedactedRecord`s and `daemon` (`{version, protocol, pid}` from the daemon's `Pong`, absent while it is down) | `{daemon, procs: table keyed by id}`, the daemon's order. **No `uptime_ms`**: `started_at_epoch_ms` instead, worked out by the agent once per `(id, gen, pid)` — `created_at_epoch_ms` is the row's creation, not the spawn — and the reader's `uptime_ms` is the frame's `ts` minus it. **No `app_metrics_rev`** either (it moves with every flush; `app/<id>.rev` has it): it reads as 0. 3 significant digits |
| `app/<id>` | one instance's application metrics, once its registry ever moved (series, or a source that only declares actions); `gone` when the instance leaves the table | `{id, name, instance, ns?, gen, rev, src: [{s, pid?, a: [{n, d?, t?, o?}]}], m: {sid: {def: {i, n, k, u?, d?, s?, l}, v \| h: {s, count, sum, z, b: {index: count}}}}}`. An action's `t` is its declared timeout (ms) and `o` its declared `output.max_bytes`, what the backend reserves for a caller that asks for its file (absent: 256 MiB; the backend bounds it again). The application's values are **never rounded**; a value JSON cannot write (NaN, ±∞) reads as 0. `sid`s are the daemon's, unique per instance and incarnation |
| `tree/<id>` | one instance's `RedactedTree`, **only while it has more than one process** (back to one, or trees off: `gone`) | `{id, gen, rev, c, m, omitted?, procs: table keyed by pid}`, a parent before its children; a member is `{p, pp, n, c, m, t, d?, x?, nt?}` (pid, parent, executable name ≤ 64 bytes, cpu, RSS, start, detached, Extended Monitoring's counters, network traffic as `RedactedNet`). 3 significant digits |
| `traffic` | the agent's link counters since its start (`since_ms`, its `gen`'s start): bytes and messages per direction (`up` agent → backend, `down` backend → agent) and per kind, every 5 s (`max(1, 5 / state_s)` ticks). A name only (`node_state::TRAFFIC`), **outside `Section`**: a backend that does not know it holds it unread | `{since_ms, up?: {kind: {b, m}}, down?: {kind: {b, m}}}`, a counter still at zero left out, so a patch carries only the counters that moved. Read leniently (`traffic_view`, never the strict derive): an entry that is not `{b?, m?}` of whole non-negative numbers is skipped, unknown kinds kept up to 64 a direction; a store writes `Traffic::known()` only (the closed list × `up`/`down`). Kinds (`threepm_link::traffic::KINDS`, a closed list): `host` `extended` `top` `procs` `app` `trees` `events` `shell` `calls` `control` `traffic` `logs` `artifacts` `blobs` `api` |

```
RedactedRecord { id, name, instance, namespace, status, pid, cpu_percent: Option<f32> /* None under 2 s of uptime */,
  mem_bytes, uptime_ms, restarts, unstable_restarts, listen_ports, exit_code, exit_signal,
  created_at_epoch_ms, next_cron_epoch_ms, waiting_for, module: Option<String>,
  interpreter: Option<String> /* basename */, watch: bool, env_keys: Vec<String> /* empty with --no-env-keys */,
  app_series /* series the instance declares */, app_metrics_rev /* its registry's revision */,
  gen /* the instance's incarnation */,
  proc_count /* live processes of the instance's tree */,
  pss: Option<u64> /* Extended Monitoring, Linux */,
  net: Option<RedactedNet> /* the tree's network traffic */ }
RedactedNet { r, t /* bytes/s received, sent over the daemon's last 5 s window, whole bytes */,
  rb, tb /* bytes since the daemon first measured the incarnation (or the process) */,
  pa /* the window lost notifications: a lower bound; elided when false */ }
```

`cpu_percent`, `mem_bytes` and `listen_ports` are the totals of the
instance's **process tree** (`docs/metrics.md` → *Process trees*);
`proc_count` is the number of live members, the fleet table's `+N`.
`net` is the tree's traffic, each socket once (`docs/metrics.md` →
*Network traffic per app*); absent on a node with no source, before the
daemon's second window, and from an older agent — `rb`/`tb` restart
with the incarnation and with a daemon update, which a reader takes as
a counter reset. A tree member's `nt` counts every socket it holds.
`app_series`, `gen` and `proc_count` are elided at zero. The record is
what a `reply` carries; the `procs` **section** above reshapes it, which
is where `app_metrics_rev` is dropped and `uptime_ms` becomes
`started_at_epoch_ms`.

**Link traffic** (`traffic`, `threepm_link::traffic`). Up, a WS frame counts its compressed payload (what `rx_bytes` counts) and one message; a `state` frame's bytes are split over its patches **pro rata** of their JSON length (`pro_rata`, integer shares, the envelope and the rounding to `control`, the shares summing to the frame's size) — `host.extended`, `host.top_processes` and each `tree/<id>` member's `x` (`procs["~r"].*.x`) carved out to `extended` / `top` (`patch_weights`) — and each patch counts one message for its kind; an envelope with no patch (the heartbeat) is one `control` message. The total per frame is exact, the split an **estimate** (a section the DEFLATE window already holds costs almost nothing yet is charged its raw size). HTTP counts the body only, one message per request, retries included. A new `gen` starts the counters over; a reconnect keeps them.

**What the agent asks the daemon**, every tick: `list` and
`host_metrics`; `app_metrics` — `id` alone, always answered whole —
for the instances whose `(gen, app_metrics_rev)` is not the
one the backend's section was shaped from; `process_trees` for those
with more than one process whose `(gen, tree_rev)` moved, and for all of
them when the figures are due. The daemon's counters only say *whether*
to ask.

**Delivery**, per section: a patch applies only under the `gen` it was
cut in and onto `rev - 1`; anything else is a gap (`StateGap::{Gen, Rev,
Crc}`), the section is left untouched, the rest of the frame applies,
and the backend answers `state_resync` naming it (at most once a
second). **Only the last patch of each section in a frame applies**
(`StateFrame::last_of_each`): an honest agent cuts a section once a
tick, but may say `gone` for it in the same frame (an instance cut, then
deleted), and the last word holds. A frame that repeats a section costs
one apply of it, never one per copy; a later copy whose base was an
earlier one is a gap, answered as any other. The agent's writer advances **only once the frame is written on
the socket**: a patch cut and never written is cut again, larger, at the
same revision. Everything is owed in full at the first tick of a
session, on `state_resync`, and when the daemon was replaced (another
`Pong.boot_id`: its revision lines start over). Nothing is said `gone`
while the daemon itself is down: an empty table then says nothing of the
instances — and the backend ages no instance out on a table that names
no daemon, so one whose section never moves is still there when the
daemon answers again. An agent that **restarted** (another `gen`) owes
every section it has in full and never says `gone` for what went away
while it was down: the backend forgets, on the first frame of the new
`gen`, every section that frame did not bring again.

A **full** patch with a wrong CRC is accepted and turns the check off
until the node's next session — refusing it would ask for another, for
ever. It is not silent: the reader says so (`crc_failed_on_full`), and
the backend logs it once per session with the section and counts it
(`GET /api/v1/info` → `link.crc_full_mismatch`). The CRC rides one patch
a minute per section, each section at its own moment of the minute, so
the sections that move every tick are not all checked on the same one. A
section that comes in full is no longer asked, even if a gap noted it a
moment before. The backend bounds what it holds: 404 sections a node,
4 MiB a section, 32 MiB a node; a section over a size bound is dropped,
and whole again at its next patch; a 405th is refused until one held
goes.

**Never on the wire:** `ProcessRecord`, `StartSpec`, environment values,
`script`, `cwd`, log paths, `stderr_tail`, `last_error`, log lines,
process arguments. A property test draws secrets from an alphabet
disjoint from every other field and asserts none reaches the JSON.
`RedactedRecord.name` and `RedactedProc.n` are text the app controls
(the ecosystem name; `process.title`, `PR_SET_NAME`): display only.

### Volume

Measured on two real 10-minute traces, payload bytes on the link
(`rd/old/bench/host-patch/results/`), host + table + application metrics
+ trees, everything included: **357 B/s** on a dev Mac with 11 apps,
**463 B/s** on a Linux fleet server with 8 apps. Agent ≈ 0.27 ms a tick,
reader ≈ 0.02 ms. At the most a node manages — 20 instances at the
1 000-series cap, every series moving every second — the daemon hands
over 3.5 MB/s of whole states (0.28 ms an answer), the agent spends
≈ 60–70 ms a tick cutting them and the link carries 176 KB/s; 20
instances × 50 series: 3 ms a tick, 8 KB/s.

## 8. Framing, keepalive, close

| Direction | WS type | Encoding |
|---|---|---|
| agent → server (everything, `hello` included) | Binary | raw DEFLATE stream, one `Compress` per connection, `FlushCompress::Sync` after every message; the server keeps one `Decompress`, caps at 1 MiB after inflation ⇒ Close 1009 |
| server → agent | Text | clear JSON ≤ 1 MiB |

Ping every 20 s from both sides; 60 s without any frame (pongs count) ⇒
the connection is dead. The agent's writer flushes every 5 s even when
idle (pongs the stack queued while reading would otherwise sleep in the
sink). A full write buffer (32 KiB growing to 1 MiB) means the peer
stopped reading: dead connection, not growing memory. Close: the agent
sends `Close(Normal)` on shutdown and waits 2 s for the echo.

### Retry classes (agent)

| Close / HTTP | Class (base, cap) |
|---|---|
| 1000/1001/1009/1011, TCP/TLS/DNS, EOF, 60 s silence, HTTP 5xx (503 `ERR_LINK_BUSY`) | Transient (1 s, 60 s) |
| 4400 / HTTP 400 `ERR_BAD_REQUEST` | Auth (5 min, 30 min): a malformed frame is a bug, not a blip |
| 4401 / HTTP 401 | Auth (5 min, 30 min) |
| 4403 / HTTP 403 `ERR_NODE_REVOKED` | **Stop**: logged once, probe every hour |
| 4403 / HTTP 403 `ERR_NODE_PENDING` | Pending (1 s, 5 s): an approval is seen within 5 s |
| 4403 with no reason (a proxy stripped it) | Auth — revoked *or* pending, so never Stop on a bare code |
| 4409 `ERR_SESSION_REPLACED` | Replaced (30 s, 60 s, floor 30 s) |
| 4426 / HTTP 426 | Auth, log "update 3pm" |
| 4429 / HTTP 429 | RateLimited (1 min, 15 min) |
| skew > 5 min | Auth, `ERR_CLOCK_SKEW` with the delta |

Full jitter: uniform in `[floor, min(cap, base·2^attempt)]`; `attempt`
resets after a session of at least 60 s. The `ERR_*` carried in a close
frame's reason (or in a preceding `error` message) wins over the bare
code; a bare code maps back through `ErrorCode::from_close_code`, except
4403, which names no code (revoked or pending: bare, it is Auth).

## 9. Error codes

| Code | HTTP | Close | Meaning |
|---|---|---|---|
| `ERR_BAD_REQUEST` | 400 | 4400 | malformed message, missing field, wrong sub-protocol |
| `ERR_LINK_VERSION` | 426 | 4426 | link version outside the accepted range |
| `ERR_TOKEN_INVALID` | 401 | 4401 | unknown token, hash mismatch, wrong cluster, revoked — one code, no oracle |
| `ERR_TOKEN_EXPIRED` | 401 | 4401 | |
| `ERR_TOKEN_EXHAUSTED` | 401 | 4401 | no use left |
| `ERR_NONCE_INVALID` | 401 | 4401 | unknown, reused or stale handshake nonce |
| `ERR_PROOF_INVALID` | 401 | 4401 | `hello.sig` does not verify |
| `ERR_CLOCK_SKEW` | 401 | 4401 | `server_time_ms` travels with it |
| `ERR_RATE_LIMITED` | 429 | 4429 | |
| `ERR_NODE_UNKNOWN` | 401 | 4401 | no such node in that cluster: only enrolling builds the link again |
| `ERR_NODE_REVOKED` | 403 | 4403 | the agent stops retrying |
| `ERR_NODE_PENDING` | 403 | 4403 | awaiting approval |
| `ERR_HANDSHAKE_TIMEOUT` | 408 | 4408 | |
| `ERR_LINK_BUSY` | 503 | — | the link upgrade found the pending handshakes full (per client address, or the nonce store); newer than some agents, which decode it to `ERR_UNKNOWN` and retry Transient on the 503 |
| `ERR_SESSION_REPLACED` | 409 | 4409 | |
| `ERR_INTERNAL` | 500 | 1011 | |

Close codes are `4000 + HTTP status` (mnemonic); `ERR_INTERNAL` closes
with 1011. Unknown codes decode to `ERR_UNKNOWN` (500 / 1011). This is
the **wire's** vocabulary: a refusal an agent reaches on its own — a
clear-text URL to a non-loopback host without `--insecure` — is its own
code string in its own `--json` output, not one of these.

## 10. Fixtures

`crates/threepm-link/tests/fixtures/link/v5/*.json` freeze one sample
per family: `server_frame`, `agent_frame`, `error_code`, `capability`,
`node_state`, `agent_request`, `agent_response`, `outcome`,
`artifact_ref`, `agent_event`, `redacted_record`, `redacted_tree`,
`state_frame`, `action_origin`, `path`, `pace`, `enroll_request`,
`enroll_response`, `blob_id`, `call_id`, `artifact_id`, `sha256`, `log_record`,
`logs_accepted`, `traffic` (and a `traffic` patch in `state_frame`). The `link_fixtures` test fails when a
frozen sample no longer round-trips, when a frame tag of either
direction has no sample, or when a fixture file is missing — it guards
shape, not values. A companion property walks each server frame as JSON,
corrupts every typed identifier it carries in turn, and asserts the
frame decodes to `Invalid` naming its call, never to an error.

Regenerate deliberately, and review the diff as a compatibility
decision:

```
UPDATE_LINK_FIXTURES=1 cargo test -p threepm-link --test link_fixtures -- --ignored update_fixtures
```

</document>

<document url="https://3pm.edgecr4ft.net/docs/protocol.md" title="Daemon wire protocol">

# 3pm wire protocol

The contract between the `3pm` CLI (or any embedder of `threepm-protocol`,
or the Node.js client in `packages/node`) and the daemon. Source of truth for the shapes is
`crates/threepm-protocol/src/{message,control,filter}.rs` and
`src/shared/` (`actions`, `calls`, `live`, `attach`, `host`, `app_metrics`, `tree` —
what the fleet link and the backend read); the frozen JSON of
every message for the current protocol generation lives in
`crates/threepm-protocol/tests/fixtures/wire/v<N>/` and is enforced by
`tests/wire_fixtures.rs`. This page explains the rules; the fixtures are
the byte-level reference.

## 1. Transport

Three Unix domain sockets under `$THREEPM_HOME` (default `~/.3pm`), all
created `0600`/`0700` and **peer-uid checked**: the daemon calls
`SO_PEERCRED`/`LOCAL_PEERCRED` on every accepted connection and drops any
whose uid is not the one that listener serves (`bootstrap::peer_uid_is`).
On its own sockets that uid is the daemon's effective uid; a root daemon
also listens for the users it hosts (*Hosted users* below), one set of
sockets each, and every accepted connection therefore carries an
identity.

| Socket | Direction | Framing | Purpose |
|---|---|---|---|
| `rpc.sock` | client → daemon, one `Request` ⇒ exactly one `Response` | NDJSON | Commands (`ls`, `start`, …). A connection may carry many sequential request/response pairs. |
| `rpc.sock` after `attached` | both directions | live framing | An `attach` answered `attached` takes the connection over: from the next byte on, both sides speak the live framing (§ *Attach*). The only request that does so (`Request::takes_connection`). |
| `pub.sock` | daemon → subscriber, fire-and-forget | NDJSON | The event bus (`Event`): log lines, lifecycle transitions, rotations. A slow subscriber receives `lagged` instead of blocking the daemon. |
| `ctl.sock` | supervised child ↔ daemon | NDJSON | Optional control channel (`ControlIn`/`ControlOut`): readiness, graceful shutdown. Gen-pinned (see §4). |
| `host.sock` | daemon → host metrics agent, one `Request` ⇒ one `Response` | NDJSON | Internal: the same `Request`/`Response` types, restricted to `ping` and `host_metrics` (anything else is `err{unknown_request}`). Version-checked with the same `pong` handshake. |

**NDJSON**: one JSON object per `\n`-terminated line, UTF-8, no framing
bytes. **Line cap**: a line longer than `MAX_LINE_BYTES` (1 MiB, newline
excluded) is a `CodecError::LineTooLong` and the connection is closed —
never buffered further. What a client reads back on `rpc.sock` is held
to `MAX_REPLY_BYTES` (32 MiB) instead (`read_msg_capped`; the Node
client's `RpcConnection` mirrors it): a `list` reply carries every row
(~1.5 KB each), which the 1 MiB cap would stop at about 700 apps.
Requests, control messages and the event stream keep 1 MiB.
`LineTooLong` names the cap it hit. A malformed line is a `Malformed` error on the
reader; on `rpc.sock` the daemon answers `{"res":"err","code":
"unknown_request",…}` for JSON it cannot map to a `Request` rather than
hanging up.

Liveness is always "connect to `rpc.sock` and get a `pong`"; pidfiles are
never consulted (`client::connect`). Daemon auto-spawn and the readiness
byte on fd 3 are described in OV `3pm-docs-archived/architecture` §4.

### Hosted users (one root daemon, per-user sockets)

A **root** daemon serves, besides its own sockets, one `rpc`/`pub`/`ctl`
set per user declared in `[users] allow` of its `config.toml`. They live
in a root-owned home under `/var/lib/3pm/users/<name>/` (`[users] dir`
moves the parent), the standard `Paths` layout plus a `hosted` marker
file; the home and its `logs/`, `pids/`, `actions/` are `root:<gid> 0750`
and only the three sockets are chowned to the user (`0600`). A daemon
that is not root hosts nobody and ignores `[users]`.

Each listener is peer-uid checked against **that** user's uid, so the
daemon knows who is calling: `Caller::Root` on its own sockets,
`Caller::Hosted(<name>)` on a user's. The caller decides four things:

| What it decides | For a hosted connection |
|---|---|
| which requests are served | the app-level ones. The daemon-level commands (`save`, `resurrect`, `kill_daemon`, `internals`, `orphans`, `handoff`) answer `err{forbidden}`. `host_metrics` is served — the figures are the machine's — minus `top_processes` and `sessions`, which name the other accounts |
| which rows it resolves | only rows whose `spec.owner` is its own name — in `list`, `describe`, every `Target` (`all` included) and every log operation. A row of another owner answers `err{not_found}`, never `forbidden`: the reply is not an existence oracle. `flush_logs` with a null target spares the daemon's `3pm.log`, and `reload_logs` reopens only its own files |
| which events it receives | the events of its own rows, plus the daemon-level ones (`lagged`, `daemon_shutdown`, `daemon_handoff`). The `EventFilter` handshake is unchanged — it narrows, never widens |
| what a `ctl.sock` hello attaches to | a row whose `spec.user` is that user (and whose `gen` is current, as always) |

What it does **not** decide is the fleet. A hosted account places `call`s
on other nodes exactly as root does — same refusals, same codes, same
words — and there is no setting to turn that on: the right comes with
being hosted. Its bytes travel on the data socket the agent binds for its
uid (§ *Calls*, `hosted_socks`). Being hosted still narrows the call in
the two ways above the table: the rows it may target locally, and — for
an input it names by `path` — the files of its own uid.

A hosted listener serves at most **64 connections at once** (root's
descriptor budget is not a hosted user's to spend); further peers are
hung up on and the refusal is logged. A client in a hosted home never
spawns a daemon — the root one owns that home — and an in-place handoff
(`3pm update`) does **not** carry the hosted listeners across the exec:
the new image rebinds them, so a hosted client sees a short gap, waits up
to 5 s for the socket to answer again, and reconnects.

## 2. Versioning and compatibility

Two numbers travel in `pong`:

| Field | Meaning | Checked? |
|---|---|---|
| `version` | `CARGO_PKG_VERSION` of the daemon build | No — informational, shown in skew messages and `3pm update` |
| `protocol` | `PROTOCOL_VERSION` (`threepm_protocol::PROTOCOL_VERSION`, currently **2**) | **Yes** — `connect()` accepts iff it equals the client's |

A `pong` without `protocol` (a daemon predating the field) decodes as `0`,
which never matches: the user is told to run `3pm update` (save → kill →
respawn → resurrect). Only `3pm kill` and `3pm update` use
`connect_unchecked()`, since they must talk to the outdated daemon.
`pong` itself never moves between generations — it is the message that
announces which one is spoken.

`pub.sock` is not version-checked: events decode leniently, so a skewed
daemon degrades to "some events invisible", and every CLI path that
subscribes has already done a checked `connect()`. Its only handshake is
optional and one-way — see *Subscription filter* below.

### Growth rules (what does NOT bump `PROTOCOL_VERSION`)

Compatibility is **additive-only**. Within one protocol generation you may:

- add a `Request` variant — an older daemon answers
  `err{code:"unknown_request"}` (never drops the connection);
- add a `Response` variant — an older client decodes it as
  `Response::Unknown` via `#[serde(other)]`;
- add an `ErrCode` — older clients map it to `ErrCode::Unknown`;
- add an `Event` / `ControlIn` / `ControlOut` kind — the other side skips
  it (`decode_lenient`);
- add a field to any struct or struct-variant, provided it is
  `#[serde(default)]` (and, if it should stay terse on the wire,
  `skip_serializing_if`). `DaemonInternals`, `HostMetrics` and every
  `Host*` struct are container-level `#[serde(default)]`: they decode from
  `{}`.

You must **bump `PROTOCOL_VERSION`** (and start a new
`fixtures/wire/v<N>/` directory) when you:

- remove or rename a variant, a field, or a tag value;
- change a field's type or meaning;
- make a previously optional field required;
- add a value to an enum carried **inside** a field when that enum has no
  `#[serde(other)]` catch-all. These are closed today: `Status` and
  `ProcessEventKind` (`threepm-core/src/process.rs`), `LimitScope`
  (`threepm-core/src/spec/limit_scope.rs`), `LogStream`. An older peer
  does not skip the unknown value, it fails the whole message: one new
  status in a `ProcessRecord` breaks an old CLI's `list`. (The fleet link
  solved this with lenient mirrors, `threepm-link/src/wire/lifecycle.rs`;
  the local wire has not.)

The fixture tests turn this into a CI gate: every frozen line must still
decode, and every key a frozen line carried must still be emitted by the
current samples. Regenerate after an additive change with
`UPDATE_WIRE_FIXTURES=1 cargo test -p threepm-protocol --test
wire_fixtures -- --ignored update_fixtures` and review the diff as
additions only. The generator writes `v<PROTOCOL_VERSION>/` and nothing
else, so `update/` (below) can never be regenerated by accident.

### 2.1 The frozen `update/` subset

`3pm update` is the one path that talks to a daemon of **another**
generation: the installed CLI asks the running daemon to hand off to the
new binary, then keeps talking to the image that comes back. It works
because none of its connections is version-checked —
`connect_identity()` for `update`, `connect_unchecked()` for `kill`, and
`probe()` inside `wait_daemon_back` — and because the handoff manifest
(`HANDOFF_VERSION`) carries no `Request`, `Response` or `Event`.
So a daemon of one generation hands off to a binary of the next, and
back again for a rollback, without restarting a single app. The
manifest's own version is 2 only when a row has `tty` (running or
stopped — the spec decides, so the pre-check's draft and the final
manifest agree; a running one's stream is a pty master, which an older
image would close); every other manifest is still written as 1, which
every image reads. An image
that cannot read a version-2 manifest refuses it at the pre-check, and
`3pm update` falls back to a cold restart.

That makes seven requests and six responses a **permanent** contract,
frozen in `tests/fixtures/wire/update/` and never regenerated:

| Request | Response read | What the update path reads out of it |
|---|---|---|
| `ping` | `pong` | `boot_id` (is this the successor image?), `protocol`, `version` |
| `list` | `process_list` | `processes[].name`, `.instance`, `.pid` — the kept-vs-restarted verdict, nothing else |
| `internals` | `internals` | `last_handoff` (`{outcome, error}`): "the new image adopted" vs. "the old one resumed" |
| `handoff` | `ok` / `err` | the `res` tag, `code` (`unknown_request`, `handoff_in_progress`, `handoff_refused`), `message` |
| `save` | `saved` | `count`, read leniently: the old daemon may predate this CLI |
| `kill_daemon` | `ok` | `affected: ProcessRecord[]` — the cold path's cross-generation decode |
| `resurrect` | `ok` | the same shape, against the freshly spawned daemon |

`ProcessRecord` is therefore load-bearing across the boundary, and so is
`DaemonInternals` — the second one silently: both are container-level
`#[serde(default)]`, so a generation that regrouped their fields would
not error, it would answer `mode: "recovered"` on the very update that
proves the handoff worked. That is why neither is restructured, and why
these thirteen messages are pinned by bytes rather than by taste.

Two tests hold it, and they hold different halves.
`wire_fixtures.rs::the_update_subset_is_frozen` pins the **bytes**: the
tags of `update/` and the sha256 of each file, written in the test, so
editing the fixtures means editing the constant beside them.
`threepm-cli`'s `commands/update_frozen_tests.rs` pins the **meaning**:
today's `Request` writes those very requests, and the `expect_*` and
`match`es `3pm update` runs read those very responses. A generation that
breaks either one has to write the `serde_json::Value` shim the plan
describes before it ships — v2 needed none, because it changed none of
the thirteen documents.

## 3. Message catalogue

Tagging: `Request` uses `"cmd"`, `Response` uses `"res"`, `Event` uses
`"event"`, control messages use `"msg"`; all tags and enum values are
`snake_case`. `Target` is externally tagged: `{"id":3}`, `{"name":"api"}`
or the string `"all"`.

### Requests (`rpc.sock`, `cmd`)

| `cmd` | Fields | Reply |
|---|---|---|
| `ping` | — | `pong` |
| `start` | `spec: StartSpec`, `if_changed: bool` (apply semantics) | `ok` |
| `scale` | `name: string`, `instances: u32` | `ok` |
| `stop` | `target` | `ok` |
| `restart` | `target`, `toggle_watch: bool`, `patch?: SpecPatch` (elided when absent; folded into the matched apps' specs before they restart — pm2's attribute update, `patch.name` renames in place and is refused on a multi-app target) | `ok` |
| `rename` | `target`, `name: string` (the one app `target` matches is renamed in place, **without** a restart: same ids and pids, the pid files and the default log files follow; refused on a multi-app target, a module app, or a taken name) | `ok` |
| `delete` | `target`, `flush: bool` (also remove log files) | `ok` |
| `list` | — | `process_list` |
| `describe` | `target` | `process_list` (every instance of a name) |
| `save` | — | `saved` |
| `resurrect` | — | `ok` |
| `kill_daemon` | — | `ok` (reply waits for the children to die) |
| `internals` | — | `internals` |
| `host_metrics` | — | `host_metrics` (proxied from the host agent's cache; `err{unavailable}` when the agent is down) |
| `flush_logs` | `target?` (null = every app + `3pm.log`) | `logs_op` |
| `reload_logs` | — | `logs_op` |
| `reset` | `target` | `ok` (records with zeroed counters) |
| `send_signal` | `target`, `signal: string` (POSIX name), `group: bool` | `ok` |
| `rotate_logs` | `target?` (null = all) | `logs_op` |
| `orphans` | — | `orphans` (read-only scan of `pids/`: live processes of a previous daemon that no managed row owns) |
| `app_metrics` | `id` | `app_metrics` — the instance's application metrics, **always whole**: the daemon cuts no diff |
| `handoff` | `exe: string`, `argv: string[]` (default empty) | `ok` (the rows handed over) once the new binary passed its pre-check and the daemon is about to re-exec in place (`3pm update` without app restarts); the connection then EOFs and a later `pong` shows a new `boot_id`. `err{handoff_refused}` when the binary or the moment is unfit, `err{handoff_in_progress}` while one is already running |
| `process_trees` | `ids: u32[]` (default empty = every instance), `gen?: u64` | `process_trees` — the process tree of each instance as of the last metrics tick (`3pm ls --tree`, `3pm show`, the fleet agent). An unknown id is left out, and with `gen` so is an instance whose current incarnation differs, so a tree is never attached to a restarted row — however many ids were asked for: alone, that is an empty `trees`, never an error. A daemon predating trees answers `err{unknown_request}` |
| `call` | `node?: string` (a node id `3pmn_…` or a hostname unique in the bucket; absent = this host), `app: string`, `instance?: u32` (default 0; **required** once there is an `input` — a 100 MB file is not fanned out), `source?: string`, `action: string`, `params` (JSON, default `null`, ≤ 64 KiB serialised), `timeout_ms?: u64`, `input?: CallInput`, `output: bool` (ask for a result file), `stream: bool` | `called` — a `dispatch` with bytes attached, on this host or on another node of the bucket (`3pm call`, `3pm trigger --file`). `CallInput` is `{path?: string` (a file of this host, read by the daemon)`, bytes?: u64, sha256?: string, ext?: string, live: bool` (default false)`}`: `bytes` present means a file (checked by the receiver), absent means a live stream (framed, *Live framing* below). The declared size is bounded by the action's `input.max_bytes`, itself ≤ 8 GiB, and the call is refused before a byte travels when it exceeds it. **Served** (`supervisor/rpc.rs` dispatches `Command::Call` to `handle_call_request`; e2e in `crates/threepm-cli/tests/{e2e_call,e2e_call_refusals}.rs`). A daemon predating the request answers `err{unknown_request}` |
| `attach` | `target` (a name or an id; `all` is refused), `instance: u32` (default 0), `read_only: bool` (default false), `mode` (`rendered` default, `raw`; unknown → `unknown`, refused), `peek: bool` (default false), `cols: u16`, `rows: u16` (default 0 = no size) | `attached`, then the connection switches to the live framing (§ *Attach*). `err{not_found}`, `err{not_tty}`, `err{unavailable}`, `err{invalid_spec}`. A daemon predating it answers `err{unknown_request}` |
| `dispatch` | `target` (a name or an id; `all` is refused), `instance?: u32`, `source?: string`, `action: string`, `params` (JSON, default `null`, ≤ 64 KiB serialised), `timeout_ms?: u64`, `remote: bool`, `input?: ActionInput`, `output?: LocalTicket`, `stream: bool` | `called` — run a custom action the app declared on its control channel, with its tickets **already prepared** (the fleet agent, `3pm trigger`, `3pm profile`). That is the whole difference from `call`: `call` names one instance and asks the daemon to prepare the bytes, `dispatch` fans a name out and carries the tickets the agent already minted (`input`, `output`, `stream`; a plain `3pm trigger` names none of them). A name without `instance` reaches every instance whose control connections declare the action (at most 16, else `err{invalid_spec}`); `source` picks the connection when several of one instance declare it (else `err{invalid_spec}` listing them). `timeout_ms` is bounded by the declaration's (default 30 s, ceiling 10 min). `remote` marks a call from the fleet backend: an action declared `remote: false` answers `refused`. `err{not_found}` when nothing declares the action (the message lists what is declared), `err{handoff_in_progress}` during a handoff. A daemon predating actions answers `err{unknown_request}` |

Any request may instead get `err`. While the daemon shuts down only
`ping`, `list` and `kill_daemon` are served; everything else is
`err{shutting_down}`.

**`start` and accounts.** Three `StartSpec` fields carry the hosted-user
model. `user` (optional) is the Unix account the app runs as: honored by a
root daemon, and only for a user it hosts — anything else is
`err{invalid_spec}`, never "runs as the daemon". `persist` (default
`true`, elided on the wire when true) is whether `dump.json` keeps the
app: `false` means it survives restarts and handoffs but is never
relaunched at boot. `owner` (optional) is **stamped by the daemon on every
entry path** — `start`, `if_changed`/apply, a patched `restart`, a rename,
`scale` — from the connection's identity: a hosted caller's own name,
absent for root. A client may send one; it is overwritten. A hosted
caller's spec is narrowed at the same time: `user` forced to its own
account, `source` and `module` cleared (they are provenance, and both
would put root to work on the user's behalf), `out_file`/`error_file`/
`log_file` refused, `watch` roots bounded and kept inside `cwd`, and at
most `[users] max_apps` rows per owner (default 32) — else
`err{forbidden}`. App names stay global: a name another owner holds is
`err{invalid_spec}` ("belongs to …"), for root as for a hosted user.

### Responses (`rpc.sock`, `res`)

| `res` | Fields |
|---|---|
| `pong` | `pid: u32`, `version: string`, `protocol: u32` (default 0), `boot_id: u64` (default 0) — a random id of the daemon *image*, redrawn when the daemon re-executes itself in place (`3pm update` handoff: same pid, new binary), so a client can tell "the daemon I was talking to" from "its successor" |
| `ok` | `affected: ProcessRecord[]` — the rows acted upon |
| `process_list` | `processes: ProcessRecord[]` |
| `saved` | `path: string`, `count: usize` |
| `internals` | `internals: DaemonInternals` |
| `host_metrics` | `host: HostMetrics` |
| `logs_op` | `touched: string[]` — files flushed / reopened / rotated |
| `orphans` | `orphans: OrphanRecord[]` — `{label, pid, pid_file}`, container-level `default`; `label` is the pid file's stem (`api`, `api-2`) |
| `app_metrics` | `id`, `gen`, `rev`, `sources: {source, pid?, connected_at_epoch_ms}[]`, `def: SeriesDef[]` (every series: `sid`, `name`, `kind`, `unit?`, `description?`, `source?`, `labels`), `val: {sid, value, updated_at_epoch_ms}[]` (every scalar), `hist: {sid, scale, count, sum, zero, buckets: [index, count][], updated_at_epoch_ms}[]` (every histogram, every bucket). `rev` moves only when something a reader sees moved (a batch that changes nothing is not a new revision): it is what `ProcessRecord.app_metrics_rev` reports, so a reader knows whether asking again is worth it |
| `process_trees` | `trees: AppTree[]`, one per instance answered, in id order. `AppTree`: `id`, `gen`, `rev` (the record's `tree_rev` at the tick), `cpu_percent` and `mem_bytes` (this tick's totals over every member, omitted ones included — the same moment and incarnation as `procs`, which a separate `list` may not be), `procs: TreeProc[]` (`procs[0]` is the spawned pid; a parent precedes its children, siblings by pid, members whose parent is not a member follow by pid; empty when the instance has at most one process or is not running), `omitted: u32` (elided at 0; members cut by the caps, their figures still in the totals). `TreeProc`: `pid`, `ppid` (as the kernel reports it, not necessarily a member), `name` (the name the OS gives the process now, read again every tick: `comm` on Linux — the executable's file name after an `exec`, or what `prctl(PR_SET_NAME)` / Node's `process.title` set since, 15 bytes —, the executable's file name on macOS — it follows an `exec` only —, the name first seen elsewhere; ≤ 64 bytes, control and bidi-control characters and the line/paragraph separators blanked to a space, private-use glyphs kept — a Nerd Font icon in a `process.title` —, never the arguments — text the app controls), `cpu_percent` (percent of one core, 0 on a member's first tick), `mem_bytes` (RSS), `started_at_epoch_ms` (second resolution; uptime and identity against pid reuse), `detached: bool` (elided when false; the process left the app's process group — `3pm stop` will not reach it; never on `procs[0]`), `listen_ports: u16[]` (elided when empty; at most 4, the record's `listen_ports` is the uncapped union), `ext` and `net` (elided when absent: Extended Monitoring's counters, the process's network traffic as `NetUsage` — `rx_bytes_per_s`, `tx_bytes_per_s`, `rx_bytes`, `tx_bytes`, `partial` elided when false; everything the process moved, its connections with the app's other members included, which the record's `net` leaves out; both shed from the reply's tail by `budget_extras` when it would overflow, `net` reserving 192 B a line). Caps: 256 lines per tree, 2 000 per reply shared as `min(256, 2000 / n)` per instance, `n` being the instances answered that list members (at least one line each; an instance alone or stopped takes no share), so a reply never drops an instance whole and stays under the line cap at the worst-case line size. Every struct is container-level `default` |
| `called` | `call_id?: string` (the id the fleet knows the call by; a `call` always carries one, a `dispatch` none — it mints one id per instance it reaches and no one of them is the answer's), `ok: bool`, `results: ActionOutcome[]` (default empty), `err?: string` and `code?: ActionErrCode` (present iff `!ok`, elided otherwise), `output?: {path, bytes, ext}` — the one answer both `call` and `dispatch` get. **The verdict is the call's, not a handler's**, and only the `call` leg gives one: a call that never reached a handler (`node_offline`, `too_large`, `refused`, `timeout`, `lost`, …) answers `ok: false` with empty `results` and a `code` from the same vocabulary `ActionOutcome.err.code` uses, so a caller reads one set of words whatever went wrong. A handler that ran and failed leaves `ok: true` here and says so in its own outcome. A `dispatch` that reaches handlers is `ok: true` with one outcome per instance, in id order; one refused as a whole is an `err{…}`, like anything that is not a call at all — an unknown app, a malformed request, a daemon shutting down. `output` is present when a `call` asked for one and the handler produced it; the file is already on this host, under `actions/`. `ActionOutcome`: `proc_id`, `name`, `instance` (default 0), `source` (the control connection that ran it), `action`, `duration_ms`, `ok`, and when `ok` `data?` (the handler's JSON, ≤ 512 KiB) and/or `file?: {path, bytes, ext}` — both together when the handler left a file and a small descriptive `data` — (an absolute path under `<home>/actions/<app>[-<n>]/` — an app name made only of dots is written `%2E` per dot —, `<epoch_ms>-<call>.<ext>` with `call` the call id, 16 lowercase hex digits, ≤ `ACTION_FILE_MAX_BYTES` — **8 GiB, the same ceiling as an input**, because nothing copies the file here: the daemon checks its size and hands the path over. Leaving the host is capped far lower and separately, by the fleet and not by this protocol: an action's declared `output.max_bytes` when the file crosses as a blob, and the backend's artifact caps — 32 MiB by default — when the agent uploads it. Retention: *Control channel* → *Files*); `output?: {blob, bytes, ext, sha256}` is the other road bytes come back by — the blob a handler of this host pushed for a caller that may be elsewhere, present on a `call` and never on a `dispatch`, and the one an outcome keeps when it is forwarded (a path belongs to the host that wrote it); when not `ok`, `err: {code, message}` with `code` one of `failed` (the handler threw — `message` is its text, cut to 1 KiB —, or the daemon could not prepare the call or accept its reply: then `message` starts with `call not prepared: ` (the instance directory could not be created) or `reply not accepted: ` (the announced file is missing or not a regular file — a link is refused —, or its extension is invalid)), `timeout`, `busy` (the action already runs on that source, the source runs `CALLS_PER_SOURCE_MAX` (16) calls, the daemon `CALLS_PER_DAEMON_MAX` (32) plain ones — sessions are counted apart, `STREAMS_PER_DAEMON_MAX` (16) —, or the connection's queue has no slot left but the one kept for `shutdown` — retryable), `gone` (the connection closed, the source reconnected, the process exited, the daemon handed off, or the call ended without an outcome, before a reply), `refused` (a remote call on a `remote: false` action), `too_large` (`data` or the file over its cap); a `call` adds seven more, listed under *Calls* below; unknown codes decode as `unknown` |
| `err` | `code: ErrCode`, `message: string` |
| `attached` | `id`, `name`, `instance` (default 0), `pid?` (absent when the app is not running: the client sees its last screen and waits), `gen`, `alive: bool`, `cols: u16`, `rows: u16` (the pty's size), `viewers: u32` (this client included, default 0), `bracketed_paste: bool` (the app enabled it; default false) — the answer to `attach`; the live framing starts on the next byte |
| *(unknown)* | decoded as `Response::Unknown`; never sent by the daemon |

`ErrCode`: `not_found`, `invalid_spec`, `spawn_failed`, `unknown_request`,
`shutting_down`, `internal`, `unavailable` (a delegate the daemon
depends on — the host metrics agent — is unreachable or has no sample
yet; retryable), `handoff_in_progress` (the daemon is handing off to a
new image — `3pm update` without app restarts — and refuses mutations
for a moment; reads pass; retry once a `pong` shows a new `boot_id`),
`handoff_refused` (the daemon declined to hand off; the message says
why), `forbidden` (the caller may not do this: a hosted user's connection
asked for a daemon-level command — see *Hosted users* — or for an
account it may not run as), `not_tty` (an `attach` to an app that does
not run on a terminal); anything else decodes as `Unknown`.

`HostMetrics.collected_at_epoch_ms` (optional) stamps when the agent took
the snapshot's fast tier; absent from a daemon predating the agent.
`rate_window_ms` and the per-NIC / per-mount `*_per_s` rates (all
optional) are derived by the agent from two consecutive fast samples.
`sampled_at_epoch_ms` (optional object of optional `fast` / `medium` /
`slow` / `extended` / `top` stamps) tells how old each tier's data is; absent from a daemon
predating tiered sampling. `3pmDaemon` (optional; the one camelCase key,
a Rust field cannot start with a digit) is 3pm's own footprint as the
agent measured it: `at_epoch_ms`, `processes[]` (`role`, `pid`,
`cpu_percent`, `mem_bytes`, optional `uptime_ms` / `open_fds` /
`threads`) and an optional `runtime` (the health subset of
`DaemonInternals`, pulled by the agent over `rpc.sock`; absent for a
tick the daemon did not answer). Absent from a daemon predating
self-metrics; every struct inside grows field-by-field.

### Events (`pub.sock`, `event`)

| `event` | Fields |
|---|---|
| `log` | `id`, `name`, `stream` (`out` \| `err`), `line`, `instance: u32` (default 0), `ts: u64` ms epoch (the daemon's receive time; the CLI stamps a replayed history line with the live log file's last write — see `docs/logs.md`) |
| `process` | `id`, `name`, `kind` (`start`, `online`, `exit`, `restart`, `stop`, `delete`, `errored`), and for `exit` only: `exit_code?: i32`, `exit_signal?: string`, `stderr_tail: string[]` (elided when empty) |
| `log_rotated` | `id`, `name`, `instance: u32` (default 0), `path` — `path` was renamed to `path.1` and reopened |
| `lagged` | `missed: u64` — the subscriber fell behind the bus |
| `daemon_shutdown` | `reason` (`kill`, `signal`, `home_removed`; unknown → `unknown`) — sent once, before any child is signalled; the socket closes when the drain completes. An EOF *without* it is an unannounced death |
| `app` | `id`, `name`, `instance`, `ts` (daemon receive time, ms epoch), `source?`, `event_name`, `level` (`debug`, `info`, `warn`, `error`, `fatal`; unknown → `unknown`), `severity_number` (OpenTelemetry: 5/9/13/17/21), `attrs` (flat scalars), `exc?: {type?, message?, stacktrace?}` (the instance's `cwd` and `$HOME` replaced by `.`/`~` in the trace), `fingerprint?`, `repeat` (events of the same fingerprint the rate limiter suppressed since the previous one) — an application event reported on the control channel |
| `daemon_handoff` | `phase` (`begin`, `done`, `failed`, `emergency`; unknown → `unknown`), `from_version`, `to_version` — an in-place handoff (`3pm update`): `begin` from the old image just before the exec, `done` from the new one once the rows are adopted, `failed`/`emergency` from whichever image ended up in charge after a problem (a resumed old image, or a new image that had to restart the apps). Daemon-level, like `daemon_shutdown` |
| `proc_action` | `id`, `name`, `instance` (default 0), `ts` (ms epoch, when the call settled), `source?`, `call` (the call id), `action`, `origin` (`local`: `3pm trigger` / `3pm profile`; `remote`: the fleet backend through the agent; unknown → `unknown`), `ok`, `err_code?` (an `ActionOutcome.err.code`), `duration_ms` — one custom action ran (or failed to) on one instance: the audit line. Neither the params nor the result ride the bus. A `trigger` refused as a whole once it named instances (several sources of one instance declare the action, none declares it, more than 16 instances) leaves one line per instance it resolved: `ok: false` without `err_code` (no call ran; the `err` answer says why), `duration_ms` 0, `source` the one asked for when it is a valid source name, a fresh `call`. One refused before naming an instance (`all`, an invalid action name, params over their cap, an unknown app or instance) leaves none: there is no process to write it about |

#### Subscription filter (`pub.sock`, subscriber → daemon, `event_filter`)

A subscriber may send **one** `EventFilter` line right after connecting;
the daemon waits at most 500 ms for it, then streams. No line, a late
line, an over-long (>64 KiB) or undecodable one all mean "everything" —
the daemon never hangs up over the handshake, and a daemon predating the
filter simply never reads the line. The broadcast cursor is attached
before the wait, so nothing published during the handshake is lost.

| Field | Meaning |
|---|---|
| `subscribe: string[]` | `*`/`?` globs over the event name; a `process` event also matches `process:<kind>` (`process:exit`) and a log line `log:<stream>` (`log:err`). Empty = every kind |
| `process?: string` | only events about this app name |
| `ids: u32[]` | only events about these process ids (what `3pm logs` sends). Empty = no restriction |

`process`/`ids` never scope out daemon-level events (`lagged`,
`daemon_shutdown`); only `subscribe` can. Every field is `serde(default)`
(additive growth); unknown fields are ignored.

A pattern is a user's word, not a wire shape: nothing decodes it. So the
three spellings a v1 subscriber could type — `log_out`, `log_err` and the
`log_*` glob — are still accepted, for one release, as `log:out`,
`log:err` and `log*` (`filter::aliased`). `log*` covers exactly what
`log_*` covered in v1: both streams and `log_rotated`.

### Control channel (`ctl.sock`, `msg`)

The spawner sets `THREEPM_CTL_SOCK` (the daemon's socket path),
`THREEPM_ID`, `THREEPM_GEN` and `THREEPM_BOOT` in every app's
environment, after the app's own `env` so an app cannot forge them.
`THREEPM_BOOT` is the daemon's lineage, 16 lowercase hex digits drawn at
every cold start (first start, crash, cold `3pm update`, `resurrect`
into a fresh daemon) and kept by an in-place handoff — unlike `boot_id`,
which a handoff redraws. The process-tree sampler adopts a daemonized
descendant through its environment only when the socket, the id, the
gen and the lineage are this daemon's and the incarnation's
(`docs/metrics.md` → *Process trees*). The CLI and host-agent launchers
strip the four from what they pass on, and the daemon from its own
environment at start, however it was launched.

Child → daemon (`ControlIn`): `hello {id, gen, source?}` (required first
line; `gen` from `$THREEPM_GEN` pins the connection to one incarnation — a
stale gen is hung up; `source` names the process within a multi-process
app, `^[a-z0-9][a-z0-9_-]{0,31}$`, unique per instance — a second hello
with the same source replaces the previous connection), `ready`,
`metrics {m: entry[]}`, `event {name, level?, attrs?, exc?, fp?}`,
`actions {list: decl[]}`, `action_reply {id, ok, data?, error?,
file?, output?}`, `call {id, node?, app, instance?, source?, action,
params, timeout_ms?, input?, output, stream}` (the app calling an action
on its own behalf — the same fields as the `call` request, plus the `id`
that correlates the answers), `action_started {id}` and the two lines of
the fleet gate, `fleet_ready {node_id, cluster_id, data_sock, secret}`
and `forward_update {id, ticket?, result?}`. Daemon → child
(`ControlOut`): `shutdown` (graceful stop request when the spec has
`shutdown_with_message`, written to every attached connection; SIGKILL
escalation still follows), `action {id, name, params, timeout_ms, file,
deadline_epoch_ms?, input?, output?, stream_out?}`, `call_ticket {id,
call_id, put?, in?, out?, deadline_epoch_ms}`, `call_reply {id, call_id,
ok, results, err?, output?}`, `forward {id, app, call}` and
`hosted_socks {socks: [{uid, gid, path}]}`.

**Calls (bytes attached).** A `call` is a `dispatch` whose input or output
is too big for the 1 MiB line, or open-ended. The bytes never ride this
socket: they travel as a **blob** — a raw stream with a `put` and a
`get`, addressed by an id and authorised by a token the daemon mints per
call (`{blob, token}`, a *local ticket*). The daemon answers a `call`
with `call_ticket` (where the bytes go, and `deadline_epoch_ms`: the
transfer budget plus the handler's timeout) and later with `call_reply`
(the outcomes, or `err` when it never ran). On the handler's side the
`action` line grows three legs: `input` (`{path?, blob?, token?, bytes?,
sha256?, ext?, live}` — `blob` absent means the bytes are local and
`path` *is* the file), `output` (the local ticket for a result the caller
asked for; the handler names it back in `action_reply.output`, `{blob,
bytes, ext, sha256}`) and `stream_out` (a streaming call's outbound leg).
`action_started {id}` says the handler has the bytes and starts now: the
daemon rearms `timeout_ms` from that moment, **once** per call and only
from the connection the call went to, never past `deadline_epoch_ms`.
`forward` and `forward_update` are the daemon's dialogue with the fleet
agent for a call that leaves the host; `fleet_ready` is how that agent
proves itself (the `secret` is per daemon launch, injected in the agent
app's environment). The secret is only half of the gate: the daemon also
reads the peer's pid on the socket (`SO_PEERCRED`) and opens the gate
only for the very process it spawned for the agent's row. That is a
constraint on the agent, not only a guarantee: the gate's connection
must be opened by the process the daemon launched — never by a child of
it, never by a detached worker, never over a descriptor handed from one.
A forked session carries a different pid and the gate simply does not
open; the refusal says nothing to the peer (a forgery must learn
nothing), so the symptom is a `fleet_ready` that is ignored and a
`warn!` in the daemon's log naming the peer's pid. A root daemon that
hosts users writes `hosted_socks` on the gate each time it opens: one
data socket per hosted user, `<hosted home>/agent/data-<uid>.sock`, which
the agent binds for that `uid:gid` (0600) — the path the SDK of a hosted
app looks for first. The list is whole and authoritative, empty when the
daemon hosts nobody: the agent closes a socket it no longer names.

**Error codes a call adds.** `ActionErrCode` grows with seven names a
plain `dispatch` never produces; the SDKs read them straight out of `e.code`.

| Code | Meaning |
|---|---|
| `lost` | the call left this host and never came back: the link fell, a blob went idle, a stream timed out. **At most once**: it may or may not have run, and nothing retries it |
| `node_offline` | the target node has no live link to the fleet, or this host's agent has no fleet gate open |
| `node_unknown` | no node of the bucket answers to that id or hostname |
| `ambiguous` | the hostname names more than one node: address it by its `3pmn_…` id |
| `forbidden` | a fleet scope that was never granted, or a target whose `callers` list holds rules and none admits the caller |
| `agent_too_old` | the agent on this host predates fleet calls, so the bytes have no road |
| `daemon_too_old` | the daemon on this host predates calls and never answered the `call` |

The other codes keep their meanings from `called` above, and an
unknown one still decodes as `unknown` — the list is additive.

Bounds, all in `threepm_protocol::{actions, calls}`: input
`ACTION_INPUT_MAX_BYTES` 8 GiB, concurrency `ACTION_CONCURRENT_MAX` 16
(at most `ACTION_INPUT_EXT_MAX` 16 extensions listed), a stream's hard
lifetime `ACTION_STREAM_MAX_MS` 24 h, calls per source
`CALLS_PER_SOURCE_MAX` 16, streams per daemon `STREAMS_PER_DAEMON_MAX`
16 (counted apart from the `CALLS_PER_DAEMON_MAX` 32 plain calls, so a
day-long session cannot freeze the daemon). The transfer budget of a
sized call is
`transfer_budget_ms(bytes)` = the size at 128 KiB/s, clamped to
`[60 s, 24 h]` — a 100 KB file gets a minute, a 2 GiB one gets 4.5 h.

Every number above is frozen in
`crates/threepm-protocol/tests/wire_samples/limits.json` and enforced by
`tests/limits.rs`. None of them travels on the wire, so an SDK's copy is
the only thing that can drift; the file is the reference it checks
against, and it carries worked `transfer_budget_ms` examples (sizes as
decimal strings, as they do not fit a double) so a reimplementation of
the formula is testable too. Regenerate with `UPDATE_LIMITS=1 cargo test
-p threepm-protocol --test limits -- --ignored update_limits`: a changed
number is a contract change for every SDK and the backend, an added key
is additive.

**Custom actions.** A connection with a `source` declares the actions it
answers to with `actions` — the whole list each time (an empty list
withdraws them all; sent again after every `hello`): `decl` = `{name`
(`^[A-Za-z0-9_.:-]{1,64}$`; `3pm:` is the SDKs' prefix by convention),
`description?` (≤ 128, cut), `remote?: bool` (default `true`: the fleet
backend may call it), `timeout_ms?` (default 30 s, ceiling 10 min),
`input?: {max_bytes: u64, ext: string[]}` (the bytes this action takes;
absent means none and a call carrying an input is refused. `max_bytes`
is `1..=8 GiB`, each `ext` follows the reply's grammar, an empty list
accepts any), `output?: {max_bytes: u64}` (the most the file this action
leaves may hold, `1..=256 MiB`; the SDK refuses to commit past it with
`too_large`, and the fleet backend reserves this much for a caller that
asks for the file — 256 MiB when absent), `stream?: bool` (default false; the handler is a session,
not a function, and `timeout_ms` then bounds one direction's idle time
rather than the call), `concurrent?: u32` (default 1, `1..=16`: calls of
this action one source may run at once)`}`; at most 32 per source, no
duplicates, or the list is refused whole and the previous one stays
(`threepm_protocol::validate_decl` holds the per-declaration bounds). A sourceless connection cannot declare. The
daemon triggers one with `action`: `id` correlates the reply, `params`
is the caller's JSON (≤ 64 KiB), `timeout_ms` the effective bound, and
`file` a path the handler may use for a result too big for the line —
it writes `<file>.tmp`, renames it to `<file>` (a regular file: a link
is refused), and names it in the reply; it must not write either after
`timeout_ms`. `deadline_epoch_ms` (ms since the Unix epoch, the host's
clock; absent from an older daemon) is when the daemon settles the call
`timeout`: an `action` read after it — the app's loop was blocked while
the line waited in the socket — must not start its handler (no reply
is needed; one would be ignored). The daemon never writes an `action`
whose call settled while it was still queued. `action_reply {id, ok, data?, error?, file?: {bytes,
ext}}` answers within `timeout_ms`: `data` (≤ 512 KiB serialised) and/or
`file` (`ext` `^[a-z0-9]{1,16}$` other than `tmp`; the daemon renames to
`<file>.<ext>`) when `ok`, `error` (the handler's message) otherwise. A
reply from another connection than the one the `action` went to is
ignored; a late reply too. One call per (source, action) at a time
unless the declaration raised `concurrent`, 16 per source, never the
last slot of the connection's outbound queue
(kept for `shutdown`), the daemon-wide bounds and every error code:
`dispatch` / `called` above.

**Files.** `<home>/actions/` keeps at most 20 accepted files per
instance directory and 200 MB in all, oldest first (by name: the call's
start); the retention runs after every accepted file and hourly. A file
of a call in flight, or of a `dispatch` not answered yet, is never
removed but counts. An accepted file modified less than 10 minutes ago
is spared by the per-instance count (whoever just got its path — the
fleet agent uploading it — has that long) and goes for the 200 MB only
once every older file is gone. A `.tmp` or extension-less file that no
such call owns (a handler that renamed after its deadline), and any
link, are removed outright.

#### Live framing (a stream's messages)

A `file` blob is raw bytes with a size and a digest, so it needs no
framing. A `live` blob has neither, so its bytes carry their own
boundaries: one message is a **20-byte big-endian header** followed by
its payload (`threepm_protocol::live`, no I/O).

```text
u32 len | u8 flags | u8 hdr | u16 rsv | u32 seq | u64 ts_us | payload[len]
```

| Field | Meaning |
|---|---|
| `len` | payload bytes, at most `LIVE_MESSAGE_MAX_BYTES` (1 MiB). A longer one is refused outright — the relay answers `422` — and the check needs only the first six bytes, so the payload is never buffered |
| `flags` | `KEY 0x01` (the payload decodes on its own: a reader may join here), `END 0x02` (last message), `INIT 0x04` (decoder parameters — H.264 SPS/PPS, `ftyp`+`moov` — kept by the window and prefixed to every reader that joins, whatever its age), `DISCONT 0x08` (messages were lost: reset the decoder). Unknown bits are carried, never refused |
| `hdr` | header bytes, `LIVE_HEADER_BYTES` (20) today. A **larger** header is accepted and its extra bytes skipped — that is how the framing grows; a smaller one is refused |
| `rsv` | reserved, written as 0, not interpreted |
| `seq` | the writer's message counter; it may jump, and the gap is what a reader dropped |
| `ts_us` | when the writer produced the message, microseconds since the Unix epoch, so per-message latency is measurable in production. It does not fit a double: read it as a 64-bit integer |

Frozen vectors:
`crates/threepm-protocol/tests/wire_samples/live_frames.json`, enforced
by `tests/live_vectors.rs` and reused by every SDK that reimplements the
framing. Each entry gives `head` (the header in hex, extension bytes
included), the payload as `payload` (hex) or `payload_fill` (one byte
repeated `times`, so the 1 MiB vector stays small), `accepts` with the
`error` name when false, `re_encodable` (the vector must come back byte
for byte out of an encoder) and the `decoded` fields — `ts_us` as a
decimal string. Regenerate with `UPDATE_LIVE_VECTORS=1 cargo test -p
threepm-protocol --test live_vectors -- --ignored update_vectors`; a
changed vector is a wire break, only additions are reviewable.

A `metrics` entry: `n` name (`^[a-z][a-z0-9_.]{0,63}$`), `k` kind
(`gauge` default, `counter` — cumulative since process start —,
`histogram`), `u` unit (UCUM, ≤ 16), `d` description (≤ 128), `l` labels
(≤ 3 keys `^[a-z][a-z0-9_]{0,31}$`, values ≤ 64 chars, reserved keys
refused: `cluster`, `node`, `app`, `instance`, `ns`, `src`, `overflow`,
`vmrange`, `le`, `hostname`, `os`, `arch`, `agent_version`, `mount`,
`window`, `core`, `iface`, `blockdev`, `sensor`, `gpu`, `array`, `proc`,
`pid` — except
the label set that is exactly `overflow="true"`, an
SDK's own fold of its excess label sets, merged with the daemon's), `v`
value (finite) or `h` histogram `{s: scale (0..4, default
1), count, sum, z: zero count, b: {index: cumulative count}}` on the
base-2 exponential grid of `threepm_protocol::exphist` (bucket `i` =
`(2^(i/2^s), 2^((i+1)/2^s)]`, a value on a boundary belongs to the bucket
below; the grid is frozen — changing it bumps `PROTOCOL_VERSION`).
Entries are decoded one by one: a bad entry is counted, never fatal; a
known kind with an unusable body is skipped; after `hello`, a line that
is not JSON is skipped too (only an over-long line or an I/O error ends
the connection — one bad line must not cost a source its series). Every batch is the SDK's
full picture (idempotent). Caps and rate limits: `docs/metrics.md`,
"Application metrics". The bucket grid is shared with the SDKs as
`packages/exphist-vectors.json` (generated by
`crates/threepm-protocol/tests/exphist_vectors.rs`, frozen like a wire
fixture); the SDKs themselves: `docs/app-metrics.md`.

### Attach (a terminal, over `rpc.sock`)

`attach` connects a client to the terminal of an app started with
`tty = true` (`3pm attach`). The request names the row and the client's
terminal; the daemon answers `attached`, and from the next byte on the
connection carries **live messages in both directions** (*Live framing*
above: 20-byte header, payload ≤ 1 MiB) until either side closes. No
fourth socket, no base64: the connection is the session.

```json
{"cmd":"attach","target":{"name":"claude"},"instance":0,"read_only":false,"mode":"rendered","peek":false,"cols":120,"rows":40}
{"res":"attached","id":3,"name":"claude","instance":0,"pid":4242,"gen":2,"alive":true,"cols":120,"rows":40,"viewers":1,"bracketed_paste":false}
```

- `cols`/`rows` are 0×0 for a pipe or a peek (no resize), otherwise each
  within `2..=1000`; out of bounds is `err{invalid_spec}`.
- Refusals, in the order they are judged: `shutting_down`,
  `handoff_in_progress`, `forbidden` (never, today: `attach` is
  app-level), `invalid_spec` (`all`, an id with an `instance` other than
  0, an unknown `mode`, a size out of bounds), `not_found` (an unknown
  app or instance, and a row the caller does not see — never an
  existence oracle), `not_tty` (the app has `tty = false`),
  `unavailable` (a `tty` row that never ran). A refusal is an ordinary
  NDJSON answer, after which the daemon closes the connection. A socket
  that serves no takeover (the host agent's `host.sock`) answers
  `unknown_request`; so does a daemon predating the request. Through
  any door but `rpc.sock` the request is judged the same way and a
  grantable one answered `unavailable`.
- A client that gives a size and may type (neither `read_only` nor
  `peek`) resizes the pty to it: the last such client wins.
- It is a mutation for the handoff gate (`handoff_in_progress` during
  `3pm update`) and is never re-sent by the client after a cut.
- A handoff (`3pm update`) ends every attached connection: the terminal
  and the screen cross the `execve` (handoff manifest version 2), the
  sessions do not (`detached{daemon_shutdown}`, then close). `3pm
  attach` on a terminal waits for the new image
  (`wait_daemon_gone_or_back`) and sends a fresh `attach` at its
  terminal's size; a pipe or a peek ends there (exit 3).

Each live message's payload is **one kind byte, then its body**
(`threepm_protocol::attach`, pure; integers big-endian):

| Direction | Kind | Body | Meaning |
|---|---|---|---|
| client → daemon | `0x01 input` | bytes (≤ 64 KiB; more closes the connection) | typed into the terminal |
| | `0x02 resize` | `cols u16`, `rows u16` | brought within bounds; ignored from a `read_only` client |
| | `0x03 detach` | — | clean end (`detached{client}`, then close) |
| | `0x04 ping` | — | answered `pong` (a pipe client knows its input was written) |
| | `0x05 eof` | — | the caller's stdin is over; only `3pm fleet ssh <node> -- <cmd>` sends it (the command's stdin is closed), a daemon ignores it |
| daemon → client | `0x10 screen` | `parts u8` (`0x01` more parts follow, `0x02` continues the previous part), bytes | a whole screen: on arrival, after `restarted`, a resize or a lag. Split when over the framing's cap; the first part carries the live `KEY` flag |
| | `0x11 output` | bytes | `raw` mode: what the app wrote, in order |
| | `0x18 diff` | bytes | `rendered` mode: scrolled history, then the difference against what the client has |
| | `0x1c alt_screen` | `on u8` | the app entered / left the alternate screen |
| | `0x1d stderr` | bytes | never from a daemon: what a fleet command without a pty wrote on its stderr (`output` carries its stdout) |
| | `0x12 resized` | `cols u16`, `rows u16` | the pty's size changed |
| | `0x13 exited` | `code i32` (`i32::MIN` = none), `signal i32` (0 = none) | the incarnation exited; the client stays |
| | `0x14 restarted` | `pid u32`, `gen u64` | a new incarnation; a `screen` follows |
| | `0x15 detached` | `reason u8`: 0 `client`, 1 `deleted`, 2 `daemon_shutdown`, 3 `input_backlog`, 4 `not_tty_anymore`, 5 `too_slow` (unknown → `unknown`) | the end |
| | `0x16 lagged` | `dropped u32` (live `DISCONT` flag) | `raw` mode: output was lost; a `screen` follows |
| | `0x17 pong` | — | |
| | `0x19 text` | `cols, rows, cursor_row, cursor_col u16`, `alive u8`, `unmodelled u32`, UTF-8 lines joined by `\n` | the answer to `peek`, followed by `detached{client}` |
| | `0x1a title` | UTF-8 | the app's window title |
| | `0x1b bell` | — | BEL |

How the daemon serves it: a `resize` is brought within bounds and
coalesced (the last one within 50 ms applies); a `ping` is answered
once the client's earlier `input` is written to the terminal; an
`exited` comes after the app's last output was sent (its alternate
screen left, if it left it); `detached{daemon_shutdown}` precedes a
daemon's shutdown or handoff, `too_slow` a client that let 32 lifecycle
notes pile up, and a client that takes more than 10 s to read one
message is cut. `rendered` bytes never switch the alternate screen
(`alt_screen` does, the client writes `CSI ? 1049 h/l`); history rows
come first, cut to the client's width, as rows written at the top of
the screen then scrolled out by `\n` on its last row, so they land in
the client terminal's own scrollback (a client taller than the pty, a
second one, scrolls from its own last row, its rows below the pty's
blanked first). The cursor shape (DECSCUSR `CSI Ps SP q`) goes out when
the app changes it; the client resets it on the way out.

Growth: a kind a side does not know — or one of the other direction — is
**ignored**; a fixed-size body may grow at its tail (the extra bytes are
skipped); a body shorter than its kind needs is an error, never a panic;
`seq` and `ts_us` are filled as for any live message. All of it is
additive within protocol v2. Frozen vectors:
`crates/threepm-protocol/tests/fixtures/wire/v2/attach.json`, enforced by
`tests/attach_vectors.rs` (regenerate with `UPDATE_ATTACH_VECTORS=1 cargo
test -p threepm-protocol --test attach_vectors -- --ignored
update_vectors`; additions at the tail only).

## 4. `--json` stability

`3pm ls --json` / `show --json` / `start --json` etc. print
`ProcessRecord`s straight off the wire. Within a protocol generation:

**Stable (scripts may rely on these):** `id`, `name`, `instance`,
`status` (`launching`, `online`, `stopping`, `stopped`,
`waiting_restart`, `errored`), `pid` (null when not running), `restarts`,
`unstable_restarts`, `uptime_ms` (null unless online),
`created_at_epoch_ms`, `exit_code`, `exit_signal`, `namespace`.

**Present, additive-only, values may be refined:** `cpu_percent`,
`mem_bytes`, `listen_ports`, `stderr_tail`, `last_error`, `watch_error`,
`waiting_for`, `next_cron_epoch_ms`, `out_log`, `err_log`, `pid_file`,
`proc_count`, `tree_rev`, `tty_viewers` (clients attached to the
instance's terminal; `0` without `tty` and from a daemon predating it).
Under `tty` there is one output stream: `err_log` is empty.

`cpu_percent`, `mem_bytes` and `listen_ports` describe the instance's
**process tree** — the spawned pid and every descendant the daemon
attributes to it (membership rules: `crates/threepm-daemon/src/metrics/tree.rs`;
neither another daemon's child nor a survivor of an earlier daemon of the
same home is ever adopted through its environment, and on macOS another
user's process is never a member): cpu is summed
(100 = one core), memory is the sum of the members' RSS (an upper bound
when they share pages), ports are the union. `proc_count` (additive, `0`
from a daemon predating trees) is the number of live members, `1` for an
app running alone; `tree_rev` (additive) moves when the listed membership
changes — a pid enters or leaves the first 256 lines, a member detaches
or takes another name, the incarnation exits — never on cpu/mem, to the
next value of one counter per daemon: it never goes back within one
image, and a daemon started cold numbers from 0 again. An in-place
handoff carries the counter (no revision is ever reused) but not the
rows': an adopted row reports `0` until its next metrics tick, and the
daemon redraws `boot_id` — a cache keyed on the pair must watch
`boot_id` and drop what it holds when it changes. So, within
one image, `(gen, tree_rev)` says whether `process_trees` has a new shape;
members beyond the per-tree cap do not move it. The spawned pid's own figures
are the first line of its tree. `THREEPM_PROCESS_TREE=off` on the daemon
restores the pre-tree meaning (the spawned pid alone, `proc_count = 1`,
empty trees).

**Best-effort:** `spec` mirrors the daemon's resolved `StartSpec`
(`threepm-core`) field for field. It is exposed for inspection and
round-tripping through `3pm apply`, not as a stability contract: fields
follow the per-app config surface and are documented in `3pm start
--help` / the ecosystem reference, not here. Two provenance fields are
daemon-stamped rather than user-set and absent when they do not apply:
`spec.source` (`{path, profile}`, the ecosystem file) and `spec.module`
(`{name, version}`, the module that owns the app — additive, absent on
user apps and on daemons predating modules).

A failed command prints one error document on stderr —
`{"error":{"code":…,"message":…}}`, the wire `ErrCode` verbatim (`cli` for
a client-side failure) — and exits **1**. The one exception is
`not_found`, which exits **2**, so a script can tell "no such app" from
"it did not work". `forbidden` is deliberately *not* another exit code: a
hosted user reading `code: "forbidden"` learns it asked for a daemon-level
command, and every other failure keeps the code it always had.

`DaemonInternals` (`3pm internals --json`) and `HostMetrics`
(`3pm hostmetrics --json`) are growable snapshots: every field is
optional on decode and platform-dependent blanks are absent rather than
faked. Treat individual fields as present-when-available.

`DaemonInternals.internal_processes: InternalProcess[]` lists 3pm's own
processes (additive; absent from a daemon predating it): `role`
(`daemon`, `host-agent` — may grow), `command`, `pid?`, `status`
(`online`, `waiting_restart` — may grow), `uptime_ms?` (this
incarnation), `cpu_percent`, `mem_bytes` (0 before the first sample or
while not running), `restarts`. Each row is `serde(default)` like its
parent. The older `host_agent_pid` / `host_agent_restarts` stay.

`DaemonInternals.boot_id` (the same id `pong` carries) and
`DaemonInternals.handoffs` (in-place handoffs this image went through)
are additive; `0` from a daemon predating them.
`DaemonInternals.last_handoff` (optional) records the last handoff this
daemon (the pid, across images) concluded: `outcome` (`done`, `failed`
or `emergency`), `at_epoch_ms`, `duration_ms`, `rows`, `from_version`,
`to_version`, and `error?` on a failure. Absent until the first handoff.
`DaemonInternals.metrics_tick_ms` (additive, `0` before the first tick
or from a daemon predating trees) is how long the last metrics tick took
— the process-table sweep that prices the process trees.
`DaemonInternals.metrics_stalled_ms` (optional, additive) is how long
the metrics sweep in flight has been running, counted from when that
sweep began, present only once it is past `max(5 × period, 5 s)` — no
sample, so no resource guard, for that long (`docs/metrics.md`, **Retrieval model**). Absent while sampling
is healthy and from a daemon predating it.

Client expectations around a handoff (what `threepm_protocol::client`
implements): a request answered `err{handoff_in_progress}` may be
repeated until it is not (bounded, 15 s); a connection cut without a
reply while `handoff.json` sits next to the socket, or after such a
reply, may be reopened and the request sent again — the old image
processes nothing it refused, so no request is ever applied twice —
except `handoff` and `kill_daemon`, which must not be repeated. A
`pong` whose `boot_id` differs from the one first seen is the successor
image; the same `boot_id` with a `last_handoff` newer than the request
means the exec failed and the old image resumed.

</document>

<document url="https://3pm.edgecr4ft.net/docs/internals/site.md" title="This site: what is online and how it is published">

# The download site — https://3pm.edgecr4ft.net

Where 3pm is distributed from: the binaries, the SDKs, the documentation
and the landing page. Everything is static files under one nginx root; no
application runs there. This page is how to change it and how to ship the
change.

## What is online

```
/                        index.html      the landing page (distribution/site/index.html)
/install.sh                              installer and updater (distribution/install.sh)
/latest/VERSION                          the current version, one line
/latest/version.json                     that release: version, published_at, install, checksums, assets[]
/vX.Y.Z/                                 3pm-X.Y.Z-<target>.tar.gz ×4, checksums.txt, version.json,
                                         install.sh, release.txt, release.sig (signed release)
/sdk/sdk.json                            every SDK: latest version, files, all versions
/sdk/node|python|cpp|rust/               the SDK archives
/simple/                                 PEP 503 pip index (threepm-io)
/sdk/cargo/                              sparse Cargo registry (config.json + th/re/threepm-io)
/llms.txt                                documentation index for LLMs (llmstxt.org)
/llms-full.txt                           every document in one file
/docs/index.json                         the same index as JSON (bytes, sha256, backend routes)
/docs/**.md                              the documentation, as Markdown
/sitemap.xml, /robots.txt
```

## The three commands that publish it

Each one is a script under `distribution/scripts/`, run from the Mac, and
each one rsyncs to `ubuntu@edgecr4ft.net:/var/www/3pm`. They share the
overrides `THREEPM_PUBLISH_HOST`, `THREEPM_PUBLISH_DIR`,
`THREEPM_PUBLISH_URL`, and they all take `--dry-run` (build, upload
nothing; output under `target/publish*/site`).

| Command | Builds | Uploads |
|---|---|---|
| `just publish` | The five release tarballs (macOS arm64 via cargo here; Linux x86_64/arm64 via cargo-zigbuild on the build host, at glibc 2.28 and as static musl `-musl` builds) | `vX.Y.Z/`, then `install.sh`, `index.html`, `latest/` |
| `just publish-sdk` | The four SDK archives, each tested and installed into an empty project first | `sdk/`, `simple/`, `sdk/cargo/`, `sdk/sdk.json`, `index.html` |
| `just publish-docs` | The documentation (below) | `docs/`, `llms.txt`, `llms-full.txt`, `sitemap.xml`, `robots.txt`, `index.html` |

**The three of them upload `index.html`**, so editing the landing page
alone changes nothing online: run one of them (`just publish-docs` is the
cheapest) to ship it.

Versions: `just publish` bumps the minor version by itself when the
current one is already online, and `--force` republishes it. Each SDK
carries its own version and is skipped when that version is online: bump
it by hand in `packages/node/package.json`,
`packages/python/pyproject.toml`, `packages/cpp/CMakeLists.txt` or
`packages/rust/Cargo.toml` (its `Cargo.lock` too — `publish.sh` stamps
both). Nothing online is ever deleted, and the indexes are rebuilt from
the archives already on the server.

### The Cargo registry

Cargo cannot depend on a tarball URL, so the Rust SDK ships as a `.crate`
plus a **static sparse registry** under `/sdk/cargo/`: `config.json` names
where the `.crate` files are, and `th/re/threepm-io` (cargo's prefix
layout, the name lowercased) holds one JSON line per version with its
`deps`, `features` and `cksum`. Both are plain files nginx serves as they
are — the index file has no extension and goes out as
`application/octet-stream`, which cargo does not mind, and no nginx
change is needed. `write_cargo_index` rebuilds the whole index from every
`.crate` in the site, reading each archive's own manifest with `cargo
metadata`, exactly as `simple/` is rebuilt from every wheel. A
dependency's `registry` is written out as crates.io explicitly: left
null it would mean *this* registry, where nothing but `threepm-io` lives.

The consumer side, in `.cargo/config.toml` beside `Cargo.toml` or in
`$CARGO_HOME`:

```toml
[registries.threepm]
index = "sparse+https://3pm.edgecr4ft.net/sdk/cargo/"
```

```toml
[dependencies]
threepm-io = { version = "0.26.0", registry = "threepm" }
```

`publish-sdk.sh` proves that chain before anything is uploaded: it serves
the staged site on a random local port, points a scratch crate's registry
at `sparse+http://127.0.0.1:<port>/sdk/cargo/` and builds it. A registry
cargo cannot read fails the publish here rather than on a user's machine.

## Signed releases

Every release carries `vX.Y.Z/release.txt` (`version X.Y.Z`, then
`<sha256>  <file>` for each tarball and for `vX.Y.Z/install.sh`, a copy of
the installer of that release) and `vX.Y.Z/release.sig`, the ed25519
signature of those bytes in hex. The `autoupdate` module verifies both
against `RELEASE_KEYS` (`modules/autoupdate/src/manifest.rs`) before it
runs anything; a fresh `curl | sh` does not (first contact).

- **Where it signs:** on the Mac, in `publish.sh` (`sign_release`, through
  `cargo run --example release-key` of `modules/autoupdate`). The key
  never leaves the Mac: not the build host, not the site.
- **The keys:** `just release-keygen` writes `~/.3pm-release/{main,rescue}.{key,pub}`
  (directory 0700, private keys 0600, never overwritten) and prints the
  public halves. `THREEPM_RELEASE_KEY` points `publish.sh` elsewhere. The
  directory is outside `3pm-rd/` on purpose: Mutagen syncs that tree to
  nirmata. `rescue.key` belongs on offline media, not on the Mac.
- **The checks:** `publish.sh` refuses to start without the key (a dry
  run only warns), the signer refuses a key that is not in
  `RELEASE_KEYS`, and the verify step compares the uploaded `release.sig`
  with the local one.
- **Rotation:** add the new public key to `RELEASE_KEYS`, publish that
  release signed by a key already trusted, drop the old key in a later
  release. A node that never updates keeps its old list: there is no
  revocation for it. Lost main key: sign with `rescue.key` (`THREEPM_RELEASE_KEY`)
  a release that carries a new main key.
- **Modules from the site:** a module installed by `3pm install <name>`
  (not bundled) comes from the site's `modules/<name>/…/version.json`,
  which is **not signed**. The autoupdate path never refreshes it:
  `install.sh`, run with `THREEPM_ARCHIVE`, calls `3pm update
  --bundled-modules-only`, so only the modules of the signed archive are
  installed. A `3pm update` typed by hand still takes the site's newer
  release, unsigned (open: sign the module index too).

## The build host

`just publish` compiles three targets for three workspaces — nine cargo
jobs. The six Linux ones run on a build host over ssh; only
`aarch64-apple-darwin` stays on the Mac, because a signed arm64 Mach-O
needs the Apple toolchain. The Mac remains the orchestrator: it bumps the
versions, stages the tarballs, writes the checksums, uploads and verifies.

| Override | Default | Meaning |
|---|---|---|
| `THREEPM_BUILD_HOST` | `unitech@10.0.0.95` (nirmata) | Where the Linux targets compile. **Set it to the empty string to build everything on the Mac** — the way out when the host is down. |
| `THREEPM_BUILD_DIR` | `3pm-rd-publish` | Its directory in the account's `$HOME`. Deliberately not `~/3pm-rd-build`, which `../deploy/test-on-linux.sh` uses: release and test state stay apart. |

`../deploy/deploy-backend.sh` reads `THREEPM_BUILD_HOST` too and builds the
fleet backend there (one target, `x86_64-unknown-linux-gnu.2.28`), in a
directory of its own, `~/3pm-rd-backend-release`: 3pm-rd's `just publish` runs
it at the same time as this script, and two `rsync --delete` into one tree
remove each other's temporary files. It syncs **both** repositories, since
`3pm-backend` path-depends on `../3pm/crates`, and the binary comes back here
to be deployed: the build host has no key for the publish host, and needs none.
The build writes the binary's sha256 next to it; the deploy refuses a binary
that no longer matches, and checks that the restarted process runs that sha.
(Measured 2026-10-09, warm: 1 min 47 on nirmata alongside this script's ten
jobs, 3 min 30 natively on edgecr4ft, whose 2017 EPYC is slow on the backend's
single-codegen-unit crate.)

### Steps, for 3pm-rd's `just publish`

Each script splits into steps, so that 3pm-rd's `just publish` (in
`../justfile`) can run the four builds at once on three machines and upload
only once all four are green:

| Script | Steps |
|---|---|
| `publish.sh` | `--step=version` (bump + stamp the tree), `--step=build` (build, package, sign the version in `Cargo.toml`), `--step=upload` (upload + verify the staging; refuses a version already online without `--force`) |
| `publish-sdk.sh` | `--step=build` (test, package, indexes; what ships is kept in `target/publish-sdk/shipped`), `--step=upload` |
| `publish-automonit.sh` | `--step=build` (3pm need not be online yet), `--step=upload` (requires 3pm vX.Y.Z online) |
| `../deploy/deploy-backend.sh` | `--step=build`, `--step=deploy` |

Without `--step` each script runs its steps in a row, as before (`just
publish` here, `just publish-sdk`, `just publish-automonit`). `--dry-run`
and `--step` do not mix. Before any of it, 3pm-rd's `just publish` runs
`../deploy/quick-tests.sh` (3pm without its e2e binaries, the backend
without `real_agent_*`, both at once on edgecr4ft, about 30 s warm).

Provision a host with `./distribution/scripts/setup-build-host.sh
[user@host]`. It is idempotent and needs no sudo — everything lands in the
account's `$HOME`. It pins rust, zig and cargo-zigbuild to the Mac's
versions (the three constants at the top of the script; bump them together
with `brew list --versions cargo-zigbuild zig`), then proves the chain by
building a hello world and reading back the highest glibc symbol it
requires, which must be 2.28.

Sources reach the host by one `rsync` — after the version bumps, since the
binaries embed the version, and before any job starts. The binaries come
back to the same paths a local build would have written, so staging,
tarballs and checksums are unchanged. A Ctrl-C stops the remote cargo
processes too: they all carry `$THREEPM_BUILD_DIR` in their command line,
which is what `stop_jobs` matches on.

The three publish scripts, `../deploy/deploy-backend.sh` and
`../deploy/quick-tests.sh` share **one**
multiplexed ssh connection per host: a fixed `ControlPath`
(`/tmp/3pm-ssh-$UID/%h`, short because macOS caps a socket path at 104
bytes), so whichever runs first opens the master and the others reuse it,
and `ControlPersist` reaps it rather than any script tearing it down.
Without this, each `ssh` and each `rsync` opens its own handshake; `just
publish` runs them side by side, and a burst of handshakes makes
sshd answer `Exceeded MaxStartups` and drop a leg mid-upload. Each upload
leg is retried three times, and the reachability probe too — one refused
connection is not an unreachable host, and it reports what ssh said.

## Changing the landing page

`distribution/site/index.html` is the whole page: plain HTML, CSS and JS
in one file, no dependency, no build step. Open it in a browser to see the
change (the version badge, the download list and the SDK block stay empty
locally — they are fetched from `latest/version.json` and `sdk/sdk.json`),
then publish it.

For a phone check, headless Chrome enforces a ~500px minimum width:
screenshot the page inside a 380px iframe instead.

## The documentation: `just publish-docs`

`distribution/scripts/publish-docs.sh` builds `target/publish-docs/site`,
then uploads it. `distribution/scripts/docs-site.py` assembles it, and its
`MANIFEST` is the one list of documents: section, path on the site, source,
title, and the line an LLM reads to decide whether to open the document. A
source that no longer exists fails the build.

Three kinds of document:

1. **Written for the site**, under `distribution/site/docs/`:
   `concepts.md` (the model: daemon, apps and instances, statuses, metrics
   layers, SDKs, the fleet vocabulary) and `environment.md` (every
   environment variable, grouped by who reads it).
2. **Generated**, so they cannot go stale:
   - `docs/cli.md` walks `3pm <command> --help` down every subcommand,
     modules included, in a throwaway `THREEPM_HOME` under `/tmp`.
   - `docs/backend/api.md` and `docs/backend/routes/<name>.md` are what a
     freshly built `3pm-backend` answers on `GET /api/v1/routes` (and
     `?route=<name>`): the script builds it from `../3pm-backend`
     (`THREEPM_BACKEND_SRC`), runs it on a random port with a throwaway
     database, asks, and stops it.
   - `docs/backend/cli.md` is `3pm-backend --help`.
3. **Copied from the repositories**: `README.md`, `FEATURES.md`, the
   `docs/` pages, the four SDK READMEs, and the backend's `README.md`,
   `FEATURES.md` and `CHANGELOG.md`.

`llms.txt` lists every document with its description and every backend
route with its method, path and auth; `llms-full.txt` is all of them
concatenated (~800 KiB); `docs/index.json` is the same index with sizes
and sha256.

Adding a document is one line in `MANIFEST`. After any change to the docs,
the CLI or the backend's route table, rerun `just publish-docs`.

## The server

`ubuntu@edgecr4ft.net` (Scaleway, Ubuntu 20.04, nginx 1.18, certbot),
also the host of `fleet.edgecr4ft.net` (the production `3pm-backend`) and
of the apps under `~/3pm-apps/`. DNS is on Route 53, with a wildcard
`*.edgecr4ft.net`.

- Web root `/var/www/3pm`, `autoindex on`, `charset utf-8`.
- Vhost `/etc/nginx/conf.d/3pm.edgecr4ft.net.conf` — the vhosts live in
  `conf.d`, which `nginx.conf` includes before its own server blocks;
  `sites-enabled` is not loaded.
- Cache headers: `no-cache` on `install.sh` and `latest/`, one hour on
  `v*` (a version's files never change without `--force`).
- TLS by certbot (webroot challenge under the same root).

`sudo` needs a password there, so nginx changes are run by hand, not by
the publish scripts.

**Pending**: `.md` is not in nginx's `mime.types`, so `/docs/*.md` is
served as `application/octet-stream` (a browser downloads it instead of
showing it). The fix, in the 443 server block of the vhost:

```nginx
    location /docs/ {
        types { text/markdown md; application/json json; }
        charset_types text/markdown application/json;
        add_header Cache-Control "no-cache";
    }
    location ~ ^/llms(-full)?\.txt$ { add_header Cache-Control "no-cache"; }
```

then `sudo nginx -t && sudo systemctl reload nginx`. `publish-docs.sh`
checks the content type after every upload and prints this block while it
is still wrong.

## Not published here

`distribution/setup-apt.sh` and `setup-yum.sh` still point at the
`releases.3pm.dev` placeholder: there is no apt or yum repository. The
`README.md` install section names them and the GitHub releases page, which
do not exist yet — and it is published as `docs/readme.md` on the site.

</document>

<document url="https://3pm.edgecr4ft.net/docs/changelog.md" title="3pm changelog">

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added
- **musl builds for Alpine**: every release also ships
  `3pm-linux-x86_64-musl` and `3pm-linux-arm64-musl` (static, modules
  included). `install.sh` picks them on a musl host (`THREEPM_LIBC=glibc|musl`
  forces the choice), and the autoupdate module of a musl build updates to
  the musl archive. On musl the logged-in-users host metric is always 0 (no
  utmp) and the glibc malloc tuning of the agent does not apply.
- **Real-time limits per app: `ulimit_rtprio`, `ulimit_memlock`**
  (`--ulimit-rtprio`, `--ulimit-memlock`, SDK `start()`): the app's
  `RLIMIT_RTPRIO` (0–99, Linux only) and `RLIMIT_MEMLOCK` (a size or
  `"unlimited"`), set soft = hard right before `exec`, so a real-time
  audio app's threads may run `SCHED_FIFO` — an app of a systemd-started
  daemon inherited 0 and 8 MB. A root daemon grants any value (set before
  the drop to `user`; `no_new_privs` stays), a non-root one up to its own
  hard limit, and an app of a hosted user what `[users]
  max_ulimit_rtprio` / `max_ulimit_memlock` grant (nothing when unset) —
  refused at `start` otherwise. `3pm show` lists them.
- **Scheduling per app: `ulimit_nice`, `ulimit_rttime`, `nice`, `ioprio`,
  `oom_score_adj`, `cpu_affinity`** (flags, ecosystem, SDK `start()`),
  same model as `ulimit_rtprio`: set by the child on itself before
  `exec`, before the drop to `user`, judged at `start`. `RLIMIT_NICE`
  (0–40) gives back the negative nice a desktop session allows (Chromium's
  audio threads at -8, the plugin host's -11 fallback);
  `RLIMIT_RTTIME` (µs) SIGKILLs a `SCHED_FIFO` thread that never blocks; the
  nice value, the I/O class (`idle`, `best-effort[:N]`, `realtime[:N]`),
  the OOM killer bias and the CPU list are the app's from its first
  instruction. What only gives up is never refused; a daemon without
  `CAP_SYS_NICE` / `CAP_SYS_RESOURCE` refuses what the kernel would, a CPU
  the daemon does not have is refused, and a hosted user's app gets a
  nice / OOM score below the daemon's only from `[users] max_ulimit_nice` /
  `min_oom_score_adj`, never the realtime I/O class. A refused step
  names its field. `3pm show` lists them.
- **Link traffic by category, the agent** (lot 2 of
  `rd/logs-bytes-counter/PLAN.md`): `3pm agent` counts the bytes and
  messages it sends and receives per kind (`host`, `extended`, `top`,
  `procs`, `app`, `trees`, `events`, `shell`, `calls`, `control`,
  `traffic`, `logs`, `artifacts`, `blobs`, `api`), ships them as the
  `traffic` section every 5 s, and shows them in `3pm agent status` (a
  `traffic` row, "as of", `--json` table) from `<state>/traffic.json`,
  written at most once a minute. State-frame kinds are a pro-rata
  estimate of an exact total. The `logs` status gains `bytes_sent`.
- **Link traffic by category, the wire** (`threepm-link`, lot 1 of
  `rd/logs-bytes-counter/PLAN.md`): the closed list of kinds and the two
  directions both sides share (`threepm_link::traffic`: `KINDS`, `UP`/`DOWN`,
  `section_kind`), the split of a `state` frame's compressed size over its
  patches (`patch_weights`, Extended Monitoring — `host.extended`, tree
  members' `x` — and the top processes carved out; `pro_rata`, integer shares summing to the frame's size), and the
  `traffic` section (`node_state::TRAFFIC`, a name outside `Section`;
  `{since_ms, up, down: {kind: {b, m}}}`, read leniently by `traffic_view`, at most 64 unknown kinds a direction;
  `Traffic::known()` yields the closed vocabulary only).
  Additive: no `LINK_VERSION` bump; the agent sends it (the entry above).
- **`[module] kill_timeout_ms`** in `module.toml`: the least stop grace of
  a module's own rows, the dynamic ones too. A module refresh (`3pm
  update`, `3pm install`) raises an existing row's `kill_timeout_ms` to it,
  never lowers it. The agent module declares 10 s, room for its logs' last
  batch at a stop (rows linked before get it at the next update, no
  relink).
- **The wire of log shipping** (`threepm-link::logs`, first lot of
  `rd/logs-shipping/PLAN.md`): the records a node will post to
  `POST /api/buckets/{cluster}/logs` — `batch`, `src`, `tpl`, `cnt`,
  `line`, `dropped`, tagged `k`, every word a build does not know decoding
  to `unknown` — their bounds (`LogRecord::check`, `cap_text`) and the
  `200` answer; `welcome.logs {apps, host, files}` (absent = ship nothing)
  and `link.toml` `[logs] {apps, host, files, host_level, exclude}`, the
  host's half, combined by `LogsSettings::effective`. Logs still never ride
  the link itself (one DEFLATE context for every frame); the protocol doc
  gains §6b. Additive: no `LINK_VERSION` bump. Nothing ships yet.
- **The host's log files, found by their writer** (`log_sources`, lot B
  of `rd/logs-shipping/PLAN.md`). On Linux the host agent walks every fd
  table it can see once a minute (first pass at start, its own task,
  `THREEPM_HOSTMETRICS_LOGFILES_MS` to change the period) for regular
  files open for writing with `O_APPEND`, names each writer (pid, comm,
  systemd unit and container id from its cgroup), keeps one entry per
  `(dev, ino)`, and decides a verdict: `private` (shell and REPL
  histories, `.lesshst`, `.viminfo`, anything under `.ssh/` or `.gnupg/`:
  never shipped), `duplicate_journal` (rsyslogd, syslog-ng, journald),
  `own` / `duplicate_bus` (this daemon's and host agent's logs and its
  apps' files; another 3pm daemon is an ordinary writer), `binary` (a NUL
  byte in the first 4 KiB), `ship` as `docker-json` (dockerd's
  `containers/<id>/<id>-json.log`, owned by the container) or `text`. A
  path is declared only when, in the agent's mount namespace, it is a
  regular file (never a symlink) with the descriptor's `(dev, ino)`; it
  is opened `O_NOFOLLOW | O_NONBLOCK` and checked by `fstat`, so a writer
  in a container or a swapped symlink cannot point a root agent at
  another file. `dpkg.log`, `apt/history.log` and
  `unattended-upgrades/*.log` are listed while they exist; a file seen
  once stays listed while its path is that file or its rotation by the
  same owner. 2 000 files at most (`ship` first, the cut logged); a pass
  stuck over 30 s is abandoned and the previous list served. The daemon proxies
  `Request::LogSources` to the agent like `host_metrics`; root only (a
  hosted caller is refused like `internals`). Other OSes answer an empty
  list. Nothing reads or ships the files yet.
- **A linked node ships its logs** (`3pm agent`, lots C and D of
  `rd/logs-shipping/PLAN.md`), by default, to a backend that says
  `welcome.logs`: the 3pm apps' output (a second `log` subscription to
  the bus, its own cursor), the system journal at `info` or worse (one
  `journalctl --follow --output=json` child, resumed after the cursor of
  the last accepted batch, first start from now, entries over 24 h old
  dropped) and the files the host agent lists with verdict `ship`
  (tailed from their end at first sight, rotation and `copytruncate`
  followed, 512 files, parked after 10 min idle; `EACCES` skipped until
  the next 60 s pass). Every line is redacted, folded, mined into
  templates (Drain, ids per source never reused, state in
  `<home>/agent/logs/miner`) and counted per 5-minute window; a raw line
  leaves only when it is news (`new`, `error` once per template per
  hour, `back`, `spike`, `return`, the 30 s of `context` before a new
  error), within 4 KiB/s per source (256 KiB burst) and a 4 MiB RAM
  ring while the backend is away. A batch every 10 s, `POST
  /api/buckets/{cluster}/logs`, gzip NDJSON, never on the WebSocket;
  file offsets and the journal cursor (`<home>/agent/logs/offsets`,
  `journal.cursor`) advance only on `200`, so a line not accepted
  before a stop is read again and stored once. `[logs]` in `link.toml`
  (`apps`, `host`, `files`, `host_level`, `exclude`) turns sources off,
  read at the agent's start. At a stop the agent closes its windows and
  tries a last batch within 5 s. Linux for the journal and files; on
  macOS only the apps' output.
- **`3pm agent status` has a `logs` row**: `24 sources (8 app, 15 file,
  1 journal), 451 lines read, 33 shipped, 33 templates; batches 16 ok,
  0 retried, 0 dropped`, then `skipped …`, `lines dropped …` by reason
  and the last error when there are any (`--json`: `logs`).
- **automonit's log files join the agent's.** When the root-installed
  `automonit` module runs, the agent asks its `logs` action every 5 min
  (`3pm-automonit/docs/logs.md`): the files a service names itself
  (nginx and Apache configuration, MySQL, PostgreSQL, MongoDB, Redis),
  with a format. Merged by `(dev, inode)` with the host agent's list,
  automonit's owner and format win. Only the module row is asked, never
  an app a hosted user named `automonit`; a file automonit adds alone
  must look like a log (`/var/log`, a `log` directory, `*.log`), sit
  outside `/etc`, `/root` and the private places, have one link and
  the owner uid it declared.
- **Compression inside the end-to-end channel for bulk sessions.**
  `3pm fleet cp` compresses by default (`--no-compress` to opt out);
  `3pm fleet ssh -C -- <cmd>` asks for it when the command runs
  without a pty; `-v` says whether it is on. zstd level 1, one frame per
  sealed record, raw when it would not shrink (one byte of overhead),
  never past 65 518 bytes once opened. Never a shell, a pty, `attach`
  or a forward (keystroke latency; CRIME-style length leaks). Agreed
  in the Noise handshake's payloads, which the handshake authenticates:
  no new request field, nothing for the backend to strip; an older agent
  or 3pm simply never compresses.
- **`lan`, a virtual Ethernet between the root Linux nodes of a bucket**
  (bundled module, marked `auto`: every 3pm installs it; off with
  `[modules] off = ["lan"]` in `config.toml` or `3pm uninstall lan`).
  Where it cannot work (not Linux, not root, no `/dev/net/tun`, not
  linked) it stays up and idle, says why once, and `lan.status` gives
  the reason: never a crash loop; it then declares no `link` action and
  its peers, which keep only nodes declaring it (`fleet:read`), never
  call it; failed opens back off per peer 5 s → 10 min with jitter, and
  past 6 a minute all dialling pauses. It makes a TAP
  `3pmlan0` on `10.77.0.0/16` (address and MAC derived from the node id)
  only once a peer shows up, removes it — and its sessions — 10 min
  after the last one leaves, and never puts it over a range already in
  use on the host (a default route split in `/1` halves, OpenVPN's
  `def1`, is no range in use; a TAP held by the previous `lan` is
  retried, not idle). It finds its peers itself (`3pm fleet nodes|apps
  --json`, needs `fleet:list`; sessions need `fleet:invoke`): at start,
  5 s later, every 15 s while alone, then every minute, and at once for
  a session from an unknown node (≤ 3 s wait). Frames ride fleet stream
  sessions, agent to agent between LAN neighbours (the node's `direct`
  setting, `all` by default; `files` keeps live legs on the relay),
  through the relay otherwise: one session per pair (the lower
  node id dials; 14 peers dialled at most, the nearest); a hello with an
  address check; backoff 5 s → 10 min; renewal after 23 h, switching at
  the new session's first frame on both sides (the serving side keeps a
  working session until the renewal's first frame: no `lan.peer_down`
  then `lan.peer_up` at each renewal); a 25 s keepalive (a ping); a session
  given up is ended and drained ≤ 2 s before it is closed; a learning
  switch (floods share one buffer); `broadcast_route`; a per-peer token
  bucket (`rate_kbit`, 4 Mbit/s, `0` = none) cut by a latency guard (a
  ping every second while busy; over a 1 s round trip the rate is halved
  down to 64 kbit/s, under 400 ms it grows back a tenth per ping); a
  warning at `peer_up` unless `expect_direct = true`; `allow` (alias
  `peers`) as an allow-list; `netns = "3pmlan"`: a namespace holding
  only the lan (a macvlan `3pmlan1`, its own address and MAC, no default
  route) for Wine games that bind to the default route's interface; the
  MAC set after udev settles, then read back. Action `lan.status`;
  metrics `lan.frames`/`lan.bytes`/`lan.dropped`/`lan.peers`/`lan.rtt`;
  events `lan.peer_up` (with `call_id`)/`lan.peer_down`. Trust boundary:
  the bucket (a member can claim another's id until the daemon passes
  actions a verified caller).
- **`[modules] off = [...]` in `config.toml`** switches `auto` modules
  off: never auto-installed, and uninstalled at the next `3pm update` or
  spawned daemon (apps deleted, copy removed, nothing written to
  `modules.off`). The daemon checks the section with the rest of the
  file.
- **Signed releases.** `just publish` writes `v<X>/release.txt` (version,
  sha256 of every archive and of a new versioned `v<X>/install.sh`) and
  `v<X>/release.sig` (ed25519, signed on the publishing Mac with
  `~/.3pm-release/main.key`; `just release-keygen` makes the main and the
  offline rescue key). The `autoupdate` module — the `update` action and
  the hourly check alike — verifies the manifest against its two embedded
  public keys, the version (the one asked for, strictly newer), then the
  archive and `install.sh` hashes, and runs that `install.sh` with
  `THREEPM_ARCHIVE=<verified archive>`; anything else runs nothing.
  `install.sh` honours `THREEPM_ARCHIVE`, makes a missing `checksums.txt`
  fatal and no longer falls back to GitHub (nor to a hard-coded version).
  A fresh `curl | sh` stays trust-on-first-use.
- **`3pm fleet ssh`: only the caller's own key opens a session.** A node
  runs a session (shell, `-- cmd`, `cp`, `attach`, `-L`, `-W`, `-R`,
  `--user`) only for a caller key listed in its own `authorized_keys`
  (`<agent state dir>/authorized_keys`, sshd `StrictModes`, re-read per
  session; absent or empty = every session refused), and only for the
  request that caller placed: the handshake's prologue (label `… v2`)
  binds the canonical request — argv, `pty`, `--user`, attach target,
  `host:port`, the `-R` listen/connection, `bind:port`, `term`/size,
  `record`. Refusal: agent code `key_not_authorized` naming the
  fingerprint, exit 255 in every mode (`-L`/`-W`/`-R` stop at the first).
  New commands: `3pm link authorize|deauthorize`, `--authorize` on `3pm
  link` (and `THREEPM_LINK_AUTHORIZE`), keys in `3pm link status`;
  `3pm fleet key` (works without a link), `fleet key new --out`, `fleet
  key push|revoke <node…|--all>`; `THREEPM_CLI_KEY` for a rescue key.
  `cli.key` moves beside `known_nodes` (`$THREEPM_HOME/cli.key`) and
  survives `3pm link delete`. The key is checked before the session
  takes its permit (16 handshake slots, 8 per caller so a stolen token
  cannot hold them all, 5 s for the first message).
  `--record` asks for recording (off by default, see Changed), the
  command line then says "this session is recorded", and `[shell]
  record = "never"` in `link.toml` wins on the node. Tablets: `TABLOID_AUTHORIZE` / an unconditional
  `authorize` provisioning step.
- **`autoupdate` only moves forward**: `update` resolves the version in
  Rust (`{}` = the latest) and refuses one that is not strictly newer;
  going back is `sh install.sh <version>` by hand.
- **Rust SDK (`threepm-io` 0.50.0): stream sessions.** `io().stream(name,
  Action::new().idle_ms(ms), |session, params| …)` declares a stream
  action (`stream: true`, the idle period as its `timeout_ms`, ≤ 24 h;
  the concurrency slot held up to 24 h); the handler runs on its own
  thread, and its return closes the session and answers its data — or
  the session's stats for `Value::Null`, or the leg's failure.
  `io().open(node, app, action, params, Open::new())` places the `call`
  (`stream: true`, `input: {live: true}`, instance 0) and returns a
  `Session` once the ticket is in (≤ 5 s, else `daemon_too_old`); `None`
  and `"self"` are this node, by its `link.toml` id (`io().node()`).
  `Session`: `send` / `send_frame` (never block), blocking `recv`
  (`Ok(None)` at the other side's `END`), `end` (half-close, ≤ 3 s for
  the agent's answer), `close` (also unblocks a `recv` in another
  thread), `stats` (`sent`, `dropped`, `bytes_out`, `received`,
  `bytes_in`, `missing`, `discont`), `failure`, `result(wait)`. The legs
  are the Node SDK's: a chunked `PUT` and a `GET ?live=1` on the agent's
  `data.sock` (`$THREEPM_DATA_SOCK`, `agent/data-<uid>.sock`,
  `agent/data.sock`), one HTTP chunk per frame written as soon as it is
  sent, the 100 ms / 8 MiB abandon sink (oldest non-`KEY`, then
  non-`INIT` first), an early `401`/`422`/`503` failing the leg at once,
  a refusal after the ticket failing the session, and a broken leg
  settling the call `lost` 10 s later. `CallError {code, message,
  call_id}` carries the wire codes; `threepm_io::live` is the framing,
  gated on `live_frames.json`. At most 16 sessions per `Io` and 16 calls
  in flight; a closed session's call stops counting against them. Unlike
  the Node SDK, a control reconnect does not fail an open session: a call
  that has its ticket gets the reply grace and is `lost` only if no reply
  came by then (one still waiting for its ticket fails at once). The
  writer sends one chunk per frame as soon as it is queued, what is
  already queued in one vectored write; `send_frame` moves its payload
  and `send_shared(&Arc<[u8]>)` shares one across sessions. Every leg
  must carry a frame at least once a minute (the reading leg's 60 s
  guard). Still not there: `io().call` and blob inputs/outputs.
  Example: `packages/rust/examples/stream_echo.rs`, proven against a real
  daemon, agent and backend with a Node caller
  (`3pm-backend/tests/it/real_agent_streams_rust.rs`).
- **`3pm fleet ssh <node> -R [bind:]port:host:hostport`: a port of the
  node forwarded back here, as ssh's.** Repeatable, with a shell or
  `-N`; refused with `-W` or a command. The node listens on its
  loopback (port 0: it picks one, `Allocated port N on <node>` on
  stderr) and hands each connection back; this side connects to
  `host:hostport` and carries it over a sealed `nodeForward`
  (`accepted`). The command line places every call, so the end-to-end
  channel, TOFU and the `3pm:forward` rule are `-L`'s. A node binds
  anything but its loopback only when its own `link.toml` says `[shell]
  gateway_ports = true` (never the backend's to change); a re-link
  keeps the setting. A connection not taken over in 10 s is closed; a
  local connect that fails is one stderr line and closes the node's
  connection at once. Ctrl-C or the shell's end closes the port.
- **Link (additive): the wire of `-R`.** A `listen {bind, port, input,
  output, from, from_hostname?, peer?, e2e}` command under
  `node:forward`, whose sealed legs carry `threepm_link::listen`
  messages (`keepalive`, `bound {port}`, `accepted {conn, peer}`; an
  unknown kind is ignored), and `accepted: {listen, conn}` on `forward`
  in place of a destination. An older agent answers `unknown command`.
  `link.toml` gains an optional `[shell]` table (`gateway_ports`,
  absent = off). The Node SDK's generated client gains `nodeListen`.
- **`3pm fleet ssh <node> -W host:port`: ssh's `ProxyCommand`.** stdin
  and stdout through the node to `host:port`, over `-L`'s sealed
  `nodeForward` (no node or backend change): `ssh`, `scp`, `rsync`,
  `git` and VS Code reach a node's sshd with no route to it — `Host
  *.3pm` / `ProxyCommand 3pm fleet ssh $(echo %h | sed 's/\.3pm$//') -W
  localhost:%p` in `~/.ssh/config`. stdout carries data only; exit 0 on
  a clean end, 255 on a refusal or a cut. Refused with a command, `-L`,
  `-N`, `-t`, `-T`.
- **`3pm fleet ssh` is end-to-end encrypted, every mode.** The shell,
  `-- <cmd>`, `fleet cp`, `fleet attach` and `-L` are sealed between the
  command line and the target's agent (`Noise_XX_25519_ChaChaPoly_BLAKE2s`,
  prologue = call id + node id; records `[u16 length][Noise message]`
  on the existing lossless legs, relay or LAN direct): the backend
  relays bytes it cannot read nor change — a record that does not open
  ends the session (`e2e_broken`, exit 255). Keys: `shell.key` in the
  agent's state directory (fingerprint in `3pm link status`, "shell
  key"), `cli.key` in the caller's `~/.3pm/agent/` (its fingerprint,
  `cli_key`, on the audit `open` line — metadata, not a credential).
- **Trust on first use: `~/.3pm/known_nodes`, `--accept-new`, `3pm fleet
  known-nodes ls|rm`.** The first session to a node asks on `/dev/tty`
  (refused with no terminal unless `--accept-new`); a key that changed
  is refused, ssh's `NODE IDENTIFICATION HAS CHANGED`, exit 255. Never
  filled from the backend.
- **The node records an end-to-end session itself.** Its agent writes
  the asciicast from the screen leg **before** sealing (the writer moved
  to `threepm_protocol::cast`, shared with the backend: same rules, 64
  MiB cap, never waits), uploads it as the call's artifact
  (`putCallArtifact`) and names it in `exited {recording}`; `record:
  false` in the request (the bucket's `record_shells` off) records
  nothing. A sealed session may go LAN direct and be recorded. An agent
  killed mid-session loses that recording (the backend's `ended` says
  `recording: lost`); its temporary file is swept at the next start.
  A recording that stopped early (cap, gap) says why on its upload
  (`x-3pm-stopped: cap|gap|gap_end`), so its `summary` matches one the
  backend made. A copy (`3pm fleet cp`, up or down) is never
  recorded, by the agent nor the backend (`threepm_link::cp`: its four
  fixed scripts, matched byte for byte, `is_cp_transfer`): a
  download's screen is the file. The audit keeps its command line and
  paths; no `ended` line says `lost` for it. `-L` is never recorded.
- **Link (additive):** `e2e` and `record` on `shell`, `exec`, `attach`,
  `forward` and `run_as`; `shell_e2e` announced in `hello`; `cli_key`
  on `ShellEvent::Open`; `recording` on `exited`; agent error codes
  `e2e_required` and `e2e_broken`.
- **`--no-e2e`**, for a node whose agent cannot encrypt only (the
  backend refuses `e2e` to it at once, `ERR_NODE_NO_E2E`): the session
  goes in clear, as before, after a warning on stderr.
- **`3pm fleet ssh` between LAN neighbours goes agent to agent.** When
  both nodes are `direct: all` (and, for a session in clear, the
  bucket does not record shells),
  the command line hands the session's legs to its own agent
  (`POST /legs` on `data.sock`, its own socket only; a lossless `PUT
  ?pipe=1`), which dials the target for the screen while the target
  dials it for the keyboard; the relay stays the fallback after a 1 s
  hold, lossless. Wire: a lenient `peer` on `Shell`, `Exec`, `Attach`
  and `Forward` (inside `RunAs` too), additive. Shell, command, attach
  and `cp`; `-L` forwards stay on the relay.
- **`3pm fleet ssh -L [bind:]port:host:hostport <node>` (repeatable),
  and `-N` for the forwards alone: a local port forwarded through a
  node, like ssh's.** One listener per `-L` (127.0.0.1 unless a bind
  address is given), each accepted connection its own call; the node
  connects to any destination it can reach. A failed connection is one
  line on stderr, never the end of the others; with `-N` the command
  runs until `SIGINT`/`SIGTERM` (exit 0). Needs `3pm:forward` in the
  target's `callers` and `node:forward` in its `caps`. A listener off
  the loopback (`*`) is warned about; at most 16 connections are
  carried at once (the target's sessions, shared with shells), the next
  ones waiting.
- **Link: the wire of `3pm fleet ssh -L`.** A new capability
  `node:forward`, in the default set, and a `forward {host, port, input,
  output, from, from_hostname?}` command: two `pipe` legs of raw bytes
  in `live` messages, `END` a half-close, answered by `exited {code}`
  or the `connect …` error. Additive: an older agent answers `unknown
  command`. `link.toml` goes to version 4 with the rule of version 3: a
  version-3 file listing exactly the five capabilities of that day
  reads as all of them (so **a node linked with the defaults gains
  `node:forward` on update without a re-link**); any other list stays
  as written. Take `node:forward` out of `caps` to refuse forwards.
- **`3pm fleet ssh -l <user>` (or `<user>@<node>`), also on `-- <cmd>`
  and `3pm fleet cp -l <user>`: the session as another unix account of
  the node.** An agent running as root becomes it (its groups, home,
  shell; `setgroups` → `setgid` → `setuid`, uid 0 checked out of reach,
  `PR_SET_NO_NEW_PRIVS` on Linux, the pty chowned to it — the daemon's
  hosted-user drop, copied); one that is not root takes only its own
  name and refuses any other; an unknown account is refused. Link: a
  new op `run_as {user, req}` wrapping a `shell` or an `exec`, so an
  older agent answers `unknown command` and never runs the session as
  itself. The name is checked everywhere (`valid_user_name`,
  `[a-z_][a-z0-9_-]{0,31}`). A refusal the agent sends before the legs
  open is now read back and printed, instead of "the screen leg was
  refused".
- **`3pm fleet attach <node> <app>`: the terminal of a `tty = true` app
  on another node**, like `3pm attach` (`--instance`, `--raw`, `--peek`,
  `--read-only`; `~.` after Enter detaches and the app runs on; exit 0
  on a detach, 255 otherwise). A new link op `attach` (capability
  `node:shell`): the target's agent relays the daemon's own `attach`
  over a fresh `rpc.sock` connection, message for message; audit
  `open`, `attach` (new event kind), the raw input lines and `close`.
- **Link: the wire of `3pm fleet ssh`.** A new capability
  `node:shell`, in the default set; a `shell {term, cols, rows, input,
  output, from, from_hostname?}` command (the two blobs are the
  keyboard and screen legs) answered by `exited {code?, signal?}`; and
  a `shell {call, from, ts, event}` audit event whose `event` is
  `open`, `line`, `screen` or `close`. Additive: an older agent answers
  `unknown command`, an older backend never sends it. `link.toml` goes
  to version 3, and **a node linked with the default capabilities gains
  `node:shell` on update without a re-link**: a version-2 file listing
  exactly the four earlier capabilities reads as all of them, any other
  v2 list stays as written. Where the agent runs as root that is a root
  shell, opened by whoever the target's `callers` rules admit: take
  `node:shell` out of `caps` to refuse it. The agent serves it, and
  `3pm fleet ssh` opens one (below).
- **`3pm fleet ssh <node>`: an interactive login shell on another node
  of the bucket, like ssh.** The node is a `3pmn_…` id, a hostname or
  an unambiguous start of one; the session is placed with this node's
  own identity, like `3pm fleet call` (the `invoke` scope on the
  target, and its `callers` rules must name `3pm:shell` when it has
  any), and both directions are lossless legs through the backend. The
  terminal is raw for the session and given back on every way out
  (return, error, a panic on any thread, a `SIGTERM`, `SIGHUP`,
  `SIGINT` or `SIGQUIT` sent to the command, which then exits 255); its size follows the window
  (`SIGWINCH`), `TERM` is passed through. `~.` right after Enter
  disconnects, `~~` types one `~`. The exit code is the shell's (`128 +
  signal` when a signal ended it), and 255 with one line on stderr when
  anything else did: `~.`, a leg cut, the node refusing. Refusals say
  which rule is missing — the `invoke` grant, `3pm:shell` in the
  target's `callers` (by the backend's `ERR_SHELL_NOT_ALLOWED`, not its
  wording), `node:shell` off in the target's `link.toml`, an agent or a
  backend too old, the node offline, its shells all in use. The keyboard
  pings every 20 s so an idle session is not swept, and is queued
  rather than awaited, so a paste into a shell that does not read never
  stops the screen. Needs a local daemon and a linked host, like every
  `3pm fleet` command.
- **`3pm fleet ssh <node> -- <cmd>`: one command instead of a shell.**
  Through the target's login shell (`sh -c`; several words each
  quoted, one word taken as the whole line). On a pty when stdin and
  stdout are terminals or with `-t`, never with `-T`; otherwise on
  pipes: stdin streamed then closed, stdout to stdout and stderr to
  stderr, binary-safe, no raw mode and no `~.`. Exits with the
  command's code (`128 + signal`; 141, silently, when stdout stopped
  being read, `… | head -1`; 255 when the session ended any other
  way). `echo x | 3pm fleet ssh n -- wc -c` prints 2.
- **`3pm fleet cp [-r] [-p] SRC… DST`: files to and from a node, like
  scp.** One side is `<node>:<path>` (a node as `fleet ssh` takes it;
  `<node>:` is its home, `~/x` works; a local name with a `:` is
  written `./a:b`); several sources need a directory. Built on the
  command sessions of `fleet ssh -- <cmd>`, one per source, so the same
  grant and `callers` rules apply and the audit shows each command (`sh
  -c '…' 3pm-cp <paths>`; a node path starting with `-` is never an
  option). A file goes up through an owner-only `<dst>.3pm-tmp-<rand>`
  renamed over its destination, and comes down into a temporary file
  here renamed once the node's command exited 0: a cut never leaves half
  a file under its name. `-r` streams a tar archive both ways (built and
  unpacked here by the `tar` crate, never the local `tar` binary; the
  node's own `tar` there); `-p` keeps modes and times (a file down with
  `-p` travels as a one-member archive). A file is never written over a
  directory; symlinks inside a tree stay links. Exits 0 when every source
  was copied, else 1 with one line per failure; the node's stderr is
  shown as it comes. A progress line (bytes, rate) on stderr when it is
  a terminal. 100 MB up and back on loopback (Linux, debug builds):
  about 0.2 s up, 0.6 s down.
- **Link: `exec`.** A new command `exec {argv, pty?, term, cols, rows,
  input, output, from, from_hostname?}` under `node:shell`, answered by
  `exited`, and a `shell` audit event `exec {command, pty}`; two new
  `attach` kinds, `eof` (`0x05`, caller → target: close the command's
  stdin) and `stderr` (`0x1d`, target → caller). All additive: an older
  agent answers `exec` with `unknown command`, an older reader ignores
  the new kinds and the new event.
- **Agent: the target side of `3pm fleet ssh`.** A `shell` call opens
  the login shell of the agent's own unix user (read in `passwd`, never
  `$SHELL`; `/bin/sh` when it names none) on a fresh pty, as a login
  shell (`-bash`), in its home, with an environment built from nothing
  (`TERM` from the caller, `HOME`, `USER`, `LOGNAME`, `SHELL`, a
  default `PATH`, the agent's `LANG`/`LC_*`): nothing of the agent's
  (`THREEPM_HOME`, the module variables) reaches it, and the shell
  holds no descriptor but its terminal. Its keyboard and screen go
  through the backend on the two blobs of the call, opened with the
  node's bearer of the moment. **Either leg ending, or the shell
  exiting, ends the session**: `SIGHUP` to the shell's process group,
  `SIGKILL` 5 s later if it is still there; the call answers `exited
  {code?, signal?}`. A keyboard leg silent for 90 s (the caller pings
  every 20 s) counts as ended: a backend gone with its host does not
  hold a shell for the call's day. **A shell survives a reconnect of
  the link**: it runs beside the connection, not in it, and answers on
  whichever connection is live when it ends. 16 shells at once per
  agent (their own count, across reconnects; refused `unavailable` past
  that), refused `forbidden` without `node:shell`. **Audit**: every command line, as the screen
  showed it (after `Tab`, after a history recall, prompt included),
  goes to the backend as a `shell` event and to the agent's log; a
  line typed while the terminal does not echo (a password) is
  `[hidden]`; under a full-screen program (vim, htop) the raw keys are
  logged instead, with an event on entering and leaving it — **so a
  secret typed into a full-screen form (`nmtui`, `pinentry-curses`) is
  logged as typed**; 4 KiB a line at most; the audit never takes more
  than half the link's control queue (a flood drops lines, never
  another call's frames). Not complete, and not claimed to be: `tmux`, an escape
  sequence or typing blind escape it — the backend records the opening
  and the end itself. **An agent that stops (`3pm restart agent` or
  `3pm update`, typed in that very shell included) hangs its shells up
  before it closes the link** and waits, at most 1.2 s, for each to
  deliver its end: the caller exits 129 (`SIGHUP`) and the call settles
  `exited`, rather than 255 and a call left open; the restart itself
  goes through. A shell that ignores the hang-up is killed 0.5 s later
  (not 5 s) by the agent still running: a shell is a session of its
  own, which the daemon's kill of the agent would not reach. Works on a macOS target as on Linux.
- **Hosted users read the host metrics.** `host_metrics` (`3pm
  hostmetrics`, the SDK's `hostMetrics()`) no longer answers
  `forbidden` to a hosted account — a kiosk running as `srt` can show
  the machine's CPU, memory, network and temperatures. Its reply drops
  the fields that name other accounts: `top_processes` (root's
  processes included) and `sessions`.
- **Per-core CPU frequency: `HostMetrics.per_core_freq_mhz`.** One
  MHz figure per logical CPU, in `per_core_percent`'s order, refreshed
  every fast tick (1 s); until now only cpu 0 was live
  (`cpu_frequency_mhz`, unchanged). Additive on the wire and the link.
- **Network traffic per app: `net ↓/↑` in `3pm ls`.** Received and
  sent per second over the last 5 s for every app (its whole process
  tree, each socket once, its traffic with the outside: a connection
  between two of its own processes counts for each process, not for the
  app), per process under `3pm ls --tree` and in the ` Processes` table
  of `3pm show`, with the totals in `3pm show` and `ProcessRecord.net` / `TreeProc.net` in
  `--json` (`rx_bytes_per_s`, `tx_bytes_per_s`, `rx_bytes`, `tx_bytes`,
  `partial`). Measured by the daemon: on Linux a `sock_diag` dump of
  the TCP sockets with their byte counters, the members' fd tables, and
  the kernel's destroy notifications so connections that close between
  two windows still count (an inbound one by its listening port; an
  outbound one no window saw is not counted); on macOS one `nettop
  -d` per socket kept running on a pty (loopback counted, the pairs
  inside an app taken out). Unprivileged, and
  what it costs follows the apps, not the host: both kernel streams are
  filtered on the apps' ports (`inet_diag` bytecode, a BPF filter on the
  notification socket), fd tables are read on a 10–30 s rhythm, and an
  idle daemon holds no socket, thread or `nettop`. `3pm
  internals` gains `network meter` (`DaemonInternals.net_meter`). The
  link carries it (`RedactedRecord.net`, `RedactedProc.nt`, additive)
  and tree rows sum it (`CompactRow.net`). The `ls` column is drawn
  when some app has a figure, and dropped right after `ports` on a
  narrow terminal. `tree::budget_ext` is now `budget_extras` (it sheds
  `net` too). `metrics/sock_diag.rs` is the netlink transport the
  listening ports and the meter share.
- **`3pm install <name>` downloads a module that is not bundled.** A
  name that is neither a path, nor bundled, nor installed is looked up
  on the download site (`$THREEPM_CDN_URL`, the one `autoupdate` uses):
  `modules/<name>/latest/version.json`, the archive for this platform,
  its size bounded and its sha256 checked before it is opened, every
  entry checked on unpack (no absolute path, no `..`, nothing outside
  `<name>/`, no link out, the manifest naming `<name>`), then the same
  install as `3pm install ./dir`. `<name>@<version>` pins (always the
  site), `<name>@latest` unpins; the installed copy carries
  `.3pm-site.json`, and `3pm update` takes a newer release of every
  unpinned site module in place (a site out of reach is a warning).
  https only, a `file://` or loopback mirror aside; the archive URL is
  rebuilt from the site, never taken from the index. `3pm install
  automonit` replaces the `curl … | sh` line in `just publish`'s
  summary, `version.json`'s `install` and the site's home page; the
  script stays for a 3pm up to 0.42.0. The site helpers
  (`$THREEPM_CDN_URL`, version grammar, `curl`/`wget` fetch) moved from
  the `autoupdate` module to `threepm_protocol::site`, shared by both.
- **`3pm rename <app> <new-name>`: rename an app in place, without
  restarting it.** Same ids, pids and counters; every instance, the pid
  files and the log files named after the app (live and rotated, the
  descriptor kept open) follow the name, and so do the running pumps'
  events. Refused on several apps, a taken name, a module app, or a log
  file already at the new name. Wire, additive within protocol v2:
  `Request::Rename { target, name }` → `ok`. Node SDK:
  `client.rename(target, name)`.
- **Interactive apps: `tty = true` (`--tty`) and `3pm attach`.** An app
  can run on a pseudo-terminal the daemon holds (`isatty` true, its own
  session, 120×40, `TERM=xterm-256color`, `COLORTERM=truecolor`), and
  `3pm attach <app>` (alias `console`; `--instance`, `--read-only`,
  `--raw`, `--peek [--json]`, `--paste`) connects a terminal to it:
  the daemon keeps a screen per app (1 000 rows of history, answers to
  the app's terminal queries), sends each client the difference against
  a mirror of its terminal, forwards the history into the client's own
  scrollback, and follows the alternate screen; `Ctrl-Z` detaches and
  never reaches the app, and the client's terminal is restored on every
  way out (a signal included). Piped, stdin goes to the app and its
  bytes come out; `--peek` prints the screen. The log of a `tty` app is
  one file, stripped of escape sequences (`threepm-core::ansi`);
  `error_file` is refused with `tty`. The terminal and the screen cross
  `3pm update` (same pid; attached clients re-attach by themselves).
  Wire, additive within protocol v2: `Request::Attach`,
  `Response::Attached`, `ErrCode::NotTty`, `ProcessRecord.tty_viewers`,
  and the attach kinds on the live framing (`threepm_protocol::attach`,
  `docs/protocol.md` § *Attach*). A `tty` app dies with a daemon that
  crashes (`SIGHUP`); the other limits are listed in `FEATURES.md`. The
  Node SDK has no `attach` method, deliberately: a terminal session is
  not a command a script calls like `restart` (`StartOptions.tty` is
  there, from `just gen`).
- **`just vendor`**: fmt, clippy and the tests of the vendored crates
  (`vendor/vt100`), part of `just ci` and of CI's `modules` job.
- **Top processes in `3pm hostmetrics`, on by default.** Which
  processes of the whole host — not only the apps 3pm manages — take the
  CPU, the memory and the network: `HostMetrics.top_processes`, one
  table of at most 30 rows sorted by pid (the ten largest by CPU, by
  memory and by network rate, merged), each row `pid`, `name`,
  `user`, `cpu_percent` (CPU time over the window, 100 = one core),
  `mem_bytes` (RSS), `footprint_bytes` (macOS) and
  `net_rx_bytes_per_s` / `net_tx_bytes_per_s`.
  `3pm hostmetrics` draws it as three sections (cpu, memory, network),
  ten rows each.
  - **A tier of its own**, `top` (`THREEPM_HOSTMETRICS_TOP_MS`, 5 s,
    floor = fast), outside the first-snapshot gate;
    `HostSampledAt.top` dates the table. `[hostmetrics] top_processes`
    in `config.toml` (default `true`),
    `THREEPM_HOSTMETRICS_TOP_PROCESSES=0|1` above it, read once per
    daemon image; off, nothing is forked, walked or sent.
  - **Sources.** macOS: `ps` (the one unprivileged way to every user's
    processes) and `nettop -n`. Linux: a `sysinfo` walk without threads
    and `ss -tinpH` — TCP only, every socket under a root daemon, the
    daemon's own user's otherwise; a socket several processes hold
    counts whole for each. Each fork has 2 s. A rate the platform cannot
    give is absent, never zero.
  - **Names, never arguments or paths.** macOS publishes the kernel's
    16-byte name, completed from the executable's file name when it may
    be cut (never from the process's own title); Linux the kernel's
    process name.
  - **Memory: the footprint on macOS, the RSS on Linux.** The RSS
    misses a Mac's biggest consumers (a VM at 8.2 GB of footprint shows
    803 MB of RSS; WindowServer, at 4 GB, fell out of the top ten), so
    macOS ranks memory on `footprint_bytes` — what Activity Monitor
    calls Memory — read from one `top -l 0 -s <period> -stats pid,mem`
    the host agent keeps running: ~1.8 % of a core at 5 s (measured
    2026-10-01; a `top` per tick would cost 15 %). A process `top` did
    not list has no memory rank rather than its RSS; with `top` down,
    memory ranks on the RSS. `top` starts with the tier, is reaped with
    it, is killed when it stops writing, and is started again at most
    once per 30 s. Linux keeps the
    RSS. `3pm hostmetrics` heads the column `footprint` or `rss`.
  - A process has a row once it has been seen twice (no figure is
    invented for a first sample), so one that lives less than a period
    never shows; a failed read empties the table.
  - **Link:** the table rides the `host` section, keyed by pid — no
    protocol or link version change; a patch carries the rows that
    moved (about 200 bytes deflated for 24 rows, measured on synthetic
    data).
- **Extended Monitoring, on by default.** `[hostmetrics] extended` in
  `config.toml` (default `true`), `THREEPM_HOSTMETRICS_EXTENDED=0|1`
  above it, read once per daemon image and handed to the host agent so
  both agree; `false` reads no extra file and sends nothing extra.
  - **Host (Linux):** a new tier, `extended`
    (`THREEPM_HOSTMETRICS_EXTENDED_MS`, 10 s, floor = fast), adds
    `HostMetrics.extended`: the kernel's own counters by family —
    `cpu`, `softirqs`, `meminfo`, `vmstat`, `diskstats`, `netdev`,
    `snmp`, `netstat`, `snmp6`, `sockstat`, `pressure`, `ksm`, `hwmon`,
    … (27 declared on this tier, those a node lacks cost one failed
    `open`), plus `cpuidle`, `cpufreq`, `thermal` on the slow tier. Kernel names kept;
    a file mixing counters and gauges split in two (`vmstat` /
    `vmstat_gauge`); nothing a typed field already carries is sent
    twice. 23 families / 1 228 values on edgecr4ft.net. macOS: present
    and empty. `HostSampledAt.extended` dates the tier.
  - **Short-lived processes (eBPF, Linux, root agent):** families
    `shortlived` (execs, exits, exits under 1 s, lifetimes, exits
    without a seen exec, full-map refusals), `shortlived_by_comm` (the
    10 busiest names per window + `other`) and `shortlived_by_comm_max`.
    One prebuilt, CO-RE-free object embedded in `3pm` (`just bpf`
    rebuilds it); loaded once per agent start, never retried.
  - **Why a source is missing:** `extended.unavailable` —
    `permission`, `error`, and for eBPF `not_root`, `bpf_disabled`
    (`ENOSYS`), `load_failed`. Tried per read, never an euid check;
    eBPF alone checks the agent's capabilities before loading.
  - **Per process:** tree members carry `ext` (faults, context
    switches, storage and logical I/O, syscalls, threads, fds + limit,
    PSS, swap on Linux; footprint, page-ins, disk I/O, threads, fds on
    macOS), read on first sight then every 10 s, each field absent
    rather than zero; the app record gains `pss_bytes`, the tree's PSS
    (Linux, only when every member's was read). A tree reply sheds
    `ext` from its tail past its byte budget (`tree::budget_ext`).
  - **CLI:** `3pm hostmetrics --extended` prints the families (one
    table each, then the `unavailable` table) instead of the overview;
    `--json` carries everything. `3pm show` reads `mem 4.8mb RSS ·
    696.0kb PSS (4 processes, sums)` and its ` Processes` table gains
    `pss` (or `footprint`), `threads` and `fds`. With the mode off both
    render exactly as before.
  - **Link cost, measured** (edgecr4ft.net, debug build, same host and
    same 4 apps, two nodes side by side for 14 min): **777 B/s up with
    the mode on, 529 B/s off: +248 B/s (+47 %)**, far under the
    +~1 200 B/s estimated; ~30 B/s more with a root agent (eBPF
    families). Backend to node: unchanged (< 1 B/s). Stored series per
    node: 2 436 on vs 663 off (+1 228 host, +~380–545 per-process
    series seen in a 2-min window, mostly a forking test app's children
    (~220 at steady state)).
  - Wire additive only (`HostExtended`, `TreeProcExt`, link
    `RedactedProc.x`, `RedactedRecord.pss`): no protocol or link
    version bump; an older peer ignores the members. The agent module
    carries them (rebuild it with the daemon).
- **pm2 compatibility: `3pm jlist` and `3pm prettylist`.** The process
  list in pm2's shape (`pm2_env.status`, `restart_time`,
  `NODE_APP_INSTANCE`, `exec_mode`, …), as JSON or as Node's
  `util.inspect`, so scripts that read pm2's dumps read 3pm's; module
  apps left out; hidden from `--help`.
- **Module toolkit: daemon replies and shutdown signals.**
  `threepm_protocol::module::{affected, unexpected}` turn a daemon reply
  into the rows it affected, or into the one sentence every module fails
  with (`daemon refused the stop (not_found): …`, `run 3pm update` on an
  unknown reply); `shutdown_signal()` resolves on SIGINT or SIGTERM, its
  handlers installed at the call so an early signal is not lost. `serve`,
  the agent and the host agent use them in place of their own copies.
  `ErrCode::as_str` / `Display` give a code's wire name.
- **Rust SDK (`threepm-io` 0.37): metric families, removal, usage.**
  `io().gauge_family(name, Metric)` (and `counter_family`,
  `histogram_family`) declares a name with **no series** until
  `.labels(…)` makes a child — `gauge()` reports its unlabelled series
  from the first flush, a phantom for a process that only ever sets
  children. `Series::remove()` and `Family::remove_all()` stop reporting
  a series (the daemon drops it after its two-minute TTL); `io().usage()`
  gives the names and series held and what was refused, folded or
  removed, so a process that makes series from what it discovers budgets
  against the caps. The Node, Python and C++ SDKs do not have it yet
  (`packages/rust/ROADMAP.md`).
- **Sticky metric meta.** The daemon now keeps a series' unit and
  description when an entry leaves them out, and announces
  `THREEPM_CTL_CAPS=sticky_meta` to every app it starts; the Rust SDK
  reads it and sends them once per series and connection (again after a
  reconnect) instead of in every batch. An older SDK keeps sending them;
  an older daemon never sets the variable.
- **Install and link in one line.** `curl -fsSL …/install.sh | sh -s --
  link 3pmc_… 3pmk_… wss://…` installs 3pm then runs `3pm link` with
  the words after `link`; optional (`sh -s -- [version] [link …]`), and
  `THREEPM_LINK_TOKEN` in the environment links too. A refused link
  keeps the install and exits 1. **Changed with it:** `3pm link` with
  the line of the current link (same bucket, same server) now exits 0,
  `Already linked … — kept` (`"kept": true`), instead of failing;
  another bucket or server still needs `--force`.
- **Python SDK (`threepm-io`): actions and calls, with bytes.**
  `io.action(name, handler, input=…, output=…)` exposes a Python
  function to `3pm trigger`, `3pm call`, any SDK's `io.call` and the
  fleet backend; `io.call(node, app, action, params, file=…, output=…)`
  places one, on this machine or on another node of the bucket, and
  `io.serve()` blocks a worker that only answers. Same wire as the Node
  SDK — `call` / `call_ticket` / `call_reply` on the control socket, the
  bytes `PUT`/`GET` on the agent's `data.sock`, size and sha256 checked
  before a handler sees a file — so a Python process and a Node one call
  each other. Each call runs in its own thread; `call` blocks and raises
  `CallError` with the wire code. Measured Mac → fleet → a Linux node:
  a 2 MB input answered in 175 ms, 2 MB in and 2 MB out in 351 ms. Not
  yet: streams, `input.stream`, `3pm:profile`. **Changed with it:**
  `init()` without `source` now takes `main` (`pid-<pid>` for a process
  it started), as in Node — a sourceless connection cannot declare an
  action — so the series of a Python app that named no source gain
  `src="main"`.
- **A module the release installs by itself: `[module] auto = true`.**
  A bundled module marked `auto` is part of 3pm the way a built-in
  command is: 3pm does what `3pm install` would have done — copy, start
  the apps, save — without being asked, on a daemon the CLI just
  spawned (a fresh machine's first command, a boot without a startup
  service) and after a `3pm update` (how a host already running learns
  a release added one). The host has the last word and it is durable:
  `3pm uninstall <name>` writes the name into `<home>/modules.off` and
  nothing puts the module back until `3pm install <name>` forgets the
  refusal. A module that will not install is a warning, never a failed
  command; only the bundle's manifest is read for this; a 3pm that
  predates the key ignores it. `autoupdate` is the first module to use
  it.
- **`autoupdate`: a node updates itself on a custom action.** A third
  bundled module, whose whole surface is two actions — it installs
  itself (`auto = true`, above), and from then on
  `3pm trigger autoupdate update` (or the same call from the fleet
  backend, or `io.call`) downloads `install.sh` from the release site
  and runs it: the new binary replaces `~/.3pm/bin/3pm` and its bundled
  modules, and `3pm update` hands the daemon off to it without stopping
  the apps. `check` answers `{current, latest, outdated}` from
  `latest/VERSION` on the same site. `$THREEPM_CDN_URL` moves both
  elsewhere, `curl` (or `wget`) does the fetching, and the installer is
  written to disk before it is run rather than piped into a shell.
  A caller may name a version (`{"version":"0.31.0"}`), checked before
  it reaches anything that could read it as an option.
  The call answers as soon as the updater is **started** — pid and log
  path — because the update's last step restarts every module row, this
  app included: the updater `setsid`s out of the app's process group and
  outlives the restart, holds `<home>/autoupdate.lock` for as long as it
  runs (a second `update` is refused, not doubled) and writes
  `<home>/autoupdate.log`.
  **And every hour by itself**: the app reads `latest/VERSION` and
  updates the node when it is behind, to that exact version — once per
  published version (`<home>/autoupdate.last`), so an update that does
  not take never loops. `THREEPM_AUTOUPDATE_EVERY` sets the interval
  (`30m`, `90s`) or turns it `off`; the first check comes within five
  minutes of the app starting, jittered.

### Changed
- **`3pm fleet nodes` / `apps` / `inspect` no longer download the
  bucket's history**: the agent reads the bucket with
  `?history=0&view=live` (also `fleet call` resolving a hostname and
  `fleet key`), 2.1 MB → ~65 KB on a 17-node bucket, ~7 s → well under a
  second on a slow link. A backend that predates the query ignores it and
  answers whole, as before.
- **The `3pm fleet` listings draw `3pm ls`'s table.** `fleet nodes`
  (`fleet ls`), `fleet apps` and the process table of `fleet inspect`
  now use the house box style, cyan headers, a greyed frame and a
  colored state, and drop their lowest-priority columns one at a time
  until the table fits the terminal (nodes: `os/arch`, `seen`, `apps`,
  `user`, `private`, `public`; processes: `pid`, `uptime`, `inst`,
  `↺`, `mem`, `cpu`). Only then is the node id or the app name cut,
  with a `…`, never below 12 / 8 characters. The process tables gain
  `pid` and `uptime` and take `3pm ls`'s names (`name`, `status`);
  `fleet top`'s apps panel follows. A box-drawing glyph in a name a
  neighbour chose prints as `·` (in `fleet top` too): it cannot draw a
  column that is not there. The responsive table itself moved into a
  crate of its own, `threepm-table`, which `3pm ls` / `3pm show` and
  the agent share. The agent's `fleet top` moved to ratatui 0.30, so
  the agent carries one crossterm (0.29).
- **`3pm ls` and the `3pm fleet` listings are never cut in a pipe.**
  Written to a pipe or a file (`3pm ls | grep`, `3pm fleet ls | grep
  10.0.`), the table keeps every column, every host-line segment and
  every name whole, without color; it was laid out for 100 columns,
  which dropped columns from `3pm ls`. A terminal is fitted as before.
- **Table widths are terminal columns.** `3pm ls` and the fleet tables
  measure a cell the way the terminal draws it, so a full-width name
  (CJK, emoji) gets a column wide enough instead of wrapping onto a
  second line, and a cut never splits a glyph.
- **The per-app network meter dumps both families in one walk
  (Linux).** Its filtered dump of the apps' sockets goes through the
  same `TCPDIAG_GETSOCK` request as the host agent's (`sock_diag::
  dump_tcp`): the kernel walks its established table once a window
  instead of once per family, whatever the filter. Same sockets,
  checked against the two per-family dumps. Measured: a walk costs
  6.3 ms of kernel time on a 64 GB host, 1.6–2.6 ms on a 94 GB one with
  a 5.4 kernel, where a daemon with 6 apps, one of them listening,
  went 0.445 -> 0.40 s of CPU a minute.
- **The host agent reads the top processes' TCP counters without `ss`
  (Linux).** One `sock_diag` dump of the connected sockets of both
  families with their `tcp_info` — the `TCPDIAG_GETSOCK` request, which
  the kernel answers with one walk of its established table where `ss`
  made one per family (6.3 ms of kernel time a walk on a 64 GB host,
  whatever the number of sockets) — and the readable processes' fd
  tables for the holders, in the process: no fork, no exec, no text.
  The same figures by the same rules (`parse_ss`), checked against
  `ss` on generated tables and on a live connection; a kernel that
  refuses the request is asked family by family, a host that refuses
  `sock_diag` falls back to `ss` for good. Measured on a quiet 32-core
  host (5.4 kernel, ~540 processes), the agent polled every second:
  1.14 -> 0.92 s of CPU a minute, `ss` included.
- **A metrics tick hashes and lists the host's processes for less
  (Linux).** The maps a tick fills with every process of the host
  (the tree's pid index and children, the refresh ticks, the identities
  alive, the probe and backoff memos) hash with a multiply-rotate
  (`metrics/pidhash.rs`) instead of SipHash, and `/proc` is listed with
  `getdents64` into one buffer rather than `readdir`'s copy of each
  entry (the host agent's walks too). Measured on a 32-core host with
  ~1 000 processes, a daemon with 5 apps: 193 -> 160 M instructions a
  minute.
- **Extended Monitoring reads a member's counters for a tenth of the
  instructions (Linux).** Its `/proc/<pid>` directory is opened once and
  each file read relative to it into one buffer, keeping only the lines
  reported, rather than through `procfs` (a map of every line of
  `status`, `limits` and `smaps_rollup`, a `read_to_end` with its
  `fstat` and `lseek`). Same fields and same absences, checked against
  `procfs` field for field (`procfs` is now a test dependency only).
  Measured on a 32-core host, 35 members read every 10 s: 103 -> 8 M
  instructions a minute; the daemon 272 -> 193 M.
- **The daemon reads its process table from `/proc` itself on Linux,
  one file a process.** A tick reads each process it refreshes from its
  `stat` (cpu time, resident set, parent, state, name) and its `status`
  once for the uid, where `sysinfo` read `stat`, `statm` and `status`
  each time, plus an `fstat` and two `lseek`s a file, and the members'
  names again from `comm`. Same figures, computed as `sysinfo` computed
  them (start = `btime` + start ticks, cpu % over `/proc/stat`'s total
  per cpu, re-read at most every 200 ms); the pids alive are listed and
  diffed against the table as sorted lists. macOS keeps `sysinfo`.
  Measured on a 32-core host with ~1 000 processes, a daemon with 5 apps
  of 7 processes polled like a linked fleet agent: 530 -> 272 M
  instructions a minute, ~103 000 -> ~34 000 system calls a minute.
- **The daemon forwards `host_metrics` as the host agent wrote it.**
  The reply line is looked at for its tag and written to the client as
  is, rather than decoded (~50 KB, most of it Extended Monitoring's
  maps) and encoded again; a hosted user's copy is still decoded, to be
  scrubbed. Same bytes on the wire. Measured on a 32-core host, a daemon
  with 5 apps polled like a linked fleet agent (`list`, `host_metrics`,
  `process_trees` every second): 767 -> 530 M instructions a minute,
  0.33 -> 0.26 s of CPU a minute.
- **The host agent parses `/proc/vmstat` and `/proc/meminfo` for a third
  fewer instructions (Linux):** a value is parsed only for the keys it
  reports, not for each of the files' ~230 lines. Same figures.
  Measured: 11.1 -> 8.1 M instructions a minute for the two.
- **The host agent reads `/proc/diskstats` once a second, not twice
  beside `/proc/mounts` and `/dev/disk/by-id` (Linux).** Block devices
  (read as `systemstat` read them) and each mount's I/O counters (as
  `sysinfo` computed them: the resolved device's sectors × 512, the last
  value kept while the device is absent) come from the same read; the
  mounts and their devices are re-listed every 10 s, the period of the
  medium tier's disk rows. Measured on a 32-core host: fast tier 58 ->
  42 M instructions a minute.
- **The host agent parses a process's `stat` line for 40 % fewer
  instructions (Linux).** The `comm`'s closing parenthesis is looked for
  within the 64 bytes the kernel allows a name, not back from the end of
  the line's 52 fields, and the fields are split on the kernel's single
  space. Same fields read. Measured on a 32-core host with ~1 000
  processes: 43 -> 26 M instructions a minute for the parse, agent
  341 -> 329 M.
- **The host agent walks the process table once per top tick, not
  also once per medium tick (Linux).** With the top-processes table on
  (the default), the medium tier's process count and state histogram
  are taken from the top tier's walk, every 5 s, while it is younger
  than the medium period; with the table off, or its tier late, the
  medium tier walks as before. The figures a medium tick reports are
  up to one top period older (5 s by default; never more than a medium
  period). Measured on a 32-core host with ~1 000
  processes: medium tier 38 -> 10 M instructions a minute, agent system
  calls 92 k -> 77 k a minute.
- **The host agent's fast tier makes two thirds fewer system calls for the
  same figures (Linux).** Kernel files are read in one go (`openat`, one
  read, the end-of-file read, `close`) instead of `std`'s size probe
  and doubling reads (five or more for `/proc/vmstat`); per-core
  frequencies come from `scaling_cur_freq` files kept open and re-read
  at offset 0 (one descriptor per core held by the agent); interface
  counters from one `/proc/net/dev` instead of seven sysfs files per
  interface, the MAC and IPs being re-read every 10 s and at once when
  an interface appears (an address change on an existing interface now
  shows within 10 s, not 1 s). Measured on a 32-core host: fast tier
  67 -> 58 M instructions a minute, agent system calls 114 k -> 92 k a
  minute, fast tier alone 0.073 -> 0.053 s of CPU a minute.
- **The host agent walks the process table for a third of the system
  calls (Linux).** The medium tier's state histogram and the top tier's
  table read each process's `/proc/<pid>/stat` once into a kept buffer
  (`openat`, `read`, `close`) instead of through `sysinfo`, which added
  an `fstat` and an `lseek` per file, read `statm` for a resident set
  `stat` already carries, and allocated per process. Same figures, from
  the same fields (checked against `sysinfo` by test); `status` is still
  read once per process for its uid. Measured on a 32-core host with
  ~1 000 processes, `host_metrics` asked once a second: 541 -> 377 M
  instructions a minute, agent system calls 263 k -> 114 k a minute,
  agent CPU 0.557 -> 0.297 s a minute (both changes of this release);
  the `ss` and `nvidia-smi` children are unchanged.
- **The host agent assembles its snapshot when it is asked for one, not
  each time a tier lands a part.** An assembly deep-copies every part —
  the Extended Monitoring map alone is ~1 200 values — and the fast
  tier lands one a second whether anyone reads it or not. Measured on a
  32-core Linux host (callgrind, 60 s, `host_metrics` asked once a
  second as the fleet agent does): 639 -> 541 M instructions a minute
  (-15 %); with nobody asking, the copy is gone entirely. Same snapshot,
  same first-snapshot gate (identity, fast and medium).
- **A compressed `fleet cp` / `fleet ssh -C` stops trying zstd on bytes
  that do not shrink.** After two frames in a row that went raw, the
  next frames go raw untried, a span that doubles on each probe that
  misses again (up to 32 frames, 2 MiB) and closes on the first probe
  that compresses: random or already-compressed data costs zstd about
  one frame in 33 instead of every one (it cost +20 % time on a tablet
  for 50 MB of random bytes up), and text that follows compresses again
  within 33 frames. The writer's choice alone: the frame's flag already
  said raw, so older agents read it as before.
- **Daemon replies and bus events are parsed once, not twice.**
  `Response` and `Event` read their tag first (the daemon always writes
  it first) and decode the variant straight from the line, instead of
  serde's buffering of every reply into a tree before looking at the
  tag; a tag further down still decodes, through the old buffered path.
  Same JSON accepted, same values, same unknown-tag handling (checked
  against the derive by proptest). Measured (Linux, release, per
  decode): `list` of 4 rows 71.7 -> 47.1 µs, `host_metrics` 44.2 ->
  26.3 µs, `app_metrics` 6.8 -> 4.2 µs, `process_trees` 4.5 -> 3.0 µs,
  a log event through `Event::decode_lenient` 1.65 -> 0.63 µs — what
  the agent pays every tick. Nothing changes on the wire.
- **`3pm fleet ssh` costs less CPU on both ends of a flood.** The target
  reads what its pty or pipe already holds beside a read into the same
  message, queues its screen leg by bytes (about one sealed record)
  rather than by 16 messages, and seals the screen as the upload takes
  it, without a task in between; the command line writes the terminal
  from a thread of its own (stdout and stderr kept in order, all of it
  on the terminal before the process exits, `| head -1` still 141).
  Measured on a 2 M-line `fleet ssh -t -- seq` (16.9 MB, loopback):
  command line 0.35 -> 0.08 s CPU, caller's agent 0.14 -> 0.04 s,
  target's agent 0.45 -> 0.36 s; `cat` of 500 MB on pipes 576 -> about
  700-850 MB/s. Nothing changes on the wire.
- **An idle linked agent spends about a third less CPU on the node's
  state**: the host section is keyed in place and diffed by walking the
  two sorted maps instead of a lookup per member (1.04 G -> 0.71 G
  instructions a minute, callgrind). Same sections and patches, byte
  for byte.
- **`3pm fleet ssh` / command / `attach` / `-L` / `-R` / `cp`: a session
  opens sooner.** On the target, the screen leg opens beside the
  keyboard leg instead of after it (the handshake's order is kept: the
  target writes nothing before the caller's first message); on the
  command line, the screen's connection to the backend is dialled
  beside the placement (a `HEAD /api/v1/info`), and a hostname is
  resolved on `listNodes` — the node rows only — instead of downloading
  the whole bucket with every snapshot (`getBucket` on a backend
  without that route). One backend round trip or more less before the
  first byte; old and new agents and backends interoperate.
- **An app calling another node by hostname (`io.call`, the fleet
  gate) resolves the name on `listNodes` too**, not by downloading the
  whole bucket with every snapshot on each lookup not in its 10-second
  memo; `getBucket` after a 404 from a backend without that route.
- **`3pm fleet ssh -v`, and a word when LAN direct was expected but
  not taken.** `-v` prints which way the session's legs go (this host's
  agent and its socket, or the backend's relay and why); without it, an
  agent that refused the legs the backend judged LAN-worthy (too old for
  `POST /legs`, no LAN listener) is one stderr line, once per process,
  instead of a silent fallback to the relay.
- **`3pm fleet ssh` / command / `attach` / `-L` / `-R`: a busy session
  costs far fewer records, chunks and writes.** What a leg has already
  queued now leaves together, both ways: one sealed record (up to its
  65 519 clear bytes) and one HTTP chunk instead of one per message,
  in clear (`--no-e2e`) and on the LAN direct legs too; nothing waits
  for more, so an idle keystroke leaves as before. The command line
  draws a screen chunk with one write and one flush, `stderr` kept in
  its place. A full read of a pty, a pipe or a forward is sized to one
  record (it was one and a 38-byte sliver), and an unrecorded shell is
  no longer woken every 50 ms for the audit's clock. Old and new agents
  and command lines interoperate: the legs are read as byte streams on
  every version.
- **A `live` leg no longer waits seconds in the agent's sockets.** Its
  upload to the backend rides a connection of its own: 16 KiB unsent in
  the kernel (`TCP_NOTSENT_LOWAT` on Linux and now macOS; a 64 KiB
  `SO_SNDBUF` where missing), one TLS record, hyper's 8 KiB minimum
  buffer, BBR on Linux when the agent may set it. Back-pressure now
  reaches the agent's sink, whose 100 ms age rule drops what went
  stale, instead of 0.5 MB+ of frames queueing below it (a `lan` ping
  over a 5G hotspot climbed to 34 s). Files keep the bulk connection
  and its throughput. A LAN neighbour's direct leg is served the same
  way. In process, Linux, 10× more offered than the link carries: worst
  delay 380 ms, against 1.14 s before.
- **A `3pm fleet ssh` / command / `attach` session is no longer recorded
  by default: `--record` asks for it.** Without it the command line
  sends `record: false` (bound in the handshake, so the backend cannot
  turn it back on); `--no-record` is still accepted, hidden, and says
  the default. The bucket's `record_shells` off and a node's `[shell]
  record = "never"` still win over `--record`; an older 3pm that sends
  no `record` is left to the bucket, as before. The announcement is
  printed only for a session actually recorded: "this session is
  recorded (screen and command lines)".
- **The command lines go with the recording.** A session that is not
  recorded sends no `line` and no `screen` audit event: the agent does
  not even read the screen for them (no `vt100`). `open`, `exec`,
  `attach` and `close` (`lines: 0`) are still sent. The node decides:
  an older agent (≤ 0.49) still sends the lines of a session it does
  not record, so update the agents to stop them.
- **An agent of this version refuses every session in clear** (`err
  {e2e_required, "this node requires end-to-end encryption: 3pm >=
  …"}`), with no setting to relax it: a 3pm that predates the channel,
  or `--no-e2e`, cannot open a shell, a command, a copy, an attach or a
  forward on it. Update the command line first.
- **`3pm fleet ssh` tells the channel's refusals by code, not by
  words**: the backend's `ERR_NODE_NO_E2E`, and the agent's own code
  (`e2e_required`, `e2e_broken`) that the call's `result.code` keeps
  under the action code (`forbidden` / `failed`).
- **`3pm restart <app> --name <new>` moves the log files too.** They
  used to stay under the old name while the new incarnation opened
  fresh ones; they now follow the name like the pid files (the same
  step as `3pm rename`), and the restarted app appends to them.
- **Handoff manifest version 2 when a row has `tty`.** Its stream is a
  pty master, which an image that knows nothing of terminals would
  close (the app would die of `SIGHUP`): such an image refuses the
  manifest at the pre-check and `3pm update` falls back to a cold
  restart. A manifest without a `tty` row is still written as version
  1, so a downgrade without terminal apps still hands off. Every image
  from this one reads 1 and 2.
- **`vt100` 0.16.2, vendored and patched** (`vendor/vt100/`, swapped in
  by `[patch.crates-io]`, out of the workspace): a scrollback counter,
  xterm's archiving under a scroll region starting at the top, `CSI 3 J`,
  and the bytes that rebuild a whole screen (`state_restorable`);
  `PATCHES.md` lists every change and `upstream.patch` the diff to
  propose. A new dependency of `threepm-daemon` (MIT, MSRV 1.70).
- **`3pm show` names the crash tail of a `tty` app "last output"**
  (one stream), not "last stderr".
- **The daemon's metrics tick costs ~1 % of a core, not ~15 %.** The
  process table no longer holds every thread of the host (sysinfo's
  `nothing()` kept tasks on, so each partial tick refreshed thousands of
  thread ids as "gone"); listening ports come from netlink `sock_diag`
  (the LISTEN hash only; `/proc/net/tcp*` if the host refuses it, logged
  once) and only the tree members' fd tables are read — again only when
  a member is new or a LISTEN socket appears that no member is known to
  hold (at most once every 5 ticks: a stranger opening a listener every
  second no longer walks every member's fd table every second — 7 to
  20 % of a core with a member of 20 000 connections — and a member's
  own new listener shows within 5 s), plus every 10 ticks — never every
  process of the host. An fd table is opened once per walk. Measured
  on a 32-core host of 1 100 processes: 14.6 % → 1.3 % of a core, and
  a root daemon no longer pays extra for reading every fd table. The
  host agent's medium tier keeps one `System` instead of building one
  every 10 s. A listener an app shares with a process outside its tree
  (inherited, passed) now stays with the app. The whole process table is
  swept every 30 ticks instead of 10: the full sweep only resyncs a pid
  that died and was reborn between two ticks (a member reborn on the pid
  of a process older than its leader now joins its tree within 30 s, not
  10); births, deaths, groups, environments and the members' own figures
  stay live on every tick, and Extended Monitoring keeps its 10-tick
  rhythm. Measured on the same host, 4 apps, 7 min side by side: 0.59 %
  → 0.51 % of a core.
- **The host agent's memory no longer grows with Extended Monitoring
  on.** glibc's mmap threshold is pinned at 128 KiB at agent start
  (Linux, glibc), so a freed multi-megabyte block no longer raises it
  and leaves its size in an arena; the eBPF load is skipped outright
  when the agent's `CapEff` holds neither `CAP_SYS_ADMIN` nor `CAP_BPF`
  + `CAP_PERFMON` (`not_root`, no kernel BTF parsed), and parses the
  kernel BTF once instead of twice when it runs. Measured on a 32-core
  host after 15 min, anonymous memory: 17.2 → 6.6 MB (mode on, user
  agent), 17.1 → 8.3 MB (root agent), 4.5 MB unchanged with the mode
  off.
- **`3pm start <name|id|all>` restarts a managed app (pm2).** `3pm stop
  api` then `3pm start api` failed with `script 'api' not found (looked
  at <cwd>/api, then PATH)` unless a file of that name happened to be in
  the cwd — in which case it replaced the app's configuration (ecosystem
  env, `interpreter = "none"`, …) with a fresh one. A bare word (no `/`,
  no blank, not a config file, no `-- args`) that names a managed app,
  one of its ids, or `all` now restarts it with the configuration it has,
  `start` flags folded in as on `restart` (`--watch` sets rather than
  toggles); it wins over a same-named file in the cwd, and `./api` still
  means the file.
- **`stop` turns watch off (pm2 parity).** `3pm stop` clears an app's
  watch setting: a later `start <name>` or `restart` runs unwatched until
  `--watch` asks again. It used to drop the watcher only, and `restart`
  re-armed it from the spec.
- **App telemetry no longer re-derives the supervisor's sets.** After
  every message the actor re-published the leaders and re-synced the
  watchers and the crons, three passes over every row; an app's metrics,
  events, a cron's next date and a watcher's status move no row, so
  those messages skip them (any other message still pays the passes).
- **`threepm-daemon` exposes only what the CLI uses.** Its modules are
  `pub(crate)` except `bootstrap`, `handoff` and `hostagent`, and the
  items only tests used are `#[cfg(test)]`; the dead ones are gone
  (`AdoptedReaper::clear`, `quiesce::SETTLE_DEFAULT`, now
  `handoff::SETTLE_DEFAULT_MS`). One clock: `threepm_core::time::epoch_ms`
  replaces the copies in the CLI, the host agent and the agent, and
  `threepm_link::hex` the hand-written hex encoders.
- **The agent's matches on requests and replies are exhaustive.** The
  shield (`commands/guard.rs`) and the reply forwarding
  (`link/command.rs`) list every variant instead of a `_`: a new daemon
  reply, or a new remote op that signals processes, does not compile
  until it is placed.
- **A hosted app reaches the fleet like any other app.** An app the root
  daemon runs for an account of `[users] allow` may now place calls on
  other nodes — `io.call({node, …})`, `3pm call --node`, files in and
  out, and streams, which needed a node and so were out of reach too.
  The daemon used to answer `forbidden: a hosted account may only call
  an action of its own daemon`; that rule is gone, with no setting in
  its place — the right comes with being hosted. Nothing else moves:
  the bytes already had their road (the agent binds
  `<hosted home>/agent/data-<uid>.sock` for that uid alone, on the same
  token store as `data.sock`), so no wire, agent or backend change was
  needed. A hosted caller is still narrowed where it was before: it
  targets only the rows it owns, and an input it names by `path` must be
  a file of its own uid.
- **Wire protocol v2: `PROTOCOL_VERSION = 2`, and the CLI↔daemon
  contract says each thing once.** A flag day for whoever reads
  `rpc.sock` or `pub.sock` directly (`@3pm/api`, a script parsing
  `3pm ls --json`): a v2 CLI refuses a v1 daemon and the other way
  round, exactly as a `protocol: 0` daemon is refused today —
  `3pm update` and `3pm kill` stay the two skew-tolerant doors, so a
  node still crosses the generation by handoff without restarting its
  apps. Nothing of the fleet link, the backend, the control channel
  (`ctl.sock`, an app's `io`) or the handoff manifest moves: an app that
  only uses `io` has nothing to do.
  - **One log event instead of two.** `log_out` and `log_err` become
    `log {stream: "out" | "err", …}`, and `ts` — the daemon's receive
    time — is always there: a history line `3pm logs` replays from a
    file carries the live log file's last write. A subscriber names a stream the
    way it already names a process kind, `log:err` next to
    `process:exit`. The three spellings a v1 user could type keep
    working as **filter strings** for one release — `log_out`,
    `log_err` and `log_*`, the last of which covers the same three
    kinds `log*` does — so no `3pm logs` or `3pm events` wrapper breaks
    at the update. `3pm events` names the stream in its kind column
    (`log:out`, `log:err`, where it printed `log_out` / `log_err`): one
    event, but the detail column is the line itself and has nowhere
    else to say which stream it came from.
  - **One answer for `call` and `dispatch`.** `trigger` is renamed
    `dispatch` — the two are named by their role now: `call` is the
    CLI's and an app's way in, `dispatch` the fleet agent's, with its
    tickets already prepared — and both answer `called {ok, results,
    err?, code?, output?}`. A `call` names the id the fleet knows it by;
    a `dispatch` fans out to one call per instance and names none of
    them there, which is what the audit line (`proc_action.call`) is
    for. `res: "triggered"` is gone.
  - **The dead fields are gone.** The `app_metrics` request keeps only
    `id` (`since_rev` and `gen` were accepted and ignored); its answer
    loses `base_rev`, `del`, `full` and `hist[].reset`, which the daemon
    never set — so `3pm show --json` no longer prints the constant
    `app_metrics.full: true`. `res: "process"`, which no daemon ever
    sent, is removed.
  - **What was optional for compatibility is required.**
    `start.if_changed`, `restart.toggle_watch`, `delete.flush`,
    `send_signal.group`, `called.ok` and both action requests' `stream`
    are plain fields: a message that omits one is an error instead of a
    silent default. `pong` does not move — it is what announces the
    version — and neither do `ErrCode`, `Target`, the 31 fields of
    `ProcessRecord`, `DaemonInternals`, the control channel or the
    `calls` vocabulary.
  - **The Node SDK speaks v2** (`@3pm/api` 0.20.0): `connect()` refuses
    a v1 daemon with the usual `protocol_skew`. `bus.on('log:out')` and
    `bus.on('log:err')` are untouched — the bus builds them from
    `log.stream`, the way it already built `process:<kind>` from
    `process.kind` — but a listener on the *wire* name (`bus.on('log_out')`)
    must move to `log` and read `e.stream`. `LogEvent.ts` is no longer
    optional. Nothing on the request side changes.
  - The frozen fixtures are `tests/fixtures/wire/v2/`; `v1/` stays in
    the tree, and beside it a new family, `update/`: the seven requests
    and six responses `3pm update` speaks across a generation
    boundary, **frozen for good and never regenerated**
    (`docs/protocol.md` §2.1). They are the same documents in v1 and in
    v2 — which is why a v2 CLI already updates a v1 daemon with its own
    types — and two tests now say so, one on the fixtures and one
    driving the update path's real decoding. A later generation that
    breaks them fails those tests instead of failing an update in the
    field.

- **An app's crash tail survives `3pm update`.** The stderr lines a
  running app had written before a handoff were dropped when the new
  image took its process back, so a crash just after an update reported
  only what the app said since. They are kept now, ahead of the newer
  lines. An adopted row also reports `tree_rev` 0 (visible in `3pm show
  --json`) until its next metrics tick, instead of a revision naming the
  tree of the image before: the manifest carries no tree, and the new
  `boot_id` a handoff draws already tells a reader to forget the old
  `(gen, tree_rev)` pairs. The daemon's own revision counter is carried
  as before, so no revision is ever reused.

- **Link 5: one mechanism for the bytes of a result — `blob_chunk` is
  gone.** `LINK_VERSION = LINK_MIN = LINK_MAX = 5`, a flag day like link
  3 and 4: **update the nodes first, the backend last**, and restart the
  agents refused during the window. What a call carries and what it
  answers are unchanged; how the file an action returns gets there is
  not.
  - **The file an action returns is uploaded, not framed.** Before it
    sends the `reply`, the agent streams the file — raw, never gzip —
    to `PUT /api/buckets/{cluster}/calls/{call}/artifacts` with the
    bearer of its `welcome`, hashing it as it goes; the `201` answers
    `{artifact_id, bytes, sha256}`, the agent checks that digest
    against its own, and the outcome's `file` names the artifact. The
    512 KiB pieces, the four in flight, the 32 MiB link cap and the
    per-session blob budget all go with the frame. The ceiling is now
    the action's declared `output.max_bytes` (up to 256 MiB), 32 MiB
    when it declares none — the old cap, kept as the default.
  - **A result too big to travel is an error, not a fetch.** `data`
    rides inline, gzip + base64url, within one 256 KiB budget per reply;
    an outcome whose result does not fit fails `too_large` — "the
    result is N gzip bytes; a result this size is a file" — and the rest
    of the reply travels. It used to become bytes to fetch, silently.
  - **One outcome type in both directions.** `reply.res.triggered` and
    `call_result` now carry the same `Outcome {…, data?, file?,
    output?}`: what a node learns of its own call it can also be told
    about another's, since both ends fetch an artifact the same way.
    `SettledOutcome`, `RemoteOutcome`, `FileRef` and the `Payload` enum
    are gone, and `ArtifactId` joins `BlobId`, `CallId` and `Sha256` as
    a validated identifier.
  - **An outcome fails alone.** No bearer from this backend, a store
    that refuses the file (413 → `too_large`, anything else →
    `failed`, with the backend's own words), a digest the store
    disagrees about: each fails that one outcome, and the reply never
    names an artifact the bytes are not behind.
- **Link 4: one enum per direction, a three-frame handshake, typed
  identifiers.** `LINK_VERSION = LINK_MIN = LINK_MAX = 4` and the
  WebSocket sub-protocol is **`3pm.link`** — no number in the name,
  since `link_min..=link_max` is the whole of the version — so a peer
  of the previous generation is refused at the upgrade (400) or at the
  handshake (`ERR_LINK_VERSION`): **update the nodes first, the backend
  last**, then restart the agents refused during the window. Nothing
  changes in what the link carries.
  - **The handshake is three frames**: `challenge` → `hello` →
    `welcome`. The signature over the challenge's nonce rides the
    `hello` it authenticates (`hello.sig`), so there is one message to
    sign and one to verify; the separate `proof` frame is gone, and so
    are `hello.agent` (a constant) and `hello.node_id` (the
    `Authorization` header carries it and the signature binds it).
  - **`welcome.pace` replaces the three period knobs.** One mandatory
    object — `{state_s, metrics_s, trees}` with `trees` one of
    `{mode: "off"}`, `{mode: "shape"}` or `{mode: "figures", every_s}`
    — instead of `metrics_every_s` / `trees_every_s` / `state_every_s`,
    where "absent" meant three different things. Every period is
    floored at a second on decode, a mode this build does not know
    reads as `off`, and a `welcome` missing a field is refused rather
    than guessed at.
  - **A frame body never ends a session.** Each side reads the tag
    first and the body second: an unknown tag is skipped, and a known
    tag whose body does not decode is skipped, **counted** and — when
    it was a `call` — answered `reply {res: err {code: "malformed"}}`
    instead of being left to die on its deadline. `3pm agent status`
    reports the count as `frames_invalid` (`--json`, and `, 3
    unreadable frames` on the `link` line when it is not zero).
  - **Identifiers are types, not strings.** `BlobId`, `CallId` (both
    `^[A-Za-z0-9_-]{1,64}$`) and `Sha256` (64 lowercase hex) validate
    on decode, so a blob id that is not a path segment or a digest of
    the wrong shape fails its field and its frame — which is what
    retires every hand-written guard the agent ran after the fact. The
    link declares its own `LinkInput` and `OutputBlob` instead of
    borrowing the local wire's `ActionInput` / `OutputRef` (which carry
    a `path` and a `token` belonging to one host), and a `call_result`
    can no longer describe bytes the backend cannot deliver.
  - **The acceptor acknowledges.** The reader no longer sends the
    `ack`; the task that takes the call does, as its first act. A call
    a busy session could not be handed is therefore never acknowledged
    and the backend's view says so, which is the truth.
- **A node is a key, not a certificate: enrollment is one HTTP POST.**
  `3pm link` no longer opens a WebSocket to enroll: it posts the
  cluster, the token, its fresh Ed25519 public key, what the host is and
  the link versions it speaks to `POST /api/v1/enroll`, and gets back
  the node id, its state, the link version both sides share and the
  server's clock (a refusal is the backend's own `ERR_*` code with its
  HTTP status). The proof signed with the request binds the token id and
  the key — no server nonce, since what bounds a replay is the token's
  use count. Nothing is written before the answer is checked: clocks
  more than five minutes apart stop the link with `ERR_CLOCK_SKEW` and
  the delta, so a skewed clock is a sentence at `3pm link` time instead
  of a handshake that fails later. The state directory is `node.key` and
  `link.toml`: the node certificate (`node.crt`) and the cluster CA
  (`ca.pub`) are gone — the backend already looked every node up by its
  public key — and a host linked before this keeps its node id and its
  key, the two stale files being removed the first time the agent loads
  its identity. **No re-enrollment anywhere.** The link upgrade now
  names the node in the clear (`Authorization: 3pm-node
  <cluster-id>:<node-id>`, both public: a claim, not a credential) and
  the proof is verified against the row's key; an unknown node is 401
  `ERR_NODE_UNKNOWN` (`ERR_CERT_INVALID` and `ERR_CA_MISMATCH` are
  gone). `welcome` loses `cert` and `accepted_caps`: what a remote call
  may do on a host is that host's own `--cap` list, declared in
  `hello.caps` and enforced by the agent — the backend grants nothing.
  `3pm agent status` drops its certificate line and the `cert` field of
  `--json`, and `3pm fleet inspect` its certificate line.
  `LINK_VERSION = LINK_MIN = LINK_MAX = 3`: a peer that only speaks
  link 2 is refused at the handshake with `ERR_LINK_VERSION` ("update
  3pm"), so every node is updated before the backend.

- **Two field lists that had drifted from the daemon's now come from
  it.** Both were hand-kept copies of the same table; both are derived
  from `StartSpec` itself now, so there is nothing left to drift.
  - `backoff_initial`, the 0.1 spelling of
    `exp_backoff_restart_delay`, was missing from `FIELD_ALIASES`: an
    ecosystem file using it was accepted (serde resolves the alias) but
    drew a spurious "unknown field" warning on the way through
    `config/resolve.rs`. It no longer does.
  - The **Node SDK** refused nine pm2 spellings the daemon has always
    resolved — `out`, `output`, `out_log`, `error`, `err`, `err_file`,
    `err_log`, `log` and `combine_logs`. `pm.start()` answered "unknown
    option" to each; it takes them now, and `StartOptions` in
    `index.d.ts` declares them.

### Internal
- **The state that crosses the `execve` is four owned structs.** The
  supervisor actor kept the image's identity, the action calls, the
  handoff phase and the clock as sixteen loose fields; they are now
  `ImageIdentity` (`boot_id`, `lineage`, `handoffs`, `next_id`,
  `tree_revs`, `next_call` — the six counters the manifest carries, with
  the rules that restore them next to them), `ActionRegistry` (the calls
  in flight plus the three things that outlive one: the ids the control
  writers read, the paths retention must spare, the probe count),
  `quiesce::HandoffState` (the phase and the last report) and `Clock`
  (the monotonic base and `booted_at_epoch_ms`, which continue from the
  previous image rather than restarting). Counters are handed out by
  method — `take_id`, `take_call_id` — so the increment-before-use of
  call ids and the increment-after-use of row ids each have one home,
  and the tree numbering gets a borrow of that counter alone rather
  than of the whole identity. `LiveCalls` and `Pins` were already shared on
  purpose and are moved in as they are: no new `Arc`, no new `Mutex`.
  `HANDOFF_VERSION` stays 1 and neither `Handoff` nor `Row` changes
  shape; the state → manifest → state test passes with its assertions
  untouched. New `crates/threepm-cli/tests/e2e_handoff_cross.rs` runs a
  handoff **between two builds** with live apps, both ways, asserting
  the apps keep their pids and the daemon keeps its own
  (`THREEPM_E2E_OLD_BIN` names the other build; the test does nothing
  without it).
- **The per-app spec is written once.** `StartSpec` carries the whole
  field table — a `///` and a `#[spec(...)]` line per field — and a new
  proc-macro crate `threepm-spec-derive` (`#[derive(SpecTable)]`)
  generates the rest: `SpecPatch` (the 50 user-facing fields, their pm2
  aliases, their `skip_serializing_if` and `deserialize_with`),
  `KNOWN_FIELDS` and `FIELD_ALIASES`, `SpecPatch::sample`, the profile
  overlay, and the part of `apply_over` that is a fold. Eleven fields
  whose fold is a rule rather than a fold (`watch`'s pm2 shorthands, a
  size string, a cron expression, `log_rotate`'s four knobs, the
  `autorestart` / `restart_policy` pair) stay hand-written in
  `SpecPatch::fold_custom`, which the macro requires to be exhaustive:
  one left out does not compile. **No name changes**, on the wire, in
  `dump.json` or in the handoff manifest — the frozen wire fixtures and
  the CLI snapshots are untouched. Around it, the tables that used to be
  copies: `3pm __spec-schema` prints the table as JSON, `just gen`
  writes it to `packages/node/lib/spec-schema.json`, the Node SDK builds
  its `RENAMES` / `PASSTHROUGH` / `ALIASES` / `KNOWN` from it, and the
  `StartOptions` block of `index.d.ts` and the field table of
  `docs/ECOSYSTEM.md` are generated between markers. `just check` now
  fails on a stale one (`just gen-check`). The flag surface stays
  hand-written — clap's help is prose and its rules are its own — but a
  test walks `SpecArgs` both ways against the table, so a flag without a
  field, or a field claiming a flag that does not exist, fails. Two
  flags are listed as deliberate exceptions: `--time` and
  `--no-autorestart` are sugar for a value, not fields.
- **The collectors, the process trees and the metric kinds stopped
  speaking the wire.** Three places where a `threepm_protocol` type was
  the daemon's own working value, each with a translation at its edge
  now. The host collectors and the platform parsers build daemon leaves
  (`hostmetrics/leaves.rs`) that `tiers::assemble` converts once: files
  under `hostmetrics/` naming the protocol went from **9 to 3** (the
  three that are the edge). A tick's process tree is a `ProcTree`
  (`metrics/proctree.rs`) with no identity at all, so the `AppTree` that
  existed between `build` and `stamped` holding `id = gen = rev = 0`
  cannot be written any more; the supervisor's `proctree::stamp` is the
  only way to get a stamped tree, and the only place a tree is cut for a
  reply. "No tree" is `None` rather than a wire default, and an
  instance's stored tree no longer carries totals it never filled. The
  two caps are unchanged and keep their own jobs: `TREE_PROCS_MAX` (256)
  is what a row remembers — the shape `tree_rev` follows — and
  `TREE_LINES_MAX / listing` (2 000 shared out, clamped to 256) is what
  one reply carries; they only ever tie below eight rows listing
  members. A validated metric's kind is `validate::Kind`, which has no
  `Unknown`: the wire kind the daemon does not know is refused by the
  conversion at parse time, where the explicit comparison used to sit,
  and becomes a `MetricKind` again where a `SeriesDef` is built. **No
  byte on any socket changed** and `threepm-protocol` was not touched.
- **The daemon's inside stopped speaking the wire.** Twenty-nine files
  built `threepm_protocol::{Request, Response, ErrCode}` by hand, the
  supervisor's twenty-two of them through a `use super::*` re-export, so
  every rule about who may do what was phrased as a socket message. The
  actor now takes a `Command`, answers an `Outcome` or a `Refusal`
  (`crate::domain`), and the wire is translated in one place per edge:
  `SupervisorHandle::{request_as, command}` for `rpc.sock`/`ctl.sock`,
  `bootstrap::daemon_main::DaemonRpc::handle` for the `handoff` it
  intercepts, and `hostagent::proxy::fetch` for `host.sock`. The
  `SupervisorHandle::request(Request) -> Response` the CLI and the
  handoff use is unchanged. Counting production code only — test files
  and `#[cfg(test)]` modules excluded, comments excluded — those **29
  files are 9 now**, two of which are the translation itself;
  the twenty-four scattered `ErrCode::` sites of `supervisor/` and
  `users/` became **zero**, replaced by two tables in `domain::outcome`,
  one per direction. The hosted-user allow-list (`Caller::refusal`) is
  exhaustive on `Command` now — still refusing to compile until a new
  command is classified, still naming the wire tag in the message it
  prints. **Not one byte on any socket changed**: same variants, same
  fields, same codes, same messages, same event order, proven by the 73
  CLI snapshots and the `e2e` / `e2e_handoff` / `e2e_users` binaries.
  One caveat, on a path nothing reaches today: `hostagent::proxy::fetch`
  used to hand the host agent's answer back untouched, and now answers
  `err{unavailable}` if that answer is neither `host_metrics` nor `err`
  — which the agent's own server (`hostagent/server.rs`) cannot send.
  `threepm-protocol` was not touched.
- What a handoff carries is pinned field by field
  (`supervisor/tests/handoff_state.rs`): a populated actor writes the
  manifest, a second adopts it and writes its own, and the two are
  compared through destructurings of `Row` and `Handoff` with no `..` —
  a field added to the manifest no longer compiles until the read side
  says whether it survives. That test is what found the crash tail
  above.
- **What `assemble` copies is pinned field by field.** The 50 hand-written
  `host.<field> = part.<field>` lines of `hostmetrics/tiers.rs` had about
  twenty of their fifty-one targets under test; the rest (`paging`,
  `pressure`, `mem_detail`, `file_handles`, `sockets`, `conntrack`,
  `cgroup`, `vfs_cache`, `kernel`, `cpu_states`, `block_devices`,
  `load_avg`, `per_core_percent`, the swap and memory gauges,
  `cpu_frequency_mhz`) were copied by nobody's assertion, and a forgotten
  line would only have shown on a real host exposing that source. A full
  set of parts is now compared against a `HostMetrics` literal with no
  `..Default::default()`, so the next field added to the snapshot does not
  compile until someone names the part that fills it. The part fixtures
  are exhaustive the same way. The tests moved to
  `hostmetrics/tiers_tests.rs` to keep both files under 500 lines.
- **Three fields the handoff manifest wrote and nobody read are gone**:
  `opts` (the launch options travel on argv, which
  `handoff::exec::build_argv` builds from the live ones) and
  `writer.protocol` / `writer.exe`. `HANDOFF_VERSION` stays 1 and a
  rollback is safe in both directions: `Handoff` and `Writer` are
  `#[serde(default)]`, so an older image reading a manifest without them
  gets their defaults, which it never consults either.
- **Dead code out.** `hostmetrics::collect` — a second, divergent
  implementation of the tier pipeline with no non-test caller — is gone,
  its real-I/O smoke test keeping its coverage through the collectors
  directly; so is the unused `NTP_SERVER_ENV` re-export (the constant
  itself still drives `hostmetrics::slow`). `watcher::null_spawn` and
  `cron::null_spawn` are `#[cfg(test)]`: every caller was already a test.
- **Module headers measured against the code.** The `//!` blocks are the
  only map of the daemon's 38 000 lines, and an audit found the first four
  invariants it checked were all false. Every header asserting one was
  re-read against the code; the eight that lied were rewritten, and two
  more that overstated their scope were tightened. Notably:
  the supervisor actor does *not* run every input through `policy::step`
  (seven of its forty message arms feed it directly, plus the watch/cron,
  shutdown, resurrect, adoption and RPC paths — the rest is hand-written
  orchestration); the manifest's `emergency` section is not readable by
  any version, because
  `read_emergency_only` has no caller; `watcher` and `cron` raise
  `TriggeredRestart` for every row of an app, not `RestartRequested` for
  the online ones; and the supervisor does reach for a Unix-ism itself, in
  `orphans::kill`.
- **The supervisor's outward borrows have a name: `Ctx`.** A helper moved
  out of the actor needs the event bus, the actor's own queue and the
  paths on top of the state it works on, so every extraction was going to
  add three parameters and every call chain re-thread them. `Ctx` groups
  those three, `action_calls::fail_calls` takes one in place of its `bus`,
  and its five call sites build one. Nothing else moves and nothing runs
  differently. Two things the attempt settled: the actor's state maps
  (`procs`, `calls`) cannot go in it, since three of those five sites run
  inside a live `procs.get_mut(&id)` borrow and could not build it at all;
  and its constructor cannot be a method, because a `fn ctx(&self)`
  borrows the whole actor and no site holding a `&mut` on one field can
  call it. The borrow checker is still not what makes the extractions
  hard — disjoint field borrows work, and the actor's loop is a `while
  let`, not a `select!` — but it does decide the shape of the type.
- **One machine behind the file watchers and the cron schedulers.** The
  two were structural twins down to the 28-line reconcile loop; they now
  share `derived`, which holds the handle whose `Drop` cancels the task,
  the injected spawner, the `desired` set read off the rows, and the
  reconcile itself. `watcher` and `cron` keep only what differs — their
  key, how it is read off a spec, and what the task does — and the
  supervisor's two `sync_*` methods are ten lines each instead of
  twenty-eight. The invariant the pair exists for is now stated in one
  place and tested there: the set is a function of the rows' specs and
  statuses, never a bit toggled by hand, which is what keeps pm2's
  phantom watcher impossible.
- **Four clusters of the actor's own state are structs, not loose
  fields.** The fleet gate and the calls out through it (`FleetLink`),
  what the metrics tick measured of the daemon itself (`SelfMetrics`),
  the replies parked until a row settles and the armed timer count
  (`Waiters`), and the open log handles with the policy they run under
  (`LogSinks`). Each owns the few methods that touch nothing else — is
  this connection the gate, are we at the ceiling of calls out, which
  policy does this row write under — so the fields are private to the
  cluster rather than reachable from every file in the module. None of
  the ten fields crosses the `execve`, which is why they could move at
  all: the handoff manifest names not one of them.
- **Four plumbing duplicates, written once.** Nothing changes in what the
  daemon does; four pieces of it were written twice, and one of the two
  copies always lagged.
  - **One accept loop.** `rpc.sock`, `host.sock` and `ctl.sock` ran two
    copies of the same thirty-seven lines — uid check, per-listener
    gate, one task per connection joined as it goes, one drain grace at
    shutdown — differing only in the words of three log lines. They now
    call `rpc_server::accept_loop`, which takes that word and what one
    connection runs. `bus.sock` stays out of it: its drain is per
    connection, on purpose.
  - **One reader for `config.toml`.** `[users]` and `[logs]` were
    declared and read in their own modules, with their own idea of a
    problem: a typo under `[users]` was named with a suggestion, the
    same typo under `[logs]` changed nothing and said nothing. The file
    is now declared in `config_file.rs` and read there once, with the
    stricter policy for both. A section that does not decode still falls
    back to its own defaults alone, so a bad rotation value never costs
    the machine its hosted users.
  - **One durable replace.** The handoff manifest and `dump.json` each
    had their own tmp + fsync + rename + fsync(dir); only the manifest
    created the file 0600 and refused to reuse a leftover `.tmp`.
    `atomic_file::replace` is both, so **`dump.json` is now 0600** — it
    carries every app's `env`, and the policy of `users::file_owner`
    always said so.
  - **The log pump reports facts.** Building the `LogOut` / `LogErr` /
    `LogRotated` events moved out of `child_task`'s delivery path and
    onto `PumpRow`, which is what knows the row, the instance and the
    stream.

### Removed
- **The Intel macOS build.** `just publish` builds three targets
  (macOS arm64, Linux x86_64 and arm64): a quarter less work per
  release. `install.sh` on an Intel Mac stops with "Intel Macs are no
  longer supported (last build: v0.14.0)"; a shell under Rosetta on
  Apple silicon gets the arm64 build.
- **The three older node-state streams of the link, and link version 1.**
  `telemetry` (`host_tick` / `host_full`, `strip_static`, `FullPolicy`,
  the frame trimming), `app_metrics` / `resync` (`AppMetricsPatch`, the
  patch reader) and `trees` / `trees_request`, with
  `welcome.full_every_s`, the agent's sampler, application-metrics and
  process-trees tasks and their three daemon connections.
  `LINK_VERSION` is 2: an agent or a backend that only speaks version 1
  is refused at the handshake with `ERR_LINK_VERSION` ("update 3pm")
  instead of being half understood. The agent's `THREEPM_AGENT_INTERVAL_MS` /
  `THREEPM_AGENT_FULL_EVERY_S` no longer pace anything: the period is
  the backend's `state_every_s`.
- **The daemon's application-metrics diff.** `app_metrics` always
  answers the whole state (`full: true`): the per-series and per-bucket
  revisions, the histogram reset marker, the tombstones and the `since`
  filtering are gone from `threepm-daemon` (`AppMetricsState::dump`).
  `since_rev` / `gen` on the request and `base_rev` / `del` / `reset` on
  the answer stay on the wire, unread and unset, so an older `3pm` CLI
  or agent still gets a valid (full) answer. `app_metrics_rev` still
  moves only when something a reader sees moved. A whole answer at the
  series cap costs 0.28 ms (950 series, 120 KB).
- **Dead parts of the link.** The `renew` / `cert` frames (the agent's
  renewal timer and its jitter, `renewal_due`): no backend ever answered
  a `renew`, and a certificate is renewed by `welcome.cert` at the
  handshake, as before. `hello.last_seq` (always 0, never read) and
  `welcome.session` (`SessionId`, only ever logged). TLS pinning, which
  was never implemented: `Pin`, `ClusterKey`, `Disconnect::PinMismatch`,
  `pin` in `link.toml` — a pasted `3pmc_<id>.<pin>` made `3pm link`
  fail, the cluster key is now the cluster id as is. Still link
  version 2: a `renew` or a `last_seq` from an older agent is skipped,
  and a `pin` key left in a `link.toml` is ignored. An agent up to 0.5.0
  needs `welcome.session` to decode the welcome, so agents are updated
  before the backend.

### Security
- **`3pm fleet ssh -L` / `-R` / the target's forward: a transfer cut
  before its end resets the connection instead of closing it.** When a
  leg ends without the sealed `END` (a cut, a record that fails, the
  agent stopping), the local or destination connection is closed with
  `SO_LINGER` 0: its peer reads `ECONNRESET`, never an EOF it could take
  for the whole of a truncated transfer. A clean `END` still closes with
  a FIN. Nothing changes on the wire.
- **`3pm fleet cp` on a node whose account's shell is fish, csh or
  anything not POSIX: the copy's script runs under `/bin/sh` as sent.**
  The target ran every command as `<login shell> -c "<line>"`, and the
  line is quoted for POSIX: a fish (`\'` inside single quotes) or csh
  (a newline) would re-read a local file name as code. A copy's argv
  (`sh -c <one of the four fixed scripts> 3pm-cp <paths…>`, byte for
  byte) is now executed directly, same account, environment and
  directory; any other command still goes through the login shell.
- **The node's `authorized_keys` is checked like sshd's `StrictModes`
  all the way up.** A symbolic link is refused (opened with
  `O_NOFOLLOW`), and so is a file whose state directory or that
  directory's parent is owned by another user than the agent's or root,
  or writable by others or by a group other than the owner's private one
  (a sticky directory such as `/tmp` excepted): whoever could write
  there could swap the file. A refused file admits nobody, and
  `3pm link status` says why.
- **`known_nodes`: a hostname pinned under another node id is refused.**
  The pin was keyed by node id only, so a backend answering the
  hostname the person typed with another node id made the impersonation
  a first contact, pinned silently under `--accept-new`. A hostname
  typed in full (case aside), already pinned under another id of the
  same cluster, is now refused before any question, `--accept-new` or
  not: `NODE IDENTIFICATION HAS CHANGED`, both nodes named, and
  `3pm fleet known-nodes rm <pinned id>` to forget the old one. The file
  gains a fifth column, the cluster id (one file serves every bucket);
  older lines without it are compared in every cluster, and older 3pm
  builds still read the new ones. A typed prefix or id is not checked.
  `fleet cp` now pins the node as its paths name it, like `ssh` and
  `attach`.

### Fixed
- **`3pm fleet ssh -W/-L/-R` no longer dies after a minute of silence.**
  A tunnel where nothing crossed either way for a minute (an `ssh`
  through `-W` waiting on a quiet remote command, with no
  `ServerAliveInterval`) was given up by the relay and the caller exited
  255. The caller now sends an empty message on its input leg after 20 s
  of local silence, as a shell pings, until its peer's EOF; nodes
  already ignore it, so only the caller's `3pm` needs the update. The
  output leg is still silent while the destination is, so a reverse
  proxy in front of the backend still bounds a fully idle tunnel by its
  read timeout (1 h on fleet.edgecr4ft.net).
- **A file the host agent stops shipping leaves the agent's set.** The
  log agent kept a file it once tailed for as long as its path existed,
  even after the host agent's current pass called it `own`,
  `duplicate_journal`, `private` or `binary` (after `3pm update`,
  `~/.3pm/3pm.log` went on shipping). Such a verdict, by path or by
  inode, now drops the file; a file merely missing from one pass still
  stays.
- **No lines lost when one automonit ask fails.** A failed or timed-out
  `logs` action emptied automonit's files for a whole ask period, and a
  file declared again was reopened from its end. The last good answer
  now stands for three periods, and a retired file's last committed
  position is kept for an hour, so a file declared again resumes where
  it stopped.
- **A `401` no longer waits days for a token it already has.** The wait
  for a new bearer compared with the token current when the `401` came
  back, so a `welcome` that renewed it while the post was in flight was
  missed until the next reconnect. The wait now compares with the token
  the post used, and `3pm link status` says `401: waiting for a new
  token` meanwhile.
- **A busy backend's `retry-after: 1` no longer means a re-upload every
  second.** A `503` with `retry-after` is retried after the longer of it
  and the doubling backoff (at most 60 s), never sooner; a `429` keeps
  the backoff as its floor too (a `retry-after: 0` is not a loop).
- **A log path with a control character is refused.** A newline in a
  declared path could inject a line into the `offsets` file (C1 controls
  and bidi overrides are refused too: they disguise a name). Such paths
  are ignored from the host agent and from automonit, and the `offsets`
  writer never writes one.
- **Source ids that start over get a new epoch.** When the `ids` table
  was lost but the template miner's state was kept, `f1` named another
  file under the same epoch. The table now records the miner's epoch,
  and a new one is drawn whenever the two disagree.
- **A malformed `[logs]` in `link.toml` is said.** A section that reads
  as closed (an unknown key, a wrong type) ships nothing; the agent now
  warns at start, and `3pm agent status` shows `logs off:` with the
  reason (`logs_problem` in `--json`).
- **The ids table forgets the least recently seen source.** A source was
  stamped only when first asked in a run, so a full table evicted the
  longest-lived files and streams first. Files are now stamped at every
  discovery pass, app streams once a minute while they write.
- **The daemon's own logs are never shipped during a `3pm update`.** The
  host agent called a file `own` only when this host agent or its
  daemon held it; across a handoff the old agent saw the new pair as
  strangers and declared `~/.3pm/3pm.log` `ship` (seen in prod: tailed
  for 19 h). The home's `3pm.log` and `hostagent.log` (`Paths`, matched
  by path and by inode) are now `own` whoever holds them, and any file
  under the home's `logs/`, or the `logs/` of a home the daemon hosts
  (the accounts of `[users] allow` it accepts, read by its own
  `[users]` loader), is `duplicate_bus` whoever holds it, so the apps'
  output is not shipped twice during the handoff either.
- **One user can no longer push system log files out of the list.** At
  the 2 000-file cap the list kept `ship` files first, then by path, so
  a local user holding 2 000 files under `/home` evicted every
  `/var/log` file and the fixed list. Files are now ranked by the
  writer's real uid, which it cannot choose (a cgroup proves nothing: a
  hosted user's app may run in the daemon's own unit). Root's writers
  and the fixed list come first and are never cut for anyone else; then
  system accounts' writers (uid below 1 000) outside the daemon's
  cgroup; then the room left shared evenly between users, a user's
  subordinate uids (100 000 and up, `/etc/subuid`) counted as one. The
  sticky set evicts users' entries first. A walk cut at its budget
  visits root's processes first.
- **A long log-file discovery pass lands instead of starting over.** A
  pass past 30 s was dropped even when it finished later, and the next
  one started afresh, so with many descriptors no list ever landed. A
  late pass is now served when it ends (its sticky set kept), a pass
  reads at most 4 096 descriptors per process, and the walk stops at
  20 s with what it found (`partial` and `fds_capped` in the pass
  stats log line).
- **More histories are `private`.** A `history` file or directory
  anywhere under a dot-directory (`~/.local/state/bash/history`,
  `~/.local/share/zsh/history`, `~/.config/zsh/history`, `.cache/…`),
  zsh's default `~/.histfile`, nvim's `*.shada`, and a `lesshst`
  without its dot are never shipped.
- **A sticky log file no longer follows a symlinked parent.** A
  rotation (a new inode at the path) was followed when the file had
  the same owner, checking only the last component: swapping a parent
  directory for a symlink could lead it to another file. A rotation is
  now followed only when no component of the path is a symlink.
- **A misspelt key in `link.toml` `[logs]` closes it.** An unknown key
  (`file = false`, `hosts = false`, `hostlevel = "err"`) was ignored and
  left the real switch on. `[logs]` now refuses unknown keys: the
  section reads as malformed and ships nothing, and
  `LinkConfig::logs_problem` gives the reason for the agent to warn.
- **Log shipping redacts more credential shapes** (the agent's
  redaction, found by a probe in review): a Python string prefix
  (`'password': b'…'`, `u'…'`, `r'…'`, `f'…'`, `rb'…'`) no longer hides
  the value behind its letter; Go MySQL DSNs (`app:pw@tcp(…)`,
  `@unix(…)`, `app:pw@/db`, any host after a `dsn` key); the password
  argument of a PHP connection (`PDO->__construct(…)`, `new \PDO(…)`,
  `new mysqli(…)`, `mysqli->connect(…)`, `mysqli_connect(…)`,
  `mysql_connect(…)`); XML credentials (`<password>`, `<wsse:Password
  …>`, `<secret>`, `<token>`, `<apiKey>`, CDATA, `<property
  name="password">`, `<entry key="db.password">`); Kubernetes/ECS env
  lists (`name: DB_PASSWORD, value: …`, `Key`/`Value`, either order,
  other keys between); a bare `auth` key holding base64,
  `user:password` or a quoted secret (docker's `auths`, npm's `_auth`,
  Node's `auth:`, Firebase's `?auth=`); `mongosh … -p X`, a glued `-pX`
  after a MySQL or Mongo client, `zip -P` / `-eP`, `7z -p`, `openssl enc
  -k` / `-pass pass:`, `--admin-pass`; Oracle (`jdbc:oracle:thin:u/pw@…`,
  `user/pw` after sqlplus, expdp, impdp, sqlldr, dgmgrl, `rman target`,
  a `CONN u/pw` statement); Redis (`requirepass`, `masterauth`, `CONFIG
  SET requirepass`); NewRelic `license_key`, SendGrid `SG.…`,
  `Ocp-Apim-Subscription-Key`, a legacy Vault `s.…` after `vault login`.
  Kept: MySQL's `(using password: YES)`; a spaced `-p` is read as a
  password only after a Mongo client in the same command (a MySQL
  client's spaced `-p` prompts, and psql's, redis-cli's, ssh's and
  docker's `-p` is a port); digits after a DSN user (a port) or as an
  env value (`PASSWORD_MIN_LENGTH`), env names that describe a secret
  (`SECRET_NAME`, `TOKEN_TTL`), templates (`{{ .Values.token }}`),
  timestamps, URLs and digests after `auth`, `connect host/db@…` in
  prose.
- **The boot daemon answers while it resurrects.** `3pm __daemon --boot`
  (the systemd/launchd service) restored the saved list before it
  served its sockets or handled SIGTERM, and the resurrect waits for
  every dependency wave: an app under `wait_ready` that never signalled
  left the daemon unreachable (`3pm ls` hung, the agent saw "daemon not
  reachable", no host agent) for its whole listen timeout, and a
  `systemctl stop` meanwhile ended in SIGKILL. The resurrect now runs
  beside the main loop; a SIGTERM during it drains, and a dump read
  that finishes after the drain began starts nothing.
- **Log shipping, found on the live fleet**: an empty line (the kernel
  writes one after some messages) is no longer mined, counted or
  shipped as a template of its own; node and bucket ids (`3pmn_…`,
  `3pmc_…`) are public names and stay readable (only `3pmu_`, `3pma_`,
  `3pmk_` are secrets); a numeric value after a `token` key (a token
  count) and a plain word after `password` (`password for root`,
  `password is expired.`) are no longer redacted.
- **The Linux network meter no longer takes other sockets' closes when
  it subscribes.** Its destroy-notification socket joined the kernel's
  groups before its port filter went on, and a classic filter only
  judges what arrives after it: every TCP close on the host in between
  stayed queued and reached the meter unfiltered (under a busy host, a
  burst of strangers' ports at each subscription). The filter now goes
  on first, then the groups are joined. Seen as
  `the_destroy_filter_lets_our_port_through_only` failing 11 runs in 30
  beside four loopback connect/close loops; 0 in 30 since.
- **`3pm fleet ssh -C -T -- cat` read slower than it came is no longer
  twice the CPU on the target.** The screen leg's queue held one sealed
  record's worth, so a pipe read in full and the upload ran in lock
  step whenever the caller or the link was the slower end, one wake-up
  per record (tablets: target agent 0.35 -> 0.85 s CPU, 0.57 -> 0.96 s
  for 50 MB of text). It now holds four records ahead. Measured on a
  500 MB `cat` read at 200 MB/s (loopback): target agent 2.16 -> 1.19 s
  CPU, command line 1.28 -> 0.79 s. `fleet cp`, the pty flood and the
  bytes on the wire are unchanged.
- **A transfer between LAN neighbours no longer hangs on a reader that
  stopped reading.** Once a file or a leg went direct, the writer's `PUT`
  waited for the reader with no deadline; it now ends with an error
  (`502`, "no byte moved for 90 seconds") once no byte moved for 90 s,
  and the writer's body is let go of at once. A quiet session is never
  cut: its legs carry the caller's ping every 20 s and its answer.
- **A relayed `PUT` whose body the backend swallowed is answered.** The
  answer had no deadline once the body was sent, so a backend or a proxy
  that never answered held the `data.sock` connection and its permit for
  ever; the agent now gives it 4 min after body end (under the SDK's
  5 min) and answers `503` "the backend took the whole body and sent no
  answer within 240s".
- **`3pm fleet ssh -R` / `-L`: a burst of connections waits its turn
  instead of being refused "… or 8 of yours opening".** A node lets one
  caller hold 8 handshakes at once, but `-R` takes over up to 15
  connections and `-L` carries up to 16: the command line now opens at
  most 8 sessions at once (the shell's own included), the tunnels still
  up to their caps once open, and places again — up to 3 times, 100 to
  400 ms apart — a forward the node turned away in the moment its slot
  was being freed. Nothing changes on the wire.
- **SDK abandon sink (Node, Rust): an `INIT` no longer outlives the
  ceiling, and a stale head no longer takes fresh frames.** Every `INIT`
  was kept, so a stream that sent many of them grew past the 8 MiB
  ceiling without bound, and the window was judged on the queue's head:
  a stale `INIT` there dropped frames that had just arrived. As in the
  agent's sink, only the newest `INIT` is kept now (an older one is an
  ordinary frame), a frame goes past the window on its own age and
  never as the newest, and the byte ceiling holds.
- **Node SDK: a frame the wire cannot carry no longer ends the leg.** A
  payload over 1 MiB or a `ts_us` that is not an integer threw inside
  the sink, which failed the stream and the session's leg with it (and
  the stream an app had `pipe()`d in). `writeFrame` now returns `false`,
  the sink counts the frame in `stats.refused` and says it on
  `'refused'`, a piped frame is refused the same way, and
  `session.send()` counts only the frames that went.
- **Node SDK: `close()` gives a stream session's slots back.** A closed
  session kept its stream slot and its call in flight until the reply,
  which for a callee that never answered was the ticket's deadline, a
  day away: 16 closed sessions made every next `io.open` `busy`. Closed,
  it releases both at once; the call stays routed, so a late reply is
  still `result`, for `reply_grace_ms` (10 s at most, one timer shared
  with a broken leg's grace), then `result` rejects `lost`. The daemon's
  own forward (64 per host) is still held until the reply or the
  deadline.
- **Node SDK: `io.mjpeg` no longer cuts a camera image at its EXIF
  thumbnail**, whose own `FFD9` sits in the `APP1` header: the end is
  looked for past the headers, from the scan (`SOS`) on. A second
  `frames()` on a reading leg that is already read throws a clear error
  instead of waiting on bytes the first reader took.
- **A blob read whole through `data.sock` spends its token.** The token
  waited for an end of stream hyper never asks for (a body that says it
  is over after its last frame, or one empty from the start), so it
  stayed valid until its deadline and the store filled up. It is now
  spent with the last frame handed to hyper, or when such a body is
  dropped.
- **One descriptor budget for every data socket.** Each hosted user's
  socket had 512 connections of its own on top of `data.sock`'s: the
  512 are now shared, the hosted sockets get half of them together and
  a quarter each, so `data.sock` keeps its half. The `503` written past
  the ceiling is capped too (64 in flight, then the connection is
  closed unanswered), and a failed `accept` (`EMFILE`) pauses 100 ms
  instead of spinning — on the data sockets and the LAN listener.
- **A call cut at its deadline settles its local tokens.** They were
  settled on the call's last lines only; a call cut by its deadline or
  aborted with its connection left them valid until their own deadline.
- **A LAN neighbour's `404` body is read to 4 KiB at most** before its
  connection is reused; a longer, cut or late one dials again.
- **A blob's answer and refusal are read to an answer's ceiling**
  (32 MiB) by `fleet call --file/--out`, not to a blob's 2 GiB.
- **A `?pipe=1` leg opens only with the grant `POST /legs` mints.** Any
  call's write token could switch its blob to the lossless pipe path;
  the pipe now has its own grant direction.
- **`3pm update` no longer runs an app twice after a clock step.** On
  Linux the start time a spawn records (the pid-reuse guard) is derived
  from the wall clock (`btime`), so when the clock moved between an
  app's spawn and the update, the new daemon image took the live child
  for a reused pid and restarted its row beside it: the old instance kept
  running unsupervised (on a tablet, two `agent`s of the same node kicked
  each other with `ERR_SESSION_REPLACED`) and the ones that exited were
  left zombies. Adoption now trusts the kernel: a pid still unreaped is
  our own child and is adopted, whatever its recorded start time (a
  warning notes the mismatch); the emergency stop no longer withholds
  its `SIGKILL` on that mismatch either.
- **A caller whose key the node refuses reads the node's sentence again,
  behind a proxy.** The node ends its legs before its answer reaches the
  backend; in production (nginx) the cut screen leg (`error decoding
  response body`) reached `3pm fleet ssh` first, the call was read back
  once, still unsettled, and the person read `the leg failed: the screen
  leg failed: …` instead of `key_not_authorized` and its `sudo 3pm link
  authorize …`. After a leg fails, `ssh`, a command, `attach`, `cp`,
  `-L`/`-W`/`-R` now read the call back every 200 ms until it settles,
  within 3 s, and print the node's refusal when there is one.
- **`3pm fleet …` no longer warns "this node's token was taken …, past
  the backend's 10-minute grace — is the agent still running?" on a
  healthy node.** The agent writes `session.json` only when its session
  changes, so a link stable for more than ten minutes left a file older
  than the grace, read as an agent killed. The writer is now judged by
  its `pid` (`kill(pid, 0)`, `EPERM` counting as alive — a root agent),
  never by the file's age; no more writes to the disk. The warning, and
  `token()`'s "the agent is not running any more" (now naming the pid),
  only come when that process is gone.
- **A forward no longer loses the end of its answer.** Once both ways
  were closed, the node let the caller's input leg go while its output
  leg still held the answer's last bytes, a window behind its reader:
  the relay gives up a pipe whose one reader left before its writer
  finished, and took the other leg with it (`408` on the `PUT`, the
  tail cut). The node now holds the input leg until the output leg's
  `PUT` has ended (`pump_then_send`, `shell/forward.rs`), and the
  command line holds the output leg until its input leg's `PUT` has
  ended (`carry`, `fleet/ssh/forward.rs`). `-L`, `-W` and `-R` alike.
- **A `3pm` killed outright no longer leaves its module running.** A
  SIGKILL (or the OOM killer) on the `3pm` behind `3pm <module>`
  forwarded nothing, and the module ran on as an orphan — `3pm fleet
  ssh` holding its legs, its keepalives and a node's `-R` port. On
  every unix, `3pm` now hands the module the read end of a pipe whose
  write end only it holds (`THREEPM_PARENT_FD`): its death is the
  module's EOF, and the module raises `SIGTERM` to itself — its normal
  way out, so `-R` still sends its `END` and the node closes the port.
  On Linux the kernel's `PR_SET_PDEATHSIG` (SIGTERM) does it too. Only
  the dispatch path: modules the daemon runs as apps are untouched, and
  `3pm` drops the variable from its own environment, so a daemon or an
  app never inherits it. An attached `3pm serve` (a page, `--attach`)
  now treats that SIGTERM like Ctrl-C and deletes its app — before, it
  died and left the server running unheld, on Linux too.
- **`3pm fleet ssh -W` closes stdout when the destination hangs up.**
  tokio's `Stdout` never closes fd 1, and `3pm` itself kept a copy, so
  the ssh running it as its `ProxyCommand` never saw the EOF it waits
  for and hung after the remote end closed. The module now owns stdout
  (fd 1 pointed at `/dev/null` behind it) and closes it on the
  destination's end; `3pm <module>` gives up its own stdin and stdout
  to `/dev/null` once the module runs.
- **`kill`, a closed terminal or Ctrl-\ reach the module behind `3pm
  <module>`.** `3pm` forwarded only `SIGINT`: a `SIGTERM`, `SIGHUP` or
  `SIGQUIT` ended `3pm` alone and left the module running (`3pm fleet
  ssh` holding the terminal in raw mode). All four are now passed to
  the module, and `3pm` exits with its status. `SIGWINCH` is not: the
  module shares the terminal. A signal `3pm` was started ignoring
  (`nohup 3pm serve .`) stays ignored by the module, as before for
  `SIGHUP`; `SIGINT` ignored by a script's `&` no longer reaches it
  either.
- **A process tree no longer adopts a stranger for good through a
  recycled parent pid.** A process the table read long ago still named
  its dead parent's pid; once an app child was born on that pid, the
  stranger joined the app's tree and stayed (memory keeps a member).
  A row older than its parent is no longer its child.
- **Nor through a member reborn on a dead stranger's pid within one
  tick.** The table kept the stranger's row for the newborn (its start,
  its parent), old enough for the stranger's orphans to pass as the
  member's children, and memory kept them for good. Nothing is claimed
  below a row the tick did not read; a stranger adopted from such a row
  brings its children the next tick, when it is read as a member.
- **A process adopted between two full sweeps no longer reports a
  spike of cpu.** Its first refresh in a while spanned every tick it
  was left alone and read as that much work done in one tick (737 %
  measured for one busy core); it now reads 0 % until two refreshes one
  tick apart, like a newborn.
- **A resource guard ends `errored` under a slow metrics sampler too.**
  A trip was judged stable past `min_uptime` + 10 s whatever the
  sampler's period: with `THREEPM_METRICS_INTERVAL_MS` at ~5.5 s or
  more, the CPU guard's first possible trip (the second sample) landed
  past that line, and an app over its limit from the start restarted
  forever. The margin is now `max(10 s, 3 × period)`
  (`GUARD_SETTLE_PERIODS`, `settle_margin_ms`); `MemorySampled` /
  `CpuSampled` carry the period (`sample_period_ms`), which the sampler
  now takes from the supervisor (`SupervisorHandle::metrics_interval`),
  and `3pm internals` reports it.
- **Nor under `wait_ready`.** A trip was judged on the uptime since the
  spawn, but a guard samples `online` rows only: an app that never
  signals ready went `online` at `listen_timeout` (15 s, say), tripped
  past the line and looped for ever. The uptime now runs from the
  incarnation's `online` (`ProcState::online_at`, carried across `3pm
  update` as `online_at_age_ms`; a manifest without it falls back to
  the spawn).
- **A process stuck in the kernel no longer stops every app's metrics
  and resource guards.** The tick read `/proc/<pid>/environ` and
  Extended Monitoring's `io` / `smaps_rollup` in place; on a process in
  `D` state those reads never return, and the whole sweep — so every
  sample, `max_memory_restart` and `max_cpu_percent` — waited with
  them, without a word in the log. They now run on a reader thread
  under a 500 ms deadline and a 1 s budget per tick (`metrics/risky.rs`,
  `metrics/guarded.rs`): a read that misses (or panics) is retried on
  that process with a backoff of 1 to 60 ticks, not at all while its
  reader is still stuck, and the reads over the budget wait for a later
  tick; meanwhile the process is not adopted through its environment
  and keeps its last extended counters. At most four readers are left
  stuck, daemon-wide; past that both reads are off until one returns.
  A read not made is no answer: a newborn deferred by the budget is
  probed on the next tick, never made to wait out a backoff. The
  refresh with `THREEPM_PROCESS_TREE=off` and `process_start_time`
  (run on the supervisor) no longer use sysinfo's default refresh,
  which read `/proc/<pid>/io` and the tasks too; on macOS
  `process_start_time` reads `PROC_PIDTBSDINFO` instead of sysinfo,
  whose refresh reads a new pid's `KERN_PROCARGS2` (still so in the
  sweep, unguarded: the stall report is the backstop there). A sweep that still
  runs past `max(5 × period, 5 s)` is logged as an error and shown by
  `3pm internals` (`metrics sweep stalled for …`,
  `DaemonInternals.metrics_stalled_ms`, counted from when that sweep
  began, additive); the loop waits for it (one sweep at a time) and
  stays cancellable. `ExtMemo::fill`'s reader now takes the
  `(pid, start)` identity and may skip a member (`ext::Skipped`).
- **The host agent always comes back.** A wedged agent (stopped,
  deadlocked) was never restarted: only its exit was noticed, and
  `hostmetrics` said `unavailable` until `3pm update`. The daemon now
  pings it every 5 s (`THREEPM_HOSTAGENT_HEALTH_MS`, min 100 ms, each
  ping bounded by `min(3 s, period)`), from one period after its
  readiness; 6 misses in a row (~30 s) put it down (SIGTERM, SIGKILL
  after 2 s) and respawn it through the usual backoff, logged with the
  reason `wedged`; `host_agent_restarts` counts the respawn. An agent
  answering with another protocol is alive and left alone, named once
  in `3pm.log` (a respawn would run the same binary). On Linux, a
  respawn after `install.sh` replaced the binary failed every time
  (`current_exe()` reads `<path> (deleted)`): the agent and the handoff
  guardian's fallback now exec `/proc/self/exe`, the running image,
  when the path is no longer usable (`supervise::self_exe`), and the
  path otherwise, so the process keeps its `3pm` name.
- **No ghost `3pm __daemon` after `kill`.** Dropping the tokio runtime
  waited without bound for its blocking threads, so a metrics read
  stuck in the kernel kept the process alive after its lock and sockets
  were released. The daemon (fresh or after a handoff) and the host
  agent now shut their runtime down within 2 s (`bootstrap::finish`,
  `RUNTIME_GRACE`) and exit.
- **Application metrics expire even when the sampler stalls.** The
  120 s TTL sweep ran on the metrics sample only; it now has its own
  10 s ticker (`APP_METRICS_PURGE_EVERY`, `SupMsg::AppMetricsPurge`).

### Added
- **A Rust SDK: `threepm-io` (`packages/rust`).** The fourth SDK of
  `docs/app-metrics.md` — metrics, events, `notify_error` and custom
  actions — with two dependencies (`serde`, `serde_json`) and std
  threads, no async runtime. Every line it writes is decoded by
  `threepm-protocol` in its own tests, and it replays the frozen
  `limits.json` and `exphist-vectors.json` gates like the other three.
  Two things differ because a thread is not an event loop: a `metric`
  callback runs on the flush thread, and nothing holds a Rust process up
  — a short-lived program flushes before `main` returns, and a worker
  that only answers actions ends its `main` with `io().serve()`. Not
  implemented yet: `io.call`, streams and blob action inputs/outputs
  (everything on `data.sock`), the built-in `3pm:profile`, and Node's
  `builtins`. Published on the download site like the others
  (`just publish-sdk`), as a `.crate` plus a static sparse Cargo registry
  under `/sdk/cargo/`: `[registries.threepm] index =
  "sparse+https://3pm.edgecr4ft.net/sdk/cargo/"` in `.cargo/config.toml`,
  then `threepm-io = { version = "…", registry = "threepm" }`. Surface,
  bounds and every difference from the Node SDK:
  `packages/rust/README.md`; an instrumented example app in
  `../examples/sdk-example-apps/rust/`.
- **A call's file goes agent to agent between LAN neighbours.** When
  the backend judges the two ends of an `io.call` with a file to be on
  one LAN (same public address, both announcing a listener), the
  caller's agent keeps the file for the target's agent, which fetches it
  over TCP + TLS 1.3 authenticated by the two nodes' own Ed25519 keys
  (raw public keys, no certificate) instead of through the backend. The
  SDKs see nothing: they `PUT` and `GET` on `data.sock` as before, and
  the relay stays the fallback — a neighbour that does not answer
  within 300 ms per address, or a reader that does not come within one
  second of the writer, and the bytes take the backend like any other
  call's (a pair that failed is not tried again for a minute). One copy,
  never two: the rendezvous is decided under one lock, and a late reader
  is told `409` and goes to the relay. The agent opens its LAN listener
  on a port the kernel picks and announces it in `hello.direct_port`
  (never an address: those are the backend's, from the node's host);
  `direct = false` in `link.toml` turns it off, and a `3pm link --force`
  keeps what was written there. The writer reports which way it went
  (`path_report`), and the call shows `path: direct | relay`. A call's
  **input** only: its output is read after the call replied, too late
  for any rendezvous, so it keeps the relay. A **session**
  (`io.open`) goes direct both ways when both nodes are set to `all`
  (`setNodeAccess.direct`): each leg through the same abandon sink as
  on the relay, and its writer says how far it got every 5 s
  (`blob_stat`), the backend having no other counter of it. Wire: `threepm_link::direct`
  (`Peer`, `Path`, `Peer::dialable`, `lenient_peer`), `Hello.direct_port`,
  `Action.peer`, `Link::{PathReport, BlobStat}`, `LinkConfig.direct`
  (fixtures under `v2/` only grew).
- **An action declares the most its file may hold: `output.max_bytes`.**
  `io.action(name, {output: {max_bytes}}, …)` — an integer, 1 byte to
  256 MiB (`ACTION_OUTPUT_MAX_BYTES`, frozen in `limits.json`); any other
  value leaves the action unregistered (logged), since the daemon refuses
  a source's whole list over one bad declaration (`validate_decl`).
  `ctx.commitFile` stats the `.tmp` first and, past the bound, deletes it
  and throws `too_large` naming both sizes — before any `PUT`, rather than
  the relay cutting the upload far from its cause. The bound crosses the
  link as `AppAction.o` (`threepm-link`, additive: an older agent or
  backend ignores it and reserves 256 MiB, as before), so the fleet
  backend reserves what the action declares for a caller that asks for
  its file: a 2 MB screenshot costs its caller 2 MB of blob quota, not
  256 MiB. The agent carries it without a change of its own (it builds
  `AppAction` through `threepm-link`).
- **One diff engine on the link: the node's whole state as `state`
  frames (link version 2).** The agent sends **one** frame a tick
  carrying what moved in the node's state — the host, the process table,
  each instance's application metrics, each process tree, as independent
  sections cut with JSON merge patches; an empty frame is the heartbeat.
  The daemon hands over complete states and the agent is the only place
  a diff is cut; the backend rebuilds the node with one reader.
  `threepm-link::node_state` (`NodeWriter` / `NodeReader`, `StateFrame`,
  `SectionPatch`, `StateGap`, the four section shapers and their typed
  views), `Link::State`, `Link::StateResync {secs}`,
  `welcome.state_every_s`, the `state_frame` fixture family;
  `modules/agent/src/node_state.rs`. A process's `uptime_ms` no longer
  travels: its start does, once, and the reader derives the uptime from
  the frame's clock. Measured floats travel at 3 significant digits (an
  application's own values are never rounded), and what does not move
  costs nothing after the first frame. Every section goes whole at the
  start of a session and on `state_resync`; its CRC then, and one a
  minute, check the backend's rebuild. Measured on two real 10-minute
  traces, application metrics and trees included: 357 B/s on a dev Mac
  with 11 apps (the host and the table alone cost 525 as whole
  `telemetry` frames), 463 B/s on a Linux server (755).
- **`require('@3pm/api/fleet')`** — the fleet backend's HTTP API, for
  Node. A method per endpoint of `3pm-backend`'s route table, the live
  feed as `api.follow(clusterId)` (read with `fetch` and a
  `ReadableStream`, the only way Node has), `api.url(name, …)` for the
  routes that are bytes rather than JSON, and `ApiError {status, code,
  detail}`. It is the browser client generated a second time as
  CommonJS, from the same template, with the browser half — the token
  prompt, the `EventSource` reader — cut out; `just api` in
  `3pm-backend` writes both, and skips this one when there is no `3pm`
  checkout beside it. The package gains an `exports` map naming `.`,
  `./renderer`, `./fleet` and `./package.json`: the three that worked
  before work unchanged, and **deep imports (`@3pm/api/lib/…`) are now
  closed**, which nothing documented used.
- **Batteries in `3pm hostmetrics`.** A `batteries` section — one row per
  pack (state, charge, health, watt-hours, draw, cycles, temperature),
  titled with whether the host is on mains — and a `bat:` segment on the
  `3pm ls` host line, coloured low-is-bad and marked `⚡` while charging.
  Linux only, from `/sys/class/power_supply`: world-readable files, no
  root and no extra dependency (the previous answer was the
  `battery.js` example app). Wire: `batteries[]` and `on_ac` on
  `HostMetrics`, additive. The kind of a supply comes from its `type`,
  never its name, a `scope=Device` supply (mouse, headset) is excluded
  while an absent `scope` means System, amp-hour gauges are converted
  through their voltage, and the draw is carried unsigned with `status`
  giving the direction. Every reading is optional and **an absent one
  stays absent**: a full pack draws a real `0 W` and a pack with no
  sensor has no temperature — the renderers show `N/A`, never a zero.
- **`@3pm/api` streams end, feed and measure themselves.** `s.end()`
  half-closes a session on either side: it writes `END` on what this side
  sends and keeps reading until the other side's own `END`.
  `s.send(readable, { end })` feeds a source one frame per chunk and ends
  the leg with `END` when the source ends, dies or `stop()` is called
  (`await sending.done` is `{ frames, error }`, never a rejection). The
  reading side counts what the path lost: `s.stats` gains `received`,
  `bytes_in`, `missing` (the holes in `seq`, which every abandon point
  leaves, where `dropped` only saw this process's) and `discont`, and the
  reading leg's `stats.latency` gives `now − ts_us` as p50/p95/max over
  the last 256 frames (exact on a round trip, clock skew included across
  hosts). The abandon sink emits `drop` for each frame it throws away, so
  a producer can lower its rate. The wire is unchanged; a stream handler
  that returns nothing replies with the four new counters too.
- **`@3pm/api` knows its own node.** `io.node` is this host's node as
  `3pm link` recorded it (`{id, cluster_id, server, linked_at_ms}`, read
  from the agent's `link.toml`, never `session.json`; `null` when not
  linked). `io.open(null, …)` — or `'self'`, also accepted by `io.call` —
  opens the session on this very node through the fleet instead of being
  refused by the daemon, and rejects `node_offline` ("this host is not
  linked") without a link; `io.call(null, …)` stays local. A script no
  longer has to guess its node with `os.hostname()` (`MacBook-Pro-2` for
  a node enrolled as `MacBook-Pro-2.local`). A `CallError` prints its
  code: `String(err)` is `code: message`, `console.error(err)` shows
  `CallError [code]: message`.
- **`@3pm/api` keeps a serving process up, and a call waits for the
  daemon.** A process that declares an action of its own (`io.action`,
  `io.stream`) no longer exits once its script has run — its control
  socket is unref'd, so a worker that only served calls used to exit at
  once and be restarted until `errored`; the SDK's `3pm:profile` holds
  nothing, `keep_alive: false` opts out, the daemon's `shutdown` lets go;
  a renderer's io (`createRendererIo`) never holds, its window does.
  `io.call` / `io.open` placed right after `io.init()` wait up to
  `connect_wait_ms` (5 s) for the control connection instead of failing
  `node_offline` ("no control connection"), and `await io.ready()` says
  when it is up. A script whose only work is `await io.call(…)` (or
  `io.open`) no longer exits mid-call with code 0: the SDK holds the
  process while a call is in flight.
- **Documentation on the download site: `just publish-docs`** puts the
  docs of 3pm, the three SDKs and 3pm-backend on
  `https://3pm.edgecr4ft.net/docs/` as Markdown, with `llms.txt` (an index
  for LLMs: every document and every backend route, one line each),
  `llms-full.txt`, `docs/index.json`, `sitemap.xml` and a Documentation
  section on the landing page. The CLI reference walks every command's
  `--help`, the backend's routes are asked of a freshly built backend
  (`GET /api/v1/routes`), and two pages are written for the site: concepts
  and every environment variable (`distribution/site/docs/`). The site
  itself — what is online, the three commands that upload it, the landing
  page, the server and its vhost — is documented in `docs/SITE.md`.
- The host snapshot says whose box it is: `HostMetrics.user` (the account
  the daemon runs as, `getuid` → passwd, T0 identity) and
  `HostMetrics.sessions` (the accounts with an open login session, from
  utmp, T2 medium, sorted, deduplicated, at most 64). Both additive and
  elided when empty; a daemon predating them sends neither, and neither
  is ever faked where the platform has no source. On the link, `user`
  joins the static allowlist (it rides `host_full` and a change of it
  triggers a fresh one), `sessions` rides every tick.
- `3pm fleet nodes` shows `user`, `public` and `private` next to what it
  already showed: the account each node runs under, the address its link
  reaches the backend from, and its interfaces' private addresses. `—`
  for a node whose agent predates the fields. On a terminal too narrow
  for the table — and only then — the `3pmn_…` ids are cut to twelve
  characters, as `3pm ls` cuts a long app name; a pipe is never cut.
  `--json` carries them untruncated and adds `sessions`, which the
  backend sends to a node only under `fleet:read`; `3pm fleet inspect`
  names `user` on its card.
- `3pm fleet ls` is an alias of `3pm fleet nodes`.
- Fleet RPC, local wire (`threepm-protocol`): `ActionDecl.{input, stream,
  concurrent}`, the `call` request and its `called` reply, the control
  kinds `call`, `action_started`, `fleet_ready`, `forward_update`,
  `call_ticket`, `call_reply` and `forward`, the `live` message framing
  with its frozen vectors, and the shared call bounds. Additive:
  `PROTOCOL_VERSION` stays 1 and every fixture grew by addition only.
- Calls with bytes: an app or the CLI can call an action of this host with
  a file (`io.call`, `Request::Call`), and the daemon reports one owning
  answer per call (`call_ticket` then `call_reply`, or `called`).
- An authenticated fleet gate: a call that names another node of the
  bucket leaves through the agent's own control connection, which the
  daemon believes only on a per-launch secret it put in that process's
  environment.
- `ActionDecl.{input, stream, concurrent}` are enforced: the bounds of an
  input are checked before a single byte moves, and the concurrency window
  of an action, of a connection and of the daemon are three named rules.
- `action_started` restarts a handler's own budget once, so the transfer of
  a large input no longer eats it.
- Fleet RPC, Node SDK (`@3pm/api`): `io.call` / `io.open` / `io.stream` /
  `io.mjpeg`, an action's `input` / `stream` / `concurrent` declaration,
  `ctx.input` and `ctx.commitFile` over the agent's `data.sock`, the
  `live` message framing (replayed against the crate's frozen vectors)
  and the abandon sink. Additive: existing exports, declarations and
  replies are byte-identical when none of it is used.
- Fleet RPC, Node SDK: two opt-ins that spare a pass over a large input,
  both off by default and neither touching the wire. `io.call(…, { file,
  sha256, bytes })` declares a digest the caller already holds instead of
  reading the whole file to find it again (the size is checked against
  the file, the digest is taken at the caller's word and verified by the
  callee as ever); `io.action(name, { input: { stream: true } }, …)` has
  the handler read the body from `ctx.input.stream()` as it arrives,
  nothing being staged on disk. Three differences from the default,
  documented in the README and in FEATURES: `ctx.input.path` is `null`
  on a call from another node (a call placed on this host still names
  the daemon's file, and `stream()` reads it); the size and digest the
  call declared are checked at the **end** of the body, so bytes are
  unvouched for until the iteration ends; and the transfer counts
  against the handler's `timeout_ms`.
- `3pm call`: one action, one instance, one answer, with bytes allowed in
  (`--file`) and out (`--out DIR`), on this host or on a node of the
  bucket (`--node`). A call that names a node carries no file from the
  command line — the CLI has no data socket — and says which command
  does.
- `3pm trigger --file` and `3pm trigger --out DIR`: the bytes of a
  trigger. `--file` makes it a call, so it names one instance.
- `3pm fleet …`: the bucket's nodes, apps and calls, from this host's own
  link — sugar over the bundled `agent` module, with a clean refusal when
  the module or the link is missing.
- Fleet RPC, link wire (`threepm-link`): `welcome.fleet_token` (the
  node's bearer, redacted in `Debug`) and `welcome.pending_calls`, the
  bytes of a remote action (`action.{input, output, stream}`),
  `RemoteOutcome.output`, and the `call_result` frame for a call the
  node placed. Local wire: `trigger.{input, output, stream}` and
  `proc_action.{placed_by, input_bytes}`. Additive: `LINK_VERSION` and
  `PROTOCOL_VERSION` stay 1, and every fixture grew by addition only.
- **`3pm fleet` on a linked host.** `3pm fleet nodes`, `3pm fleet apps
  [node]` and `3pm fleet inspect <node> [app]` (sugar for `3pm agent
  fleet …`) read the node's own bucket with the node's own bearer, so
  the owner's account token never has to sit on a server. One request
  answers the first two; `inspect` adds the host, an app's series and
  its declared actions when the node is granted `fleet:read`. `--json`
  everywhere, `--ca-file` for a private certificate authority. A node
  is named by its `3pmn_…` id, its hostname, or an unambiguous start of
  one.
- The agent keeps the bearer of the last `welcome` in
  `<home>/agent/session.json`, now **owner-only (0600)**, with the
  instant it arrived: a command run while the link has been down longer
  than the backend's ten-minute grace says so before it tries.
- `3pm fleet top`: the bucket live in two panels (nodes, processes), from the
  backend's event stream, redrawn once a second; `q` / `Esc` / Ctrl-C quit.
  Needs `fleet:list` only.
- Fleet calls with bytes: the agent serves `agent/data.sock` (HTTP/1.1 on a
  Unix socket, per-call local tokens, one blob one direction) and relays to
  the backend without holding anything, opens an authenticated fleet gate on
  the daemon's control socket, places a node's calls on `nodeAction`, and
  hands the result back as `forward_update`. `3pm agent fleet call` places one
  from the command line, `--file` / `--out` included. `--out` builds its
  file through a temporary only this call can have created, so neither a
  link left at that name nor the leftover of a crash is written through.
- Live streams across the fleet: `io.open` on one node and `io.stream` on
  another exchange framed messages through `agent/data.sock` — a chunked
  `PUT` out, a `GET ?live=1` back, relayed to the backend as they arrive.
  The agent cuts what it relays on message boundaries, never mid-message,
  and drops its oldest ordinary messages instead of queueing them when the
  backend is behind, keeping the `INIT`, the last `KEY` and the `END` a
  later reader joins on — never the only message it holds, however big.
  A session's ticket names both legs (`in` and `out`, no `put`) and its
  tokens live as long as the session, up to 24 hours. A chunked `PUT`
  whose bytes are not `live` messages, or that stops in the middle of
  one, is refused `422`; a session that names an `output` of its own is
  refused before it is placed (its answer is its second leg).
- `sdk-example-apps/fleet-call/camera.js`: the streaming half of the
  example — `ffmpeg -f mjpeg -flush_packets 1` into `io.mjpeg` and
  `io.open`, with `vision.js` answering one summary per image on a
  `watch` stream action.
- `3pm cd <app>`: the app's working directory. On a terminal it opens
  `$SHELL` there (`exit` returns); piped or in `$(…)` it prints the path,
  so `cd "$(3pm cd api)"` moves the current shell; `--json` gives
  `{"name","cwd"}`. One name or id: `all` and patterns are refused.

### Changed
- **`3pm update` restarts every internal process, every time.** The
  apps are what a handoff keeps running; 3pm's own processes are not.
  The daemon was already re-executed and the host agent already
  replaced at every handoff; the module apps — the fleet agent, the
  `3pm serve` sites, a module's `[[apps]]` — were restarted only when
  the bundle's version had moved, so a same-version `update` left them
  on their old process (and the dev loop ended with a manual
  `3pm restart agent`). Every **live** module app is now restarted once
  the new daemon serves: same row, same spec, same stamp, nothing
  reconciled. A module app whose every instance is stopped stays
  stopped; the cold path, which has just restarted everything, still
  restarts only the rows behind their stamp; `--skip-modules` leaves
  them all alone. The price: at every `update` the fleet link drops for
  the time of an agent restart (it comes back on its own, no new
  enrollment) and a served site blips. The report says
  `Module [agent] 0.10.0 — 1 app(s) restarted`; in `--json`,
  `modules[]` is now there at every handoff that has a live module app,
  with `from == to` for a module restarted as it is.

### Fixed
- **Process trees during a handoff.** `3pm tree` (the `process_trees`
  request) was refused with `handoff_in_progress` while the daemon
  quiesced, although the client re-sends it as a read; the daemon now
  serves it like every other read. Both sides classify requests through
  one exhaustive match each (`Request::is_read_only`,
  `Command::is_read`), pinned to agree by a test.
- **`completions` and `__man` without a usable home.** Like `validate`,
  they no longer resolve `THREEPM_HOME` first, so a home too deep for a
  socket (or none at all) no longer fails them. `validate` now exits 1
  by returning, not with `process::exit`, so its output is flushed.
- **Modules resolve the home like the CLI.** `3pm link`, `unlink`,
  `fleet`, `serve`, `autoupdate` and the agent itself used `$THREEPM_HOME` or
  `~/.3pm`; they now use `client::resolve_paths`, which also finds the
  home the root daemon hosts for the account when run directly.
- **The agent survives a poisoned lock.** Its data socket, tokens,
  hosted sockets and `fleet top` model took their mutexes with
  `.lock().expect(…)`: one panic under a lock made every later caller
  panic too. They go through the poison-tolerant `lock()` now.
- **An app started after a `3pm update` can be stopped again.** The
  handoff blocks SIGHUP/SIGINT/SIGUSR2/SIGTERM around its `execve` so a
  signal arriving in the window reaches the new image; the new image
  unblocked them on the thread that boots, but a signal mask is per
  thread and survives `fork` **and** `exec`, so every tokio worker built
  before that kept them blocked — and every app forked from one of those
  workers inherited the blocked mask. `3pm stop` sent the app its
  `kill_signal` (SIGINT by default), the app could not receive it, and
  the daemon SIGKILLed it `kill_timeout_ms` later: no flush, no clean
  socket close, +1.6 s on every stop, restart, reload, redeploy and
  `3pm link --force`, and an app listening on SIGHUP or SIGUSR2 silently
  stopped reacting. Nothing showed it but the daemon log (`signal=9`)
  and the delay. The spawner now clears the mask in the child, between
  `fork` and `exec` — a supervisor hands its children a clean mask
  whatever thread forks them, and the standard library deliberately
  passes the parent's on. **The host agent was hit the same way**, by
  its own launcher: respawned by the new image at every handoff, it was
  reborn with SIGTERM blocked, so the *next* `3pm update` waited its 2 s
  grace and SIGKILLed it (`host agent ignored SIGTERM; killing` in the
  daemon log, once per update). Both spawn sites now share one helper
  (`sigmask::clear_in_child`). A daemon started cold was never affected,
  macOS was never affected, and Node apps never were either (Node resets
  its own mask at startup) — Rust, Python and shell apps were. 3pm's own
  processes heal at the update that brings this (they are restarted by
  it, see above); an app already running with the blocked mask picks the
  fix up when it is next restarted, and that stop is one last SIGKILL.

### Changed
- **One version for the whole release.** `just publish` writes 3pm's
  release version into the agent (`Cargo.toml`, `module.toml`), the
  Node, Python and C++ SDKs (`package.json`, `pyproject.toml`,
  `__version__`, `CMakeLists.txt`, the install lines of the docs) and
  the backend checked out next to 3pm (`Cargo.toml`, the version of its
  generated clients), so every release ships all three SDKs and the
  backend it deploys reports the same number. The agent and the backend
  used to report 0.1.0 whatever the release; the SDKs went from 0.1.x
  to the release number.
- **A node awaiting approval retries every 5 s at most** (was 1 min,
  growing to 10 min): approving it on the dashboard now brings it
  online within 5 s instead of up to 10 min later.
- Node SDK types: `ActionInputFile.path` is now `string | null`. It is
  `null` only for an action that declared `input: { stream: true }` and
  was called from another node — every other path is unchanged, and no
  consumer in this repository dereferences it unguarded. A TypeScript
  consumer under `strictNullChecks` that reads `ctx.input.path` without
  a check will need one.
- **The agent (`3pm link`) runs on four tokio workers and a blocking
  pool capped at sixteen**, instead of one worker per core — 33 threads
  on a 32-core host, 5 at rest now. The blocking work the module writes
  itself (`commands::pack`, the gzip of a call's result) is one bounded
  pass per call, but the pool is shared with the libraries and DNS lives
  there: `lookup_host` and hyper-util's resolver are `spawn_blocking`
  calls that cannot be cancelled, so sixteen keeps a burst of slow
  lookups from queueing every later packing behind them. The constant
  carries that inventory and a test fails if a new caller appears
  outside it. Same change, and the same absence of cost, as the host
  metrics agent's 38 threads to 8. The resident set never depended on
  it: `MALLOC_ARENA_MAX=2` is what settled that.
- **`@3pm/api` is no longer published on the npm registry**: it ships
  from `https://3pm.edgecr4ft.net` with the other SDKs (`just
  publish-sdk`), and `"private": true` makes a stray `npm publish` fail.
  The 0.1.0 already on npm stays there, not updated.
- **Release binaries use thin LTO, and `just publish` builds everything
  in parallel**: fat LTO optimised all of `3pm` on one core, about a
  minute per target, and the four targets and their three workspaces
  built one after the other. The twelve builds now run at once, each in
  its own `target/dist/<triple>/<workspace>` with its log in
  `target/publish/logs/`; the first failure stops the others and prints
  its log.
- **`3pm link` needs no server address**: `3pm link <cluster-key>
  <token>` links to the hosted backend, `wss://fleet.edgecr4ft.net`. A
  third positional still names another one, and `THREEPM_LINK_SERVER`
  wins over both, so a machine or a provisioning script points its links
  elsewhere without touching the pasted line.
- **Python apps log without delay**: the daemon sets `PYTHONUNBUFFERED=1`
  for an interpreter named `python*`, a `.py` script run by its shebang
  (pm2), or a quoted command line starting with one (`"python3
  app.py"`), under the app's `env` (`PYTHONUNBUFFERED = ""` there turns
  it off). A `print()` without `flush()` used to reach the log
  8 KiB late.
- **A config file's script without an interpreter must be executable**,
  checked when the file is read (`validate` reports it) instead of
  failing at spawn as `errored` — the rule `3pm start <script>` already
  applied.
- **`app.config.js` is a config file** (pm2): `3pm start x.config.js`
  evaluates it instead of running it as a script.
- **An app's `cpu` and `mem` are those of its whole process tree** — the
  spawned pid and every descendant (children by parent pid, orphans by
  the leader's real process group, the classic double-fork daemon by
  the `THREEPM_ID`/`THREEPM_GEN`/`THREEPM_CTL_SOCK` it inherited and
  the daemon's lineage `THREEPM_BOOT` — so a survivor of an earlier
  daemon of the same home is never adopted —, the cgroup leaf under
  `cgroup_enforce`, the members already seen for as long as they live,
  and every descendant of a member; never a process older than the
  spawned pid, never 3pm's own daemon or host agent, which strip the
  identity variables from their environment however they were
  launched), not the spawned pid alone: an `npm run dev`
  that starts Electron and its helpers reads `62.2% / 3.0gb` instead of
  `0% / 71mb`. `listen_ports` is the union over the members. Memory is
  the sum of the members' RSS, an upper bound when they share pages.
  **The guards judge the tree too**: `max_memory_restart` and
  `max_cpu_percent` fire on the sum, so a child that grows recycles the
  app — per app, `limit_scope = "leader"` (ecosystem file) keeps them
  on the spawned pid, and an app saved by an earlier 3pm (`dump.json`,
  the `3pm update` manifest) reads back as `leader`, so an upgrade
  never widens a limit (`tree` is the default of a new `start`);
  `THREEPM_PROCESS_TREE=off` on the daemon restores the old sampler for
  every app. The tick sweeps the whole process table on its first run
  and every tenth tick; in between it refreshes only the pids born or
  gone since, the leaders, the members and 3pm's own, so a busy host
  costs the sampler its apps' processes, not the host's (one `sysinfo`
  pass, cpu and memory together, off the runtime, no stat file held
  open — a full sweep is about 12 ms for a thousand processes, priced
  by `3pm internals` as `last metrics tick`); a daemon with nothing
  running does not sweep. Never a thread, a zombie or, on macOS,
  another user's process; macOS withholds the environment of Apple
  platform binaries, so a `/bin/bash` that daemonizes before the first
  tick is not adopted (a `node` or `python` that does is, and so is the
  `bash` once it `exec`s one: an unreadable environment is read again,
  1 then 2, 4… up to 60 ticks apart), unless it was already a member.
  `docs/metrics.md` → *Process trees*.
- **Two more label keys are reserved for applications**: `proc` and
  `pid`, the dimensions the fleet backend queries an app's process tree
  by (one series per member, `subprocess.*`). Refused like the six
  below, by the SDKs and by the daemon; rename such a label.
- **Six label keys are now reserved for applications**: `core`, `iface`,
  `blockdev`, `sensor`, `gpu` and `array`, alongside the fifteen that
  already were. They are the dimensions the fleet backend queries a
  host's metrics by (CPU per core, per interface, per block device, per
  sensor, per GPU, per RAID array), so an application's own label of the
  same name would shadow them. An app that used one of these keys sees
  it refused like any other reserved key — by its own SDK, as a silent
  no-op series that `THREEPM_IO_DEBUG=1` explains, and by the daemon
  behind it (`app_metrics/limits.rs`, mirrored in the Node, Python and
  C++ `limits` and pinned by a test); rename it. `device` and `cpu` were
  deliberately *not* taken — they are the two an application is most
  likely to want for itself.
- A control connection's outbound queue holds 32 messages (was 8) and one
  source may run 16 calls at a time (was 4, and was one per action —
  `ActionDecl.concurrent` now says how many).
- The action listing of `3pm trigger <app>` and `3pm show` gained two
  columns, `kind` and `input`, and their `--json` gained the `stream`,
  `concurrent` and `input` fields every declaration now carries.

### Fixed
- **`--interpreter /bin/sh` runs `/bin/sh`.** A path given as the
  interpreter was resolved to the file it links to (`/usr/bin/dash` on
  Ubuntu). On Alpine, `/bin/sh` is BusyBox, which picks its applet from the
  name it is called by: called as `busybox`, it did not start the app. A
  venv's `.venv/bin/python` became the system Python, without the venv's
  packages. The path is now made absolute and its links are kept.
- **Docker containers are no longer counted as disks.** Each running
  container added an `overlay` mount the size of the disk under it (five
  extra 937 GB disks on the build host), which inflated the dashboard's disk
  space and added series at each new container. Overlay mounts are left out,
  except the one on `/` (3pm running inside a container).
- **Linux: another account's process no longer joins an app's tree
  through its environment.** The account check of the environment rule
  compares the process's uid with the row's, but the process table never
  asked Linux for uids (sysinfo reads one only on request), so every
  process passed it: under a root daemon hosting users, one account's
  process carrying another's `THREEPM_*` variables was counted in that
  app's tree, and its memory against the app's `max_memory_restart`. The
  uid is now read, once per process.
- **The LAN direct path no longer runs dry after 1 024 sessions a day.**
  An agent kept each direct transfer's grant and rendezvous slot until the
  call's deadline — a day for a session — in tables of 1 024, so past that
  every session silently took the relay. A leg now gives its room back when
  it ends (a 60 s tombstone still answers a late reader `409`), a call's end
  releases what it never used, and a full table is logged once a minute.
- **The LAN listener serves the LAN only.** It listened on every address;
  a connection from anything but a private address (`is_private_ip`) is
  now closed before the TLS handshake, so a node with a public IP exposes
  nothing.
- **A reader gives a neighbour one hold, not five seconds.** Behind a
  firewall that takes the TCP connection and never delivers it (macOS),
  a live leg lost 5 s of frames before its fallback; the whole dial —
  addresses raced 100 ms apart, TLS, head — now ends with the writer's
  1 s hold, and a reader that came before the writer knew it keeps asking
  for that hold instead of once. A writer no longer blames a neighbour
  that asked before it knew of the blob.
- **A hosted user's app reaches the agent's data socket.** Under a root
  daemon with `[users] allow`, an app run as a hosted user could neither
  push nor fetch a blob: the agent's only socket is root's
  `agent/data.sock` (0600, in a 0700 directory), and the
  `agent/data-<uid>.sock` the SDK looks for first was created by nobody.
  The daemon now creates `agent/` in each hosted home (root:gid 0750) and
  names one socket per hosted user on the fleet gate each time it opens
  (`ControlOut::HostedSocks`, `hosted_socks` on `ctl.sock`, additive; the
  whole list, empty when it hosts nobody); the agent binds each for its
  `uid:gid`, 0600, on the same server as `data.sock`, and closes one the
  list no longer names (a handoff re-reads `[users]`). An action of a hosted app can now return a file with
  `output: true` without a relay or `THREEPM_DATA_SOCK`. An older agent
  skips the line; an older daemon never sends it.
- **Node SDK: a stream session whose leg broke settles its `result`.**
  When a leg of an `io.open` session breaks under the caller (the link
  fell, the backend restarted), `session.result` waited for the call's
  reply until the ticket's deadline — a day for a session — so an
  application that reopens once `result` settles stayed stuck. The reply
  is still the answer when it comes; without one, `result` now rejects
  `lost` `reply_grace_ms` after the break (default 10 s,
  `OpenOptions.reply_grace_ms`). A session the application closes itself
  is not a break.

### Changed
- **`just publish` publishes the SDKs too**: after the 3pm tarballs it
  runs `publish-sdk`, which ships each SDK whose version is not online
  yet (`--dry-run` is passed on; `--force` is not — re-uploading an SDK
  version stays `just publish-sdk --force`). `@3pm/api` is 0.1.3.
- **The `state` stream, after its review** (before its first release).
  *Daemon:* a histogram whose scale alone changed while it held no
  bucket (every observation in the zero bucket) did not move
  `app_metrics_rev` — the rule a reader of full states lives by is "the
  revision moves exactly when the state does", and the agent would not
  have asked. Fixed, with the property test that holds the rule over
  random batches, source removals, TTL sweeps and resets
  (`app_metrics/state_prop_tests.rs`, the successor of the link-reader
  proptest that went with the daemon's diff). The gate the daemon's
  diff was removed behind is now measured on a realistic registry
  (labels, unit, description, histograms): a full dump at the series
  cap blocks the supervisor 0.71 ms on the build host, 0.16 ms on an
  M-series (`measure_a_full_dump_on_the_actor_thread`, ignored; figures
  in `3pm-rd/rd/old/bench/host-patch/results/daemon-full-dump.txt`).
  *Agent:* a section's CRC falls at its own moment of the minute
  (`paces::crc_phase_ms`) instead of every moving section being
  serialized and checked on the same tick; a tick no longer clones each
  patch to build its frame (`p` moves out) nor writes one to measure it
  (`node_state::json::json_len`); the host section is shaped in one
  pass. *Link:* `NodeReader::retain_gen` (a backend forgets what an
  agent that restarted did not bring again — it never says `gone` for
  it), `ProcsTable` (the table decoded once, read at any clock),
  `NodeWriter::holds`. `link-protocol.md` says why the sub-protocol
  stays `3pm.link.v1` and in which order a fleet crosses link 2 (the
  nodes first, out of band; the backend last).
- **Fleet calls under load, in the agent.** A command past the four in
  flight now waits for a permit until its own deadline instead of being
  refused at once (`commands.rs::CallCtx::admit`; a session is still
  refused at once). A `call_result` that arrives before the call's ticket
  was written is kept (up to 6 s, 64 calls) and follows the ticket out,
  where it used to be dropped and the caller waited 30 s
  (`fleet_gate/waiting.rs`). A `replied` result with no outcome and no
  error reads `lost`, not `ok`. Commands reuse their `rpc.sock`
  connections (`commands/daemons.rs`) instead of dialling and pinging the
  daemon per call and logging a line each time, and the HTTP client
  drops idle connections after 30 s, before a proxy closes them.
- **`@3pm/api` reads `link.toml` once per version of the file** for
  `io.call('self')` / `io.open(null)`, not on every call.
- **A `live` stream relayed by a macOS agent is no longer capped at
  6-7 MiB/s.** The agent pinned its relay sockets' `SO_SNDBUF` to
  128 KiB off Linux, which switches macOS's send-buffer auto-sizing off:
  each relayed `PUT` carried the buffer over the round trip, about a
  sixth of what the path held (40 MiB/s measured with the kernel's own
  sizing). macOS now keeps the kernel's buffer; `SO_SNDBUF` stays where
  there is neither auto-sizing worth keeping nor `TCP_NOTSENT_LOWAT`
  (Windows, the BSDs).
- **The agent's live sink drops frames by age, not only by bytes.** A
  queued ordinary message older than 100 ms (`QUEUE_AGE_MAX`, the SDK's
  and the backend's own window) is dropped instead of sent, checked on
  push and again just before a send. It used to wait out a 512 KiB
  queue — seconds of stale images near the link's cap. `INIT`, the last
  `KEY`, `END` and the newest message are still never dropped, and the
  byte bound stays as the memory ceiling.
- **Ending a stream no longer throws or crashes in `@3pm/api`.**
  `s.close()` during `for await (const f of s.output.frames())` threw
  `ERR_STREAM_PREMATURE_CLOSE`; the iteration now returns, and a failed
  call still throws its failure. `io.mjpeg(readable)` emitted `'error'`
  when its source failed or closed mid-image (`ffmpeg` killed), which
  killed an application with no handler; it now ends, drops the partial
  image and keeps the reason on `out.source_error`. Forwarding a received
  frame with `writeFrame(f, { key: false })` kept the `KEY` it came with:
  a named flag now decides its own bit, unknown bits still carried. A
  `writeFrame` after the leg ended is ignored (`false`) instead of failing
  the session.
- **One `live` stream was capped at about 4 MB/s over a link with
  latency.** The agent's relay set a 128 KiB send buffer on the socket
  it carries a blob over, to keep a frame's latency at 20 ms instead of
  800. Setting `SO_SNDBUF` does more than choose a size: it turns
  Linux's send buffer auto-tuning off for the life of the socket,
  pinning the bytes that may be in flight at twice what was asked.
  256 KiB over the 60 ms round trip of a fleet link is 4.37 MB/s, and
  4 MB/s of an offered 5 is what a hundred 50 KiB images a second
  measured under `tc netem delay 30ms`, the rest thrown away by the
  backend's window behind a queue that stood at 200 ms. On Linux the
  buffer is now the kernel's to size and `TCP_NOTSENT_LOWAT` alone
  keeps the queue shallow, which is what it is for: it bounds what is
  queued **unsent**, not what is in flight. Measured end to end under
  the same `netem`: p50 216 → **61.0 ms** at a hundred images a second
  (p90 61.3, floor 60), 261 frames lost → 10 of 1 485, and nothing lost
  at thirty. On the socket, 2 626 560 bytes of send buffer where there
  were 262 144 — up to about 43 MB/s a stream where the path permits it,
  a bound read off the socket rather than measured. Where there is no
  such knob
  (macOS, the BSDs, Windows) `SO_SNDBUF` is unchanged and so is the
  latency it bought. The backend carries the same change on the sockets
  it accepts; a fleet only half updated stays capped by the other end.
- **`3pm fleet …` no longer calls a healthy node stale.** The agent read
  the age of its fleet bearer whatever the state of the link, but a
  bearer is minted at a handshake and is good for as long as that
  session lasts — so any node linked for more than ten minutes had
  `this node's token was taken … ago, past the backend's 10-minute
  grace — is the agent still running?` printed above an answer the
  backend had just accepted. The token's age is now read only when
  nobody is renewing it: the link is down, or the session file has been
  left behind by a killed agent. A live link with a current file says
  nothing, and the warning for a genuinely dead one is unchanged.
- **The host metrics agent no longer grows by a task per connection
  served.** Each accept loop (`host.sock`, `rpc.sock`, `ctl.sock`) kept
  every finished connection task in its `JoinSet` instead of joining it,
  so a linked node — whose agent asks for `HostMetrics` once a second,
  which the daemon answers by reconnecting to `host.sock` — leaked about
  17 MB an hour: `3pm __host-agent` reached 860 MB of resident memory
  after 45 hours. The loops now join their connections as they go. The
  same growth was reachable from a **hosted, non-root account**: its
  connection cap only ever counted the *live* ones, so a connect/close
  loop on its own socket grew the root daemon's memory without bound.
- **The host metrics agent runs on two worker threads.** It took one per
  core — 37 of them on a 32-core host, each with its own allocator arena
  whose high-water mark never came back — to run four tickers and answer
  one connection a second. Collectors were already on the blocking pool,
  which is sized apart, so nothing samples any slower.
- **The agent runs with `MALLOC_ARENA_MAX=2`** (set on its own app spec
  unless the operator set one): glibc hands every contending thread an
  arena and never returns its pages, which read as a resident set that
  followed the bytes relayed; with two arenas and 64 KiB socket buffers
  the agents stay within a mebibyte across a hundred megabytes.
- **The agent's fleet gate no longer keeps one finished task per settled
  call.** Two of its three spawn sites (a settled call, a `welcome`'s
  catch-up) were never joined while the connection lived, so a node that
  placed nothing while the link kept settling calls grew by one dead task
  each time; all three now go through one reaped set (`fleet_gate/tasks.rs`),
  the same defect and the same shape as the daemon's connection loops.
- **An agent counts `live` sessions apart from plain commands.** A
  session holds its permit until the stream closes, so four cameras on
  one node made the fifth `unavailable` with nothing else running; the
  sessions now have the daemon's own budget of 16
  (`STREAMS_PER_DAEMON_MAX`) and short commands keep their four. Nothing
  on the wire moved.
- **A session refused after its ticket ends at once** in the Node SDK,
  with the code and sentence the backend sent: `io.open` answers on the
  ticket, and a `forbidden` or `unavailable` arriving with the reply
  used to leave the reading leg open until its idle guard fired a minute
  later.
- **A `PUT` of a file is no longer given up on after ten seconds.** It is
  answered when its body ends, so the Node SDK gives it no header budget
  at all (it inherited the ten seconds of a `GET`'s head, which capped a
  relay at about 125 MB over a 30 ms link); sixty seconds of silence on
  the way out ends it instead. Every write, and every read of a blob the
  caller does not know settled, now takes a connection of its own rather
  than one of the eight keep-alive sockets, so eight slow transfers no
  longer make the ninth wait two minutes.
- **A body the relay has parked** — the backend's window full, no reader
  yet — is no longer cut with a `408` on the first poll that follows: the
  agent's idle budget measures the source's silence, not the relay's wait.
- **The agent's relay reuses its connection after a `PUT`**: the `201`'s
  body was left unread, so the warm connection was thrown away and every
  upload paid a fresh TCP and TLS handshake. Reading it is bounded by the
  head budget (8 s): a backend that sends the head and never the body
  costs the reuse, not the caller's request.
- **A backend name that resolves to several addresses** no longer spends
  the whole dial budget on the first one that does not answer: each gets
  a quarter of a second before the next is tried.
- **The agent's live sink rebuilt a message in time quadratic to its
  size** (a one-mebibyte message crossed thirty-two mebibytes of copying)
  and its HTTP buffers were capped at 16 KiB, which cut a hundred
  megabytes into six thousand frames; both fixed, the head block staying
  at 16 KiB and refused `431` past it.
- **The C++ SDK builds inside another project**: `THREEPM_IO_TESTS`
  defaulted to `ON`, so a parent's `add_subdirectory` or `FetchContent`
  compiled the SDK's tests in its own tree (and failed on an archive
  without `tests/`). It now defaults to `ON` only when `packages/cpp` is
  the top-level project; `packages/cpp/tests/consumer` checks it in `just
  sdks` and CI.
- **A root install no longer leaves files owned by uid 501**: the
  release tarballs recorded the Mac user that built them, and
  `install.sh` run as root restored that owner. `just publish` now
  archives as `root:root` and `install.sh` extracts with `tar -o`, so
  the files belong to whoever installs whatever the archive says.
- **Linux binaries load on glibc older than 2.34** (Debian 11, Ubuntu
  20.04, RHEL 8): the handoff guardian called glibc's `close_range()`
  wrapper, a hard symbol dependency; it now issues the raw syscall and
  keeps its one-by-one fallback for kernels before 5.9.
- The handoff's emergency path decides what to stop from the kernel
  rather than from the process table: an app still unreaped is alive and
  still ours, which a reused pid never is. It used to read the process
  start time, a read that can fail under load and then left a live app
  running beside the one the fresh boot resurrected, and left a
  `<defunct>` child behind that nothing would ever collect. The identity
  check remains where it matters, before the escalation to SIGKILL.
- A resource guard's restart (`max_memory_restart`, `max_cpu_percent`)
  counts like a crash. After the graceful stop (`kill_signal`, SIGKILL
  after `kill_timeout`) the app emits `restart`, sits in `waiting
  restart` for `restart_delay` / the exponential backoff, then
  respawns, `↺` +1 — it used to respawn at once, and since the exit
  always came past `min_uptime` an app over its limit from the start
  cycled forever. A trip within `min_uptime` + 10 s of the spawn
  (measured at the trip, not at the exit: the stop does not count) is
  unstable and counts toward `max_restarts`; a later one starts the
  count over. `max_restarts` unstable trips in a row end in `errored`:
  with the defaults, 16 trips for an app over its limit from the start,
  never for one that runs more than about 11 s between trips. A user
  `restart` / `stop` / `delete` during the guard's stop, a watch or
  cron restart start a new series (immediate, backoff reset), `3pm
  reset` clears the counts, and a guard stop or a pending restart
  crosses an in-place `3pm update`. `max_restarts = 0` ends the app
  `errored` on its first trip, stable or not, as on its first crash.
  `restart_policy` (`never`, pm2's `autorestart = false`) and
  `stop_exit_codes` do not stop a guard restart: they judge the exits an
  app makes on its own, and the exit after a trip is its answer to the
  guard's `kill_signal` — a recycle of the daemon's own, like a watch or
  cron restart, and like pm2's `max_memory_restart`.
  `threepm-core/src/policy/guards.rs`.
- A log file that cannot be opened (`out_file`, `error_file`,
  `log_file`: a directory that cannot be made, a permission) fails the
  spawn before the process starts, and `last error` names the file. It
  used to be opened after the spawn: the failure left the app running,
  unsupervised and never killed, and — under `cgroup_enforce` — its
  leaf never removed, one of each per retry until `errored`.
- `3pm ls` failed past about 700 apps: a client read the daemon's reply
  under the 1 MiB cap meant for what a peer sends the daemon, and a
  `list` carries every row (~1.5 KB each). Replies are now read up to
  32 MiB (`MAX_REPLY_BYTES`) by `threepm_protocol::client`, by the
  handoff's reconnection and by the Node client (`RpcConnection`);
  requests, control messages and the event stream keep 1 MiB.
  `CodecError::LineTooLong` names the cap it hit.
- `cgroup_enforce`: the child joins its leaf before it `exec`s — the
  daemon makes the leaf and writes `cpu.max` / `memory.max` first, the
  spawner's pre-exec hook writes the child into `cgroup.procs` — so
  nothing it forks starts outside; the pid used to be attached after
  the spawn. A join that fails fails the spawn (`cgroup_enforce: cannot
  join <leaf>/cgroup.procs: …`, a spawn failure like any other) and the
  leaf is removed. The leaf is `<name>-<id>-<instance>-<lineage>-g<gen>`,
  never shared by two incarnations nor with a daemon started cold after
  a crash (the lineage, `THREEPM_BOOT`, is drawn at every cold start
  and kept by a handoff); a leaf that already exists is refused. When
  the incarnation is over — exit, `delete`, death during a handoff —
  everything left in the leaf is killed (`cgroup.kill`, else SIGKILL
  member by member, nested cgroups included, the leaf frozen through
  `cgroup.freeze` for the pass so that no member's pid can be reused
  between the read and the kill, then thawed) before it is removed,
  members that left the process group included — the cgroups the app
  made below its leaf are removed first, deepest first, never through a
  symlink —, then the empty leaves of the instance's older incarnations
  go too. A leaf whose `cpu.max` / `memory.max` cannot be written is
  removed with the failed spawn. The empty leaves a crashed daemon left
  behind are not cleaned.

### Internal
- Fleet RPC, simplification pass, no behaviour change: the gate tests
  register their control connection and their `fleet_ready` through two
  shared helpers instead of three hand-written `SupMsg` literals, a
  local call refusing a promised blob is proved once rather than twice,
  and `agent_dir` is inlined into its only caller.
- The agent module drops `FleetHttp::get_blob`, which nothing has called
  since `--out` streams its answer (`get_blob_into`), and builds its
  settled `CallResult` fixtures from one helper.

### Added
- **Hosted users: one root daemon, sockets per account.** A root daemon
  now hosts the accounts listed in `[users] allow` of its `config.toml`
  (`dir`, `max_apps`; read at start, so `3pm update` applies a change).
  Each gets its own `rpc`/`pub`/`ctl` sockets (0600, chowned to it,
  peer-uid checked one account per listener) in a root-owned home under
  `/var/lib/3pm/users/<name>/` marked `hosted`: its CLI and the Node
  client find that home by themselves and never start a daemon there. A
  hosted connection is an identity — it sees, lists and targets only the
  rows it owns (anything else is `not_found`, never an existence oracle),
  and the daemon-level commands (`save`, `resurrect`, `kill`,
  `internals`, `hostmetrics`, `orphans`, the `update` handoff) answer the
  new `forbidden` error code. Two axes: files follow the new per-app
  `user` field (the account the app runs as: uid, gid, groups, a clean
  environment built from scratch, `PR_SET_NO_NEW_PRIVS`, no descriptor
  and nothing of root's environment inherited — `3pm start api.js --user
  srt` works for root too), visibility follows `owner`, which the daemon
  stamps from the connection on every entry path (`start`, `apply`, a
  patched `restart`, a rename, `scale`) and a client can never set.
  Logs, rotated generations and pid files of such an app are
  `root:<gid> 0640` in that home (0750, root-owned: the account reads,
  root writes), a hosted app may not choose its log paths, its `watch`
  roots stay inside its `cwd`, one account runs at most `max_apps` rows
  (default 32) over at most 64 live connections per socket, and app
  names stay one global namespace (root frees a squatted one with
  `3pm delete`). An account needs a private primary group; uid 0, a
  shared group and a duplicate uid are refused, each with a log line,
  never fatally. A daemon that is not root ignores `[users]`.
  `FEATURES.md` → *Hosted users*, `docs/protocol.md` → *Hosted users*.
- **`persist = false`** (`--persist false`, or in an ecosystem file): an
  app supervised and restarted like any other, carried across `3pm
  update`, but left out of `dump.json` — so out of `3pm save`, out of the
  implicit save of `3pm update`, and absent after a reboot. For the
  ephemeral backend that must not come back before the app that owns the
  hardware starts it. Toggling it never restarts the app — it is not part
  of the configuration `3pm apply` compares, only of what `save` writes —
  so a backend holding a camera keeps it. A spec saved by an earlier 3pm
  reads back as `true`, the historical behavior.
- **Self-hosted releases: `just publish`** builds the four tarballs on a
  Mac (Linux via `cargo-zigbuild`, glibc ≥ 2.28) and uploads them to
  `https://3pm.edgecr4ft.net` (`vX.Y.Z/` + `checksums.txt` +
  `version.json`, then `latest/VERSION`, `latest/version.json` — version,
  date, install command, every asset's url/size/sha256 —, `install.sh` and the landing page
  `distribution/site/index.html`). Each run ships a new minor version
  (bumped in `Cargo.toml` when the current one is already online, and
  stamped on the bundled modules so `3pm update` refreshes them);
  `--force` republishes the current version, `--dry-run` packages only. `install.sh` now
  defaults to that site, takes the version from `latest/VERSION`, and is
  also the updater: rerun, it hands a daemon started from the same
  install off to the new binary (`3pm update`, apps keep running).
- **SDKs on the download site: `just publish-sdk`** puts the three
  application-metrics SDKs on `https://3pm.edgecr4ft.net`: `@3pm/api` as
  an archive (`npm install
  https://3pm.edgecr4ft.net/sdk/node/3pm-api-0.1.1.tgz`), `threepm-io`
  in a pip index (`pip install --index-url
  https://3pm.edgecr4ft.net/simple/ threepm-io`), the C++ headers as an
  archive for CMake's `FetchContent`, and `sdk/sdk.json` with every
  version, file and sha256 (the landing page lists the SDKs from it).
  Each SDK ships when its own version is not online yet, after its tests
  and an install of the fresh archive into an empty project; older
  versions stay installable, and the archives are reproducible (a
  `--force` republish keeps their sha256).
- **Custom actions: `3pm trigger <app> <action> [params]`** — an app
  exposes functions on its control connection (`io.action(name, [opts],
  handler)` in `@3pm/api`); the daemon triggers one by a correlated call
  (`action` → `action_reply` on `ctl.sock`, `trigger` / `triggered` on
  `rpc.sock`), on every instance of a name (16 at most) or on one
  (`--instance`, `--src`), with the result inline (JSON ≤ 512 KiB) or as
  a file under `~/.3pm/actions/<app>[-<n>]/` (≤ 64 MiB, 20 per instance
  and 200 MB kept; a file is never pruned while its call gathers and is
  spared by the count for 10 minutes, what a timed-out call leaves
  behind is collected, a symlink or the `tmp` extension is refused).
  Bounded: the caller's `--timeout` under the declaration's (30 s
  default, 10 min ceiling), one call per (source, action), four per
  source, 32 per daemon, and never the last slot of an app's control
  queue, kept for `shutdown` (`busy`); typed per-instance errors
  (`failed`, `timeout`, `busy`, `gone`, `refused`, `too_large`);
  `remote: false` keeps an action off the fleet backend. `3pm trigger
  <app>` and `3pm show` list what each source declares (`--src` keeps
  one; an unknown source exits 2 and names the known ones); every call
  is a `proc_action` event on the bus (`3pm events -f proc_action`),
  without its params or result — a trigger refused as a whole leaves one
  per instance it named. Timeouts show to the millisecond (`1.5s`).
- **`3pm profile <app> [--heap] [--duration 10s] [--src S]`** — the SDK's
  built-in `3pm:profile` action (declared by `io.init()` on the `main`
  source — `profile: true` adds it on any other, `profile: false`
  withholds it — and counted toward the 32 actions per source;
  `--duration` from 1s to 60s, a usage error outside):
  a CPU or heap sampling profile through
  `node:inspector`, left as a `.cpuprofile` / `.heapprofile` file (opens
  in Chrome DevTools and speedscope) and summarised — the hot frames by
  self time or bytes, and for a CPU profile how long the process was
  busy, its idle time apart (`idle` in the JSON)
  (`threepm_core::profile_summary`, shared with the fleet backend). Its
  params are checked, never coerced (`kind` `cpu` or
  `heap`, `duration_ms` 1 000 to 60 000), and a call whose timeout cannot
  hold the duration plus 15 s to write the file is refused before
  sampling. In an Electron app the main process's `io.ipcBridge`
  profiles each renderer over `webContents.debugger`: a debugger already
  attached (DevTools) is an immediate `debugger already attached`, each
  CDP command gets at most 10 s, cut shorter so setup, sampling and 5 s
  to write the file fit the call's timeout, the relay detaches only what
  it attached, and a source shared by several windows is refused (give
  each window its own). A renderer's own `io.action` calls are relayed
  over IPC (`3pm:actions`, `3pm:actions_ack`, `3pm:action`,
  `3pm:action_reply`): each window keeps its own actions, dropped when it
  commits a navigation (a reload too), crashes or closes — a navigation
  that never commits keeps them, and the new page is asked to declare
  its own. A declaration is acknowledged and resent until it is, so one
  made before `io.ipcBridge()` still arrives; a renderer's `shutdown()`
  withdraws it; renderers cannot take `main`, `pid-*` or `worker-*`; and
  only the window a call went to can answer it.
- **`@3pm/api` names every connection**: `io.init()` without a `source`
  says hello as `main` in the process 3pm spawned (its series carry
  `src="main"` from now on), `worker-<pid>-<threadId>` in a worker
  thread (thread ids restart in every process) and `pid-<pid>` in a
  forked child or cluster worker (told apart through `THREEPM_IO_MAIN`),
  so they never evict each other. Two processes that both claim `main`
  (`sh -c 'node a & node b'`) settle it: the refused one falls back to
  `pid-<pid>`, and that collision never counts toward the SDK giving up
  for good. A child started by a non-Node parent otherwise cannot be told
  apart — name it. An invalid source
  falls back to that default. Action replies and the declaration have
  their own budget on the socket (1 MiB a line), so metrics cannot crowd
  them out: a reply that cannot be written fails the call instead of
  timing it out, and the declaration goes right after `hello`, resent
  until written. `io.action('3pm:…')` throws (the prefix is the SDK's),
  `ctx.commitFile` past the call's deadline deletes the `.tmp` and
  throws, and a lone surrogate in a result becomes U+FFFD. Whatever a
  handler throws answers `ok: false` (no unhandled rejection, no stuck
  "already running"), and a result over 512 KiB or nested deeper than 64
  levels fails `too_large` instead of timing out.
- **`3pm link` runs the fleet backend's commands** — the agent answered
  `unavailable` to every command until now. Each one runs in its own
  task: checked against the capabilities the backend accepted, narrowed
  to those the node offered and its certificate grants (`forbidden`),
  four at a time (`ping` apart; `unavailable` beyond), bounded by the
  call's deadline — counted from receipt, less a second for the transit,
  so the agent stops before the backend gives up (nothing is sent past
  it) — and tied to its connection
  (a call cut by a drop is abandoned, never answered on the next link).
  `restart`, `stop` and the new `action` go to the daemon; an action is
  always triggered `remote: true`, so one declared `remote: false`
  answers `refused`, and only `remote` actions are published with the
  sources. A remote `restart` or `stop` never reaches a module app (the
  agent itself included): it is refused `forbidden`. New capability
  `proc:action`, part of the default set; a `link.toml` written before
  this release (`version = 1`) holding exactly the earlier three is read
  as all four, one written now (`version = 2`) means exactly what it
  lists. A result travels gzip + base64: inline while the whole reply
  stays under 256 KiB compressed, else as `blob_chunk` frames (512 KiB
  each, four in flight) sent ahead of the reply; a result file must be a
  regular file under `~/.3pm/actions/` (a symlink is refused), read,
  gzipped and hashed in one pass, at most 32 MiB − 64 KiB over the link
  (refused before reading; 64 MiB stays the local cap for `3pm
  trigger`), and every gzip byte of a reply stays within 64 MiB so the
  backend never drops a blob. An outcome's data and file are weighed
  before anything is sent, and an error never names local paths.
- **More Node built-in metrics in `@3pm/api`** (`builtins: true`):
  `nodejs.eventloop.utilization`, `nodejs.eventloop.delay.p99` / `.max`
  (native `monitorEventLoopDelay`), `nodejs.gc.duration{kind}` (every GC
  pause), `nodejs.heap.committed` / `.limit`, `nodejs.memory.external` /
  `.array_buffers`, counters `process.page_faults.major` and
  `process.context_switches.involuntary`; `builtins: { heap_spaces: true }`
  adds `nodejs.heap.space.used{space}`. The four original metrics are
  unchanged.
- **`io.electron(require('electron'))`**, Electron main process: Chromium
  processes by type (`electron.processes`, `electron.process.cpu`,
  `electron.process.memory`), power state (`electron.power.on_battery`,
  `.thermal_state`, `.speed_limit`), counters `electron.process.gone{type,reason}`
  and `electron.unresponsive`, and events for crashed processes, hung
  windows, sleep/wake and power changes.
- **Renderer built-ins**: `createRendererIo(ipcRenderer, { builtins: true })`
  reports `renderer.eventloop.delay`, `renderer.longtask.duration`,
  `renderer.heap.*` (Electron's `process.getHeapStatistics()`, else
  `performance.memory`), `renderer.blink.*`, `renderer.cpu.utilization`
  and `renderer.dom.nodes`, still without any Node builtin module.
- **`3pm start "npm run dev"`**: a quoted command line (a target with a
  blank that is not a file) runs under `bash -c`, else `sh -c` (pm2), on
  the CLI, in ecosystem files and in `@3pm/api`, named after its words
  (`npm-run-dev`, `python-src-app.py-port-3000`; `--name` overrides).
  Separate args, an interpreter or its flags beside it are errors (pm2
  drops them).
- **`--interpreter none`** runs the script itself (pm2); on `restart` it
  clears the app's interpreter.
- **`--interpreter-args` / `--node-args`**, `interpreter_args` in files
  (pm2's `node_args`, `interpreterArgs`; string or list): the
  interpreter's own flags, before the script. `3pm show` row
  `interpreter args`; additive `StartSpec.interpreter_args` on the wire,
  never sent over the fleet link.
- **pm2 spellings** `exec` / `cmd` / `command` (`script`) and
  `exec_interpreter` (`interpreter`) in files and `@3pm/api`; two
  spellings of one field are an error.
- **YAML and JavaScript configs**: `.yaml`/`.yml` and pm2's
  `*.config.js|cjs|mjs` (evaluated by the CLI's `node`, in the shell's cwd
  and environment, 10 s / 16 MiB at most), everywhere a config file is
  taken; JSON, YAML and JS also take pm2's bare list of apps and single
  app shapes.
- **`3pm start -`** / **`3pm validate -`**: the config piped on stdin
  (JSON or TOML); `apply -` is refused.
- **`3pm start`**, **`3pm validate`**, **`3pm apply`** without a target use
  the current directory's `ecosystem.{toml,json,yaml,yml,config.js,config.cjs,config.mjs}`,
  first found wins.
- **`3pm start npm -- run dev`**: a start target that is a bare command
  name and not a file in the cwd is looked up in your PATH (pm2), on the
  CLI, in ecosystem files (`script = "npm"`) and in `@3pm/api`
  (`{script: 'npm'}`). The cwd still wins; a target with a `/` never
  searches PATH. The app runs the PATH entry itself (`…/bin/npm`, which
  survives an upgrade) under the interpreter inferred from the file it
  links to (`npm-cli.js` → node, resolved against your PATH), and is
  named after it (`npm`). A miss says `looked at <cwd>/npm, then PATH`.
- **`3pm ls --tree`** (`-t`): each app's process tree under its row,
  compacted — a chain of single children is one line (`npm › bash ›
  node › Electron`), twins are grouped (`Electron Helper ×4`), `⚠` marks
  a process that left the app's process group and that `3pm stop` will
  not reach — `--all` unfolds it, one row per process; with `--watch`
  the period defaults to 1 s (the daemon's tick). `3pm ls` itself
  writes `saw +13` when an app has other processes. `3pm show` says
  `cpu 62.2% (14 processes)`, `mem 3.0gb (14 processes, RSS sum)` and
  lists every process under ` Processes` (`⚠ detached` on the
  escapees). `--json`: `proc_count` and `tree_rev` on every record;
  with `--tree`, and in `show`, a `tree` object per app (absent when
  the app runs alone or is stopped). An older daemon renders both
  without trees and says `run \`3pm update\``. `packages/node`:
  `proc_count?` / `tree_rev?` on `ProcessRecord`, `limit_scope` among
  the `start` options. Wire, all additive,
  no protocol bump: `process_trees {ids, gen?}` → `{trees: AppTree[]}`
  (an unknown id, or one at another `gen`, left out; 256 lines per
  tree, 2 000 per reply shared between the trees that list members,
  `omitted` for the rest, the totals of the same tick as the lines;
  names cleaned of controls and bidi marks, private-use glyphs kept),
  `ProcessRecord.proc_count` / `tree_rev` (moves when the listed
  membership changes — a member enters or leaves the first 256 lines,
  detaches, takes another name (read again every tick: `comm` on
  Linux, which follows an `exec` and `prctl(PR_SET_NAME)` — Node's
  `process.title` —, the executable's file name on macOS, which follows
  an `exec` only), the
  incarnation exits — to the next value of one counter per daemon,
  which an in-place `3pm update` carries over and a daemon started
  cold starts again from 0), `DaemonInternals.metrics_tick_ms`;
  `limit_scope` on the
  spec (`tree` | `leader`), through the `SpecPatch` chain.
- **Process trees on the fleet link.** The agent (`3pm link`) sends the
  backend each app's process tree — the same lines `3pm ls --tree`
  draws — on the model of the application metrics, never inside the
  1 Hz frame: the shape whenever an instance's `(gen, tree_rev)` is
  newer than the one last written — and every instance once at the
  start of a session and when the daemon image changes, an app alone
  as an empty tree —, the figures every `welcome.trees_every_s` (1 s
  by default) for every app with more than one process, packed under
  the message cap. The telemetry's record gains `proc_count`, the fleet
  table's `+N`. Wire (`threepm-link`, additive): `trees {ts, trees:
  [RedactedTree]}`, `trees_request {ids}` (backend → agent),
  `welcome.trees_every_s` (absent = a backend predating trees: nothing
  is sent), `RedactedRecord.proc_count`; `budget::fit_app_metrics` and
  `fit_trees` share one packer; member names are cleaned again by the
  link and by the backend. A daemon predating trees
  (`unknown_request`) silences the task for a minute, then it probes
  again. `docs/backend-agent/link-protocol.md` §6c, `FEATURES.md` →
  *Process trees on the link*.
- **`3pm restart` takes every `start` flag** (pm2's attribute update):
  what you type is folded into the app's stored configuration, which
  then restarts on the result — `3pm restart api --name web -i 4
  --max-memory-restart 300M`. `--name` renames the app **in place**
  (same ids, same instance numbers, restart counters kept, pid files
  moved along); `-i N` reconciles the instance count like `start` does;
  `--env K=V` merges into the environment instead of replacing it. The
  fold is per app — a spec belongs to a name that owns N instances — so
  a patch through one instance covers the whole app. Renaming is
  refused when the target matched several apps, when the new name is
  taken, or when a module owns the app (it is named by its manifest);
  an invalid patch restarts nothing. `--watch` alone still
  toggles watch mode (pm2); `--watch=PATH` sets the paths.
  `--update-env` keeps re-reading the ecosystem file and cannot be
  combined with flags. Wire: `restart` carries an optional
  `patch: SpecPatch`, additive, no protocol bump.
- **3pm's own metrics in `3pm hostmetrics`** (`3pmDaemon` in `--json`):
  two tables after the sensors — `3pm processes` (the daemon and the
  host agent: pid, uptime, cpu, RSS, open descriptors, threads) and
  `3pm runtime` (apps by status, restarts, timers, bus subscribers, log
  sinks/rotations/bytes, `RLIMIT_NOFILE`, handoffs, agent respawns,
  `boot_id`). Measured by the host agent on the medium period, from the
  outside: the process rows from the OS, the counters pulled over
  `rpc.sock` (`Request::Internals`, one connection per tick, 2s bound —
  a daemon mid-update blanks the counters for one tick, never stalls
  the tier). Optional on the wire, never gates the first snapshot; `3pm
  internals` is unchanged. No protocol bump.
- `pong` and `3pm internals` carry a `boot_id`: a random id of the
  daemon *image*, redrawn at every boot and at every in-place handoff
  (the coming `3pm update` without app restarts), so a client can tell
  the daemon it was talking to from its successor at the same pid.
  `internals` also counts `handoffs`. Additive (`0` from an older
  daemon); no protocol bump.
- Two additive error codes for the coming handoff: `handoff_in_progress`
  (mutations refused for a moment, reads pass) and `handoff_refused`.
- **`3pm update` no longer restarts the apps.** The daemon hands itself
  off to the new binary in place: same pid, same children, same sockets
  — the apps keep running, their log lines are all kept (down to the
  byte read but not yet written at the switch), an exit during the
  switch keeps its real code, timers resume where they were. The new
  binary is checked first (`__daemon --handoff-check`): a binary that
  cannot take over is refused and `update` falls back to the previous
  cold restart (`--no-fallback` exits 3 with the apps untouched; `--cold`
  asks for the old behaviour). A failed exec resumes the old image with
  nothing lost; a new image that cannot adopt stops the apps cleanly and
  restarts them from the saved list; a new image that dies is caught by
  a guardian process that restarts the daemon on the same sockets. The
  report says how it went (`kept running` / `restarted`; `--json`:
  `mode` `handoff`/`cold`/`emergency`/`recovered`, `kept`, `restarted`,
  `duration_ms`, `fallback_reason`); `3pm internals` shows the boot id,
  the handoff count and the last outcome. Commands issued during the
  switch wait it out (a message says so); `3pm logs`/`events`/`runtime`
  followers stay attached. On the wire, all additive: a `handoff`
  request, a `daemon_handoff` event and `internals.last_handoff`.
- `3pm update` refreshes the modules after the daemon, in place: an
  installed module whose bundled version changed is reinstalled and its
  rows restarted (same rows, sites included; manifest apps added or
  removed by the release are started or deleted), and an app running
  from the bundle without an install — the agent after `3pm link` — is
  restarted on the new binary and re-stamped. Rows already at the
  bundle's version are left alone (a second `update` touches nothing);
  the comparison is per row, not per module, so a module whose files
  were copied but whose rows were not restarted (a refresh that failed
  midway) is picked up by the next `update`; `--skip-modules` skips the
  step. One line per module in the report;
  `--json` adds `modules: [{name, from, to, source, restarted, started,
  deleted}]`.
- `3pm install <name>` on a module already installed now restarts its
  rows in place instead of deleting and recreating them: the sites
  served with `3pm serve` (and any app started from the module's files)
  are kept, with their row and their history.
- `3pm ls --watch`: pm2's live table, redrawn in place where the cursor
  is (the history above stays, no blink: the cursor climbs over the
  previous frame and repaints it line by line) every 300ms — `--interval
  <MS>`, 100ms floor — under a `Last refresh:` stamp; the same body as
  `ls` (link line, orphan notices, host line). The terminal is re-read
  at every tick, so the table follows a resize; a frame taller than the
  screen repaints from the top of the visible screen instead. Off a tty
  the frames simply follow each other (a blank line between them) and a
  reader that leaves ends the command with exit 0; `--json --watch`
  prints one JSON array per tick (NDJSON).
- Application-metrics SDKs, one surface in three languages
  (`docs/app-metrics.md`): `io` in `@3pm/api` for Node and Electron
  (`packages/node`: `gauge`/`counter`/`meter`/`histogram`/`metric`/
  `event`/`notifyError`, opt-in built-ins — event-loop lag histogram,
  heap, handles, CPU —, `io.ipcBridge(ipcMain)` + `@3pm/api/renderer`
  for renderers), `threepm-io` for Python (`packages/python`, standard
  library only, daemon flush thread) and `threepm_io.hpp` for C++17
  (`packages/cpp`, header-only, wait-free `inc`/`set`/`observe` for
  real-time threads). Same behavior everywhere: disabled outside 3pm,
  never blocking, never throwing, whole registry every second,
  reconnection 1 s → 30 s, give-up after five refused `hello`s, events
  queued while connecting, labels bounded locally (3 keys, 50 sets then
  `overflow="true"`), histograms capped at 128 non-empty buckets by
  lowering their scale (OpenTelemetry style; C++ by a fixed 128-slot
  window), every cap counted in UTF-8 bytes like the daemon's, lines split
  under 32 KiB, histogram buckets identical bit for bit to the daemon's
  grid — `packages/exphist-vectors.json`, generated by the Rust tests and
  replayed by every suite. `just sdks` runs the three suites;
  `crates/threepm-cli/tests/e2e_io.rs` runs the three under a real
  daemon.
- The daemon accepts the label set `overflow="true"` from an SDK that
  folds its own excess label sets (the one reserved label an app may
  send); it merges with the daemon's own fold, under the same name cap
  and creation budget as any series.

- Application metrics and events on the fleet link, every second and
  differentially: the agent asks the daemon what changed since the
  last patch it wrote on the socket (`app_metrics {id, since_rev,
  gen}`) and relays it as `app_metrics {ts, procs: [AppMetricsPatch]}`
  (short keys, the daemon's `sid`s untouched; everything at the start
  of a session, on the backend's `resync {ids}`, and every 300 s per
  instance); `emit {app}` carries `io.event` / `notifyError` as the
  daemon admitted them. `welcome.metrics_every_s` sets the period
  (1 s floor). `threepm-link` gains the patch types, the pure reader
  `patch::apply` (a `Gap` on a patch off its base or incarnation) and
  `budget::fit_app_metrics`; `RedactedRecord` gains `app_series`,
  `app_metrics_rev` and `gen`. Fixtures `link/v1` extended (additive).
- `ProcessRecord.gen`: the incarnation of the instance (bumped at every
  spawn), so a reader tells a restart from a revision that happens to
  land on the same number.
- `app_metrics` request: `gen` names the incarnation `since_rev`
  belongs to — another one gets a full answer. `hist[].reset` in the
  answer says the bucket table was cleared after `since_rev` (scale
  change, count going backwards), so a differential reader replaces
  its buckets instead of merging.

### Fixed
- The `overflow="true"` fold could sum two huge counters (or histogram
  sums) to `inf`, which serializes as `null` and made the instance's
  `app_metrics` answer undecodable (`3pm show` failed, the agent dropped
  its daemon connection every second). A non-finite fold is now ignored.

### Changed
- A metrics batch that defines or changes nothing is no longer a new
  revision: an SDK flushing an idle registry every second leaves
  `app_metrics_rev` alone, so a differential reader has nothing to ask.
- Control channel: after `hello`, a line that is not JSON is skipped
  (debug-logged) instead of ending the connection — one bad line from an
  SDK no longer costs a source its series, nor makes the SDK give up
  after five reconnections. An over-long line or an I/O error still
  ends it.
- A histogram entry may carry up to 128 non-empty buckets (was 64), so
  a scale-1 histogram spans 64 doublings before an SDK has to lower its
  scale.
- Application metrics and events (pm2's custom metrics, for any language
  and any number of processes per app): a process of the app connects to
  `$THREEPM_CTL_SOCK`, says `hello` with a `source`, then sends
  `metrics` batches (gauges, cumulative counters, base-2 exponential
  histograms; up to three labels, a unit, a description) and `event`
  lines (OpenTelemetry log-record shape with an optional exception and a
  fingerprint). `3pm show` gains a "Custom metrics" section (approximate
  quantiles for histograms) and `show --json` an `app_metrics` object;
  `3pm events -f app` follows the events, with `×N` for the repeats the
  per-fingerprint rate limiter suppressed. New `app_metrics` request —
  full or differential by revision (`since_rev`), so a reader carries
  only what changed — and `app` bus event; `ProcessRecord` carries only
  counts (`app_series`, `app_metrics_rev`, `app_metrics_rejected`,
  `app_events_dropped`). One control connection per process of the app
  (a source that reconnects replaces itself; `shutdown` is written to
  all of them), peer pid recorded, every line validated on the
  connection task. Bounds: 1 000 live series per instance, 200 names,
  50 label sets then `overflow="true"`, creation budgets against churn,
  120 s TTL, 10 events/s. `docs/metrics.md`, "Application metrics".
- `3pm startup --level user` / `3pm unstartup --level user`: the boot
  service in your own session (a LaunchAgent, `systemctl --user`),
  without root — what a daemon that must share the GUI session (macOS
  AudioUnits) needs.

### Changed
- Crash restarts are immediate by default, as with pm2: a crashing app
  respawns at once and a crash loop reaches `errored` (red in `3pm ls`)
  after 16 quick exits in a moment, instead of sitting in
  `waiting restart` for minutes while the former always-on exponential
  backoff climbed to 15 s. Backoff is now opt-in with pm2's name:
  `exp_backoff_restart_delay = 100` in the ecosystem file or
  `3pm start --exp-backoff-restart-delay 100` (the 0.1 spelling
  `backoff_initial` / `--backoff-initial` is still accepted);
  `restart_delay` / `--restart-delay` is a fixed pause before every crash
  restart. `backoff_factor`, `backoff_max` and `backoff_reset_after`
  only apply once backoff is on. Wire shape unchanged
  (`backoff_initial_ms`, now `0` by default).

### Internal
- The two bundled modules shared nothing but a copy-paste: the child
  side of the control channel (`hello`/`ready`, the `shutdown` watch),
  the `--json`/`THREEPM_JSON` resolution, the `__run` role split, the
  `[3PM][ERROR]` exit mapping and raw `println!` calls. They now build on
  `threepm_protocol::module` (`is_run_role`, `Out`, `exit_code`,
  `ctl::{Identity, announce_ready}`) and `threepm_protocol::out` (the
  CLI's pipe-safe `oprintln!` family, moved there with the `[3PM]`
  chrome line and the color decision). Visible consequences: `3pm serve
  … | head` no longer panics on the closed pipe, a module's `[3PM]`
  prefix is colored like the CLI's, and under `--json` its notices go
  to stderr instead of being dropped. `clippy::print_stdout`/
  `print_stderr` are denied in the modules as in the CLI; the daemon
  and the modules name `THREEPM_CTL_SOCK`/`THREEPM_ID`/`THREEPM_GEN`
  through one set of constants; `ControlOut::decode_lenient` mirrors
  `ControlIn`'s.

### Added
- `3pm ls` warns about the processes a daemon that died uncleanly
  (`kill -9`) left running: they were invisible — a fresh daemon manages
  nothing — while still holding their ports. Below the table: `1 process
  from a previous daemon still running unmanaged: api (pid 4321)` and the
  way out (`3pm resurrect`, or `3pm start <name>`). Read-only: nothing is
  killed by `ls`. Backed by a new `orphans` request (a scan of `pids/`
  off the actor thread, re-filtered against the managed rows at reply
  time); `ls --json` is unchanged and an older daemon prints nothing.
- The fleet link above the process table, pm2's `⇆ PM2+ activated | …`
  line: a host that is linked opens `3pm ls` with `⇆ 3pm link | node
  3pmn_… | dash: <bucket page>`, green when the link is established, red
  with the reason when it is not (`link down for 5s (transport error),
  retrying`, `parked: the node was revoked`, `app \`agent\` is
  stopped`…). A 3pm with no backend prints nothing, and `ls --json` is
  unchanged. The line costs no request: the CLI reads the agent's
  `link.toml` and `session.json` itself and finds the app's row in the
  list the table just printed. On a narrow terminal the address is
  dropped first, the reason last.
- The backend's page for a cluster is now printed wherever the link is:
  `3pm agent status` (and `3pm link`) gained a `dashboard` line, and
  `3pm link <cluster-key> <token> <server>` ends on `[3PM] Dashboard
  <url>` (`"dashboard"` in `--json`). It is derived from the recorded
  server — `ws://` → `http://`, `wss://` → `https://`, path
  `/b/<cluster-id>`.
- Target patterns: a target holding `*` or `?` selects processes by glob
  instead of naming one — `3pm logs 'victoria*'`, `3pm restart
  'worker-?'`, `3pm delete 'bench*'`. No flag and no mode; `*` is any run
  of characters, `?` exactly one, the match is whole-string and
  case-sensitive (`*victoria*` is the substring form), and it applies to
  the app name and the namespace, the same two fields a literal name
  matches. Available on `logs`, `stop`, `restart` (with `--update-env`
  too), `delete`, `reset`, `show`, `sendSignal`, `flush`, `rotate` and
  `events --process`; `scale` refuses one, an instance count belongs to a
  single app. A pattern that selects nothing is a notice when other
  targets matched and the run's error otherwise — for a mutation, raised
  before the first request, so nothing is left half-done. Patterns are
  resolved in the CLI against the daemon's list and only the selected
  names reach the wire (`events --process` excepted: a server-side
  filter, so the glob travels). Module rows are shielded from a pattern
  on a mutation exactly as they are from `all`, never from a read.
- `3pm link <cluster-key> <token> <server> [flags]` and `3pm unlink
  [flags]`: built-in sugar over the bundled `agent` module, so the line
  pasted from a backend's bucket page runs as printed. Every argument is
  handed to the agent untouched (`3pm link --help` is the agent's help,
  with its flags), the module owns the exit code, `--json` on either side
  of the verb reaches it through `THREEPM_JSON`. Without an `agent`
  module the command fails with a reinstall hint before any daemon is
  spawned; `link` / `unlink` are reserved from module names. `3pm link
  stop` pauses the link (the agent app is stopped and the list saved,
  the enrollment kept) and `3pm link start` resumes it on the existing
  enrollment, registering the app again if it was deleted; `3pm link
  status` is `3pm agent status`, `3pm link delete` (`rm`, `remove`) is
  `3pm unlink`. `stop` and `start` refuse a host that is not linked. `3pm link` alone is a check: the status plus a `link` line
  (`connected for …` or `NOT established — <why>`), exit 0 only when the
  link is established, 1 with the reason on stderr otherwise; `--json`
  adds `session {connected, for_ms, sessions, retries, parked,
  last_disconnect?, server_version?}`. The running agent keeps
  `<state dir>/session.json` up to date for it (pid-checked, removed on
  a clean exit).
- `3pm agent link|unlink|status` (`modules/agent`, the second bundled
  module, own workspace and lockfile): the fleet link to a `3pm-backend`.
  This step ships the skeleton — the CLI dispatched by 3pm, inputs
  resolved from the pasted line with `THREEPM_LINK_*` overriding it
  (clear text to a non-loopback host refused with `ERR_INSECURE_SERVER`
  unless `--insecure`; the token is never echoed), `THREEPM_AGENT_*`
  options for the supervised process, the state directory
  (`<home>/agent`: owner-only `node.key` written temp+fsync+rename,
  `node.crt`, `ca.pub`, `link.toml`), `status` (link + certificate expiry
  + the app's row) and `unlink` (delete, save, forget). `link` enrolls
  for real: Ed25519 node key, the challenge/proof dialogue over a
  bounded WebSocket (`ws://` clear text or `wss://` against the Mozilla
  roots; pinning comes later), the issued certificate verified against
  the cluster CA before anything is written, then the supervised `agent`
  app is started (`wait_ready`) and saved. Refusals exit 1 with nothing
  written; `--force` re-enrolls with a fresh key. The supervised app
  holds the link: handshake (challenge → proof + hello → welcome, 10 s,
  version range and clock skew checked), one writer (control frames
  never dropped, telemetry last-wins with a `dropped` count, ping 20 s,
  flush 5 s, full write buffer ⇒ dead) and one reader (calls acked at
  once, 60 s of silence ⇒ dead), reconnection on the plan's schedule
  (full jitter, class per close code / HTTP status, revoked ⇒ parked
  with an hourly probe), polite close on `stop` / signals, re-issued
  certificates verified and stored. Telemetry at `THREEPM_AGENT_INTERVAL_MS`
  (1 s): the redacted process table (`RedactedRecord`, never a
  `ProcessRecord`: env key names at most, no values, paths or log lines)
  and the host snapshot — identity stripped from the tick, verbatim
  `host_full` on the first frame, on an identity change and every
  `THREEPM_AGENT_FULL_EVERY_S` (the backend may ask for another period)
  — plus the daemon's version/protocol/pid; `process` and
  `daemon_shutdown` events forwarded as `emit`, the latter followed by a
  polite close. Commands, renewal and TLS pinning follow.
- `3pm-backend` ingests the link: `GET /api/v1/link` admits a node by
  its certificate (signature under the bucket CA, key and state read
  from the database on every handshake; expired-but-signed re-issued in
  `welcome.cert`), runs the challenge/proof/hello dialogue, then feeds
  the hub — presence `online` → `stale` (3 s) → `offline` (30 s or the
  socket gone, both tunable), the last snapshot per node, a 300-point
  `{t_s, cpu, rss_kb}` ring per process, a second session of the same
  node closing the first 4409. `GET /api/buckets/{id}/events` streams
  `init` / `node` / `telemetry` / `emit` with monotonic ids,
  `Last-Event-ID` replay and `reset`; `GET /api/buckets/{id}` carries
  the same `init` shape. `NodeView.state` is now the displayed state
  (`pending` / `revoked` / presence) with the database state in
  `enrollment`.
- `3pm-backend` serves its first dashboard: `/` lists and creates
  buckets, `/b/<id>` mints a token into the line to paste, lists the
  nodes with their state and, per node, the process table (status,
  pid, cpu, memory, restarts, uptime) refreshed by the SSE feed at
  1 Hz with the raw `telemetry` JSON one click away. Vanilla JS, no
  build step, assets embedded in release builds. `tests/real_agent.rs`
  runs the vertical for real — backend, a real `3pm`, `3pm agent link`,
  the feed — under `just e2e-modules`.
- `3pm-backend` node API: `GET /api/buckets/{id}/nodes`, `POST
  …/nodes/{node}/approve` and `…/revoke` (revoking closes the live
  session 4403 `ERR_NODE_REVOKED`; approving again re-admits the host
  with the key on record). `NodeView` gains `cert_expired` and
  `last_auth_error {code, at}` — the last time the link turned the node
  away while pending or revoked, announced on the feed — and the bucket
  page gets Approve / Revoke buttons per node.
- `stop`, `restart`, `delete`, `reset` and `logs` accept **several
  targets** (pm2 parity: `3pm delete 0 1 api`), mixing names, ids, `all`
  and ecosystem files. `all` absorbs the rest of the list and duplicates
  fold, so no row is acted on twice. An unresolvable target no longer
  costs the others: the batch acts on everything that resolves, notices
  each failure and exits with the shared wire code (still 2 for an
  all-NotFound run); `--json` keeps its one-document contract by emitting
  the error doc alone. `logs` treats an unmatched name as a notice and
  streams the rest — only an empty selection is an error.
- Modules (`pm2 install` parity, bundled with the install instead of
  downloaded): a module is a directory with `module.toml` (`[module]`
  name/version/description/`command`, optional `[[apps]]` in ecosystem
  syntax). `3pm install <name|dir>` copies it into `~/.3pm/modules/`,
  starts its apps (named `<module>:<app>` pm2-style, namespace = module,
  tagged with `spec.module`) and saves the process list; `3pm uninstall`
  deletes them (plus any app running from the module's files) and the
  directory; `3pm module list` / bare `3pm install` show installed and
  bundled versions. `3pm <module> …` runs a module's command (installed
  copy first, else the bundled one — no install needed for command-only
  modules); unknown subcommands get a did-you-mean over built-ins and
  modules. Module apps sit in a second **Modules** table in `3pm ls`,
  are skipped by `stop|restart|delete all`, `delete --flush all`,
  `apply --prune-all` and `restart --update-env`, and vanish from
  `resurrect` once their directory is gone. Bundle lookup:
  `THREEPM_MODULES_DIR`, `<3pm dir>/modules`, `<3pm dir>/../lib/3pm/modules`
  (tarball, install.sh, deb/rpm all ship `modules/`).
- `3pm serve [dir] [port]` (pm2 `serve`), the first bundled module
  (`modules/serve`, its own workspace and lockfile): `--spa`, `--ftp`
  directory listing (trailing-slash redirect, dirs first, dotfiles
  hidden), `--basic-auth-username/--basic-auth-password`, `--host`,
  `--homepage`, custom `404.html`, CORS, HEAD, 405 for other methods,
  path-traversal 403, malformed escapes 400 (GHSA-9prm-75wm-xpg6),
  symlinks resolving outside the served directory 403 (and hidden from
  `--ftp` listings — a departure from pm2, which follows them; enforced
  by the open itself, resolved from a descriptor of the served
  directory, so a link swapped after the check is still refused),
  streamed bodies; the app is `wait_ready` (online once bound), named
  `static-page-server-<port>`, tagged with the module (so it lists under
  **Modules** below the apps started by hand and `all` targets skip it),
  restarted in place when served again on the same name or port, an
  explicit busy port is reported up front while the default moves to
  the next free port, and `PM2_SERVE_*` env is honored for ecosystem
  files written for pm2. The command prints the `Local`/`Network` links
  (LAN address for a phone) and a QR code of the network link
  (`--no-qr`); it no longer prints the process table. Given a single
  `.html` file, it serves the file's directory with that page as the
  homepage, opens it in the browser when run from a terminal (links
  only, no QR code) and stays attached: Ctrl-C stops the server (`--detach` / `--attach` override
  the per-mode default). `3pm <module>` forwards Ctrl-C to the module
  and waits for its exit status.
- `3pm internals` ends with an `internal processes` table: every process
  3pm runs for itself (the daemon, the host metrics agent) with pid,
  status (`online` / `waiting restart`), uptime, cpu, RSS and respawns.
  `--json` carries it as `internal_processes: InternalProcess[]`
  (additive; the agent's cpu/mem are sampled on the daemon's metrics
  tick and attributed only to the incarnation they were taken for).
- `--attach` on `start`, `restart` and `stop` streams the affected apps'
  logs right after the action (pm2 parity: table, then live stream with
  no history, Ctrl-C to leave). Conflicts with `--json`.
- `3pm bench` starts a demo fleet for testing — 4 node apps (HTTP,
  CPU burner, memory ballooner, a 15 s crasher), 2 python apps (HTTP,
  JSON worker) and 2 bash loops — written to `<home>/bench/` (or
  `--dir`), all in the `bench` namespace; rerun restarts in place,
  `--delete`/`--clean` removes them, `--json` returns the `start`
  contract. Missing `node`/`python3` skip their apps with a warning.
- `3pm ls` (and every command that reprints the table) ends with pm2's
  one-line host summary — `host metrics | cpu: … | load: … | ram: … |
  gpu: … | en0: ⇓ … ⇑ … | disk: ⇓ … ⇑ … /data 85%` — read from the host
  agent's cache under a 500ms budget (absent, never an error, when the
  agent has no sample). Throughput comes from new agent-derived rates on
  the wire: `HostNic.rx_bytes_per_s/tx_bytes_per_s/errors_per_s`,
  `HostDisk.read_bytes_per_s/write_bytes_per_s`,
  `HostMetrics.rate_window_ms` (additive, optional). `ls --json` unchanged.
- Host metrics agent: `3pm hostmetrics` is now served from a separate
  process (`3pm __host-agent`) the daemon spawns, respawns with backoff
  and terminates — host collection can no longer take the supervisor
  down. The agent samples in cost tiers on independent tickers — cheap
  gauges every 1s (`THREEPM_HOSTMETRICS_INTERVAL_MS`), process/socket
  walks, sensors, GPU tools and `statvfs` every 10s
  (`THREEPM_HOSTMETRICS_MEDIUM_MS`), NTP and identity every 5min
  (`THREEPM_HOSTMETRICS_SLOW_MS`) — so a hung `nvidia-smi` or a stalled
  mount never delays CPU/RAM, and the fast collector keeps its CPU
  window across ticks (no 200ms warmup after the first sample). The
  `ls` throughput is a 1s rate. Wire: additive
  `HostMetrics.sampled_at_epoch_ms.{fast,medium,slow}`; the overview
  adds `sampled (medium)`/`(slow)` rows when a tier lags. The agent
  answers from its cache in milliseconds; the overview shows a
  `sampled … (Ns ago)` row, `3pm internals` a `host agent` row (pid,
  restarts). An unreachable agent is a clean `err{unavailable}` naming
  `hostagent.log`. Home gains `host.sock`, `hostagent.{lock,pid,log}`.
  Wire: additive (`HostMetrics.collected_at_epoch_ms`,
  `DaemonInternals.host_agent_pid/host_agent_restarts`,
  `ErrCode::Unavailable`); `client::connect_at`/`ReadyPipe` for embedders.
- `@3pm/api` (`packages/node`): Node.js client with pm2's programmatic
  shape (`connect`/`start`/`list`/`stop`/`restart`/`delete`/`describe`/
  `scale`/`save`/`launchBus`, promise-based, zero dependencies) over
  `rpc.sock`/`pub.sock`, daemon auto-spawn, `EventFilter` handshake,
  protocol-version check, and the frozen wire fixtures in its tests.
- `3pm runtime <file|script>`: container entrypoint (PID 1). Runs the
  daemon in the foreground as a child, streams app output to
  stdout/stderr (`--raw`, `--format`, `--timestamp`, `--json`), drains
  on SIGTERM/SIGINT, reaps orphans, and exits `2` when nothing is left
  to supervise (zero apps, or every app stopped/errored; `--no-auto-exit`
  keeps it up), `0` after a drain, `1` if the daemon died. Log files are
  off unless `--log-files`/explicit paths; `--delay SECS`. Every `start`
  flag applies. Wire/daemon: `SpawnMode::Foreground`, `__daemon
  --foreground`.
- `3pm events`: follow the daemon event bus, one line per event, with
  repeatable `-f GLOB` event filters (`process:*`, `log_err`, …),
  `-p NAME` app scoping, `--raw`, `--timestamp FMT` and `--json`.
- Server-side bus filtering: a `pub.sock` subscriber may send one
  `EventFilter` line (globs, app name, ids) and the daemon drops the
  rest before serialising it. `3pm logs` uses it; filter-less clients
  keep receiving everything. Wire: additive (`event_filter` fixture).
- `daemon_shutdown` event (`reason`: `kill`, `signal`, `home_removed`)
  broadcast once before any child is signalled and drained to every
  subscriber before the socket closes. `3pm logs` / `3pm events`
  followers now exit 0 with the reason on a deliberate stop, and fail
  with "daemon connection closed unexpectedly" only on a real death.
- Ecosystem values expand `~` and `$VAR` / `${VAR}` (`$$` escapes); an
  unset variable is an error, not an empty string.
- `3pm stop|restart|delete <ecosystem file>` acts on every managed app
  the file declares.
- Resource limits beyond memory: `max_cpu_percent` (soft CPU ceiling,
  same observe-and-restart guard as `max_memory_restart`),
  `cgroup_enforce` (Linux only: per-incarnation cgroup v2 leaf under
  `/sys/fs/cgroup/3pm` with hard `cpu.max` / `memory.max`; unsupported
  hosts fail the spawn before it happens) and `deny_gpu` (CUDA/ROCm
  visibility env vars). CLI: `--max-cpu-percent`, `--cgroup-enforce`,
  `--deny-gpu`. Wire: three additive `StartSpec` fields.

### Fixed
- `watch` on a single file fires. A root that is a file — a binary a
  build rewrites, `watch = ["target/debug/app"]` — is watched through
  its parent directory and only its own events count: the filter used
  to drop them as "the root itself" (a directory's own mtime event,
  which is noise), so the app never restarted; and on Linux a watch on
  the file itself was lost the moment a build replaced it (new inode).
  Its siblings in that directory — `app.d`, `deps/` — do not restart it.
- Fleet link, after the adversarial review of phases A–C. Backend: the
  60 s silence close really fires (one deadline reset only by agent
  frames; `--handshake-timeout-ms`, `--ping-every-ms`, `--silence-ms`);
  a refused link is drained before the socket drops, so the agent sees
  its close code instead of an RST; the hub view is read fresh after
  `welcome` and the node state re-checked after registering (a revoke
  racing the handshake now cuts the session; a re-issued certificate
  shows its serial); graceful shutdown cannot hang on a feed opened
  during it; enrollment has its own nonce store, releases nonces on
  every exit and budgets pending handshakes per client address
  (`--enroll-pending-per-ip`, 4429) so a tokenless flood cannot starve
  the links; enrollment persists use + node + serial + certificate in
  one transaction (no burnt use on a failure); WebSocket messages and
  frames are capped at 1 MiB before inflation; hyper's header-read
  timeout is armed (`--header-timeout-ms`); asset paths with `..` are
  refused before the embed store; mutating routes refuse cross-site
  requests (403 `ERR_FORBIDDEN`) and, with `--admin-token`, require a
  bearer (401 `ERR_UNAUTHORIZED`; the dashboard asks once). Link crate:
  `RedactedRecord.status` and `AgentEvent::Process.kind` are link-local
  enums with `Unknown` (a newer peer's variant no longer breaks the
  frame); a telemetry frame over 1 MiB is trimmed (`env_keys`, then the
  tail of `procs`, `procs_dropped`) instead of tearing the link down
  forever; a bare 4403 is Auth, not Stop; `version_overlap`. Agent: the
  handshake skips unknown and early `cert` frames; shutdown is honoured
  while connecting and shaking hands (no SIGKILL by the daemon when the
  backend is unreachable) and never sleeps a backoff afterwards; the
  event-bus forwarder backs off after an EOF; the identity ping is
  bounded; enrollment checks the server clock (`ERR_CLOCK_SKEW`). Plus
  the tests the review found missing on every side.
- `3pm ls | head` (any command whose reader closes the pipe early) died
  with a Rust panic — "failed printing to stdout: Broken pipe" on stderr,
  exit 101 — because `println!` panics on a write error. Every CLI write
  now goes through the output boundary (`oprint!`/`oprintln!`/
  `oeprintln!`, and the `__man`/`completions` generators render into a
  buffer instead of the raw fd): a vanished reader ends the command
  silently with exit 0, streamers (`3pm logs`, `3pm events`) included.
- `3pm <module> … --json` on a first run (daemon not yet up) printed the
  "Spawning 3pm daemon" notice on stdout ahead of the module's JSON
  document; the module's `--json` is now honored by the dispatcher
  (notice to stderr, `THREEPM_JSON` set), so stdout stays one document.
- An init-managed daemon (`3pm startup` units, `3pm __daemon --boot` by
  hand) released its own singleton lock at startup: readiness was
  signalled on fd 3 unconditionally, and without the CLI's pipe that
  descriptor is the daemon's `daemon.lock` — the byte went into the lock
  file, closing it dropped the `flock`, a second daemon on the same home
  could come up and steal the sockets, and the dangling lock `File`
  closed whatever had recycled fd 3 (an app log) on exit. The launcher
  now announces the pipe as `THREEPM_READY_FD`; unannounced, no
  descriptor is touched. Restart a daemon started by an older build to
  regain the lock.
- Config workflow (post-audit): `apply --only … --prune` no longer
  deletes apps that are still in the file but outside the subset
  (`--prune-all` + `--only` is refused); `instances = 0` no longer
  makes `apply` report a change on every run, and an instances-only
  change is a `scale` step (survivors keep their pid); `--only` naming
  a broken app reports that app's error instead of "no app named", and
  a broken app outside the subset no longer blocks the selected ones;
  duplicate app names are refused by `start`/`apply` (not only
  `validate`); env values may be bare numbers/booleans; a malformed
  `env_<n>` table or `[defaults]` misuse is reported per app instead of
  failing the file or passing silently; `validate` runs without a
  socket-capable `THREEPM_HOME`; `restart --update-env` prints loader
  warnings and, under `all`, skips CLI-started apps with a notice;
  `start <file>` / `apply` warn when a remembered profile is dropped;
  unknown-field notices no longer claim the field was "ignored" where
  `validate` rejects it.
- Daemon shutdown (`3pm kill`, SIGTERM) is bounded by a 30 s drain
  deadline; an unkillable child no longer hangs the daemon forever.
- A panic in a per-child task is reported as the incarnation's death
  (synthetic `ChildExited`) instead of leaving the row `online` with a
  stop that never resolves.
- Specs decoded from the wire or `dump.json` are validated
  (`StartSpec::validate`) before becoming rows: an invalid `Start` is
  refused with `InvalidSpec`; an invalid dump entry is skipped with a
  warning while the rest resurrects.
- Lifecycle timers (restart backoff, kill escalation, ready wait,
  dependency polls) are cancelled when their incarnation is superseded
  instead of lingering until they fire into the gen guard.

### Changed
- `modules/agent` is a git submodule of its own repository
  (`3pm-agent`, next to this one). Clone with `--recurse-submodules`;
  the `modules` and `msrv` CI jobs and the release build check it out.
  Nothing changes for users: the agent is bundled as before.
- The fleet backend left this repository: `3pm-backend` is now its own
  repository, checked out next to this one and path-depending on
  `../3pm/crates/…`. `just backend` / `just backend-dev` and the CI
  `backend` job are gone from here; the real-agent test runs there with
  `just e2e`.
- `3pm internals` reports `armed timers` (wire: `timers_armed`, additive).
- The `max_memory_restart` comparison and the "only online rows restart
  on a watch/cron tick" rule moved from the daemon into the pure policy
  (`MemorySampled{gen, rss_bytes}`, `TriggeredRestart`).
- The supervisor actor no longer does disk I/O inline: `save` (two
  fsyncs), the `resurrect` dump read and orphan probe, the per-spawn pid
  file write and the `internals` fd/load/file gauges run on blocking
  threads. A slow or hung `THREEPM_HOME` can no longer stall every
  process's timers and signals. The pid file lands a few ms after the
  process is reported online.
- The `depends_on` wait budget (`depends_timeout_ms`) is decided by the
  pure policy (`dep_wait_since` on the state, `Action::ForceSpawn`),
  not by the daemon; the model-based property test now covers it.

### Internal
- `Launching` sub-states are a policy-internal `Launch` enum
  (`SpawnInFlight` / `ReadyWait` / `ParkedOnDeps`) instead of
  `waiting_deps` + `pid.is_some()` probes; `Status` stays the wire type.
- Policy transitions share `begin_stop` / `park` / `reset_series`
  helpers; `delete` is a real transition instead of a filtered `stop`.
- Unit tests for `bus`, `ctl_server`, `metrics` and `bootstrap`.
- E2E: every fixed settle sleep replaced by a conditioned wait
  (`show --json`-backed `wait_for_*` and `expect_no_change_for`), port
  tests let the app choose its port, and the CI `retries` on the e2e
  group were dropped.

### Added
- Wire compatibility is now decided by `PROTOCOL_VERSION` (carried as
  `protocol` in `pong`) instead of the crate version: CLI and daemon of
  different builds interoperate as long as the protocol generation
  matches; a daemon predating the field is reported as skewed with the
  `3pm update` hint. Frozen JSON fixtures for every message
  (`crates/threepm-protocol/tests/fixtures/wire/v1/`) gate additive-only
  growth in CI; `docs/protocol.md` documents transport, versioning rules,
  the message catalogue and which `--json` fields are stable.
- `3pm sendSignal <SIG> <target> [--group]` (aliases `send-signal`,
  `signal`): deliver a signal to the running process (pm2) or its process
  group; not a lifecycle operation.
- `cron_restart` (ecosystem) / `--cron-restart` (start): restart on a
  cron schedule, 5 fields (pm2) or 6 with seconds (OxMgr); `show` prints
  the schedule and its next fire, `ls --json` carries
  `next_cron_epoch_ms`. Only online rows are restarted, like watch mode.
- Ecosystem profiles: `[defaults]`, `[apps.profiles.<name>]` (any
  field) and pm2's `[apps.env_<name>]`, selected with `--env <name>` on
  `start`/`apply`/`validate` (an undefined profile is an error);
  `--only a,b` on the same three. The file path and profile are recorded
  on the spec (`show`: `config file` / `profile`).
- `3pm validate <file>…`: check ecosystem files without starting
  anything (unknown fields are errors here, plus spec errors, duplicate
  names, dependency cycles, external `depends_on` — `--check-managed`
  asks the daemon); exit 1 on errors, `--json` findings.
- `3pm apply <file>… [--prune|--prune-all] [--dry-run]`: idempotent
  reconcile — create / update (restart with the new spec) / restart
  stopped / leave unchanged apps running; `--prune` deletes apps that
  came from the file and are gone from it.
- `3pm restart <target> --update-env`: re-read the app's ecosystem file
  (same profile) and restart on the fresh spec.
- Restart policies: `restart_policy = "always" | "on_failure" | "never"`
  in ecosystem files and `3pm start --restart-policy always|on-failure|never`.
  `on_failure` respawns only after a non-zero exit, a signal death, or a
  failed spawn; a clean `exit 0` leaves the app `stopped`. pm2's
  `autorestart = false` / `--no-autorestart` remain as the `never`
  shorthand (a contradiction between the two is a config error), and
  pre-existing `dump.json` files keep their meaning. `3pm show` gains a
  `restart policy` row.

### Fixed
- `hostmetrics` TCP state counts on Linux now read `/proc/net/tcp{,6}`
  and see every socket; pid-attributed enumeration missed other users'
  sockets and ownerless TIME_WAIT when running unprivileged.
- `hostmetrics` no longer counts Linux userland threads as processes
  (`processes_count`/`process_states` now match `ps ax`).

### Changed
- A typo'd `kill_signal` now fails at `start` with a did-you-mean hint
  instead of silently never taking effect.

### Added
- Watch mode: `3pm start --watch` (the cwd) or `--watch=PATH` (repeatable)
  restarts the app on file changes, with `--ignore-watch GLOB` (added to
  the built-in dotfiles + `node_modules`), `--watch-delay MS` (trailing
  debounce, floor 100ms) and a hidden `--watch-poll MS`; ecosystem keys
  `watch` (`true` | path | list), `ignore_watch`, `watch_delay`,
  `watch_poll`. Native OS events via `notify`, automatic polling fallback
  reported as `watch_error` on records (`ls` note, `show` row). One
  watcher per app, derived from state (no phantom "enabled"). `3pm restart
  <app> --watch` toggles like pm2; `ls` gains a `watching` column and
  `show` a `watch & reload` row.
- Exit context on crashes: `ls`/`show --json` records and the `process`
  `exit` event on the bus now carry `exit_signal` (POSIX name of the
  killing signal, e.g. `SIGSEGV`) and `stderr_tail` (the last 30 stderr
  lines, byte-bounded), next to `exit_code`. Both are cleared by the next
  successful spawn. `3pm show` prints the signal (or exit code) as a row
  and the stderr tail under the table; `3pm logs --json` forwards process
  events.
- `3pm hostmetrics` reports GPUs: model/vendor/driver, utilization,
  dedicated VRAM (NVIDIA via `nvidia-smi`, AMD via `/sys/class/drm`) or
  unified memory + core count + driver-recovered resets (Apple via
  `ioreg`), temperature, power, fan. Absent fields stay absent — cards
  no source can instrument still appear as identity-only rows.
- Child control channel (`ctl.sock`): apps can signal readiness and
  receive graceful-stop messages over a Unix socket (any language;
  env `THREEPM_CTL_SOCK`/`THREEPM_ID`/`THREEPM_GEN`). New start flags
  `--wait-ready` (+ `listen_timeout`, pm2 semantics) and
  `--shutdown-with-message`; all three settable in ecosystem files.
- Cluster mode: `3pm start -i N` runs N instances of one app (`-i 0` =
  one per CPU; also `instances = N` in ecosystem files), each with
  `THREEPM_INSTANCE`/`NODE_APP_INSTANCE` env and per-instance log/pid
  files; `3pm scale <name> <n>` resizes without restarting survivors.
  `3pm show` reports every instance of a name (`--json`: an array).
- `backoff_factor` is configurable from ecosystem files.
- Global `--json` flag: machine-readable output on stdout for all
  daemon-backed commands (NDJSON events for `logs`); errors become
  `{"error":{code,message}}` on stderr with unchanged exit codes.
- `3pm startup` / `3pm unstartup`: boot-service registration for
  launchd, systemd, OpenRC, and rc.d — prints the privileged command,
  service runs the daemon as the invoking user.
- `justfile` task runner, `CLAUDE.md`, release automation and
  distribution packaging (deb/rpm, APT/YUM repos, install script).

## [0.1.0] - 2026-07-07

Initial development version: pm2-workflow process manager as a single
`3pm` binary (CLI + auto-spawned daemon).

### Added
- Process lifecycle: `start` (scripts, binaries, ecosystem `.toml`/`.json`),
  `stop`, `restart`, `delete`, `ls`, `show`, `logs` (tail + realtime
  stream), `save`/`resurrect`, `kill`, `update`, shell completions.
- Supervision: auto-restart with exponential backoff (on by default),
  crash-loop protection via consecutive-quick-exit accounting,
  `--max-memory-restart`, per-app timing knobs, group-kill stop
  sequence with SIGKILL escalation.
- Robustness: pure lifecycle state machine, pid-reuse-safe
  signal/reap serialization, orphan reconciliation after unclean
  daemon death, capped byte-oriented log pumps, `flock` daemon
  singleton, version-skew detection on every connection.
- Security: sockets are `0600` and peer-uid-checked.

</document>

<document url="https://3pm.edgecr4ft.net/docs/backend/changelog.md" title="3pm-backend changelog">

# Changelog

All notable changes to 3pm-backend. The format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased]

### Added
- **A re-enrolled host replaces its old row** (`http/supersede.rs`): when
  a node becomes active (enrollment, `approveNode`, approval turned
  off), every older row of the bucket with the same hostname (ASCII case
  ignored, never empty) that has no live session and is `offline` is
  deleted as `deleteNode` does (`api_nodes::forget`, shared). Kept: a
  linked, `stale` or revoked row; a pending node replaces nothing; the
  newest of twins admitted together stays (row id order). Fixes the
  reinstalled tablets of 2026-10-10 (new `node.key`, the old row
  offline under the same name, `3pm fleet ssh <host>` ambiguous). Rows
  already duplicated are not swept retroactively. Tests: `FakeAgent`
  gains `hostname`; `real::link_node` waits for the node to be `online`
  in the bucket, since a rig's two daemons share this machine's name.
- **`getBucket` takes the feed's query** (`history=0`, `match=`,
  `view=live`, applied as to the feed's `init`; unknown values 400):
  `3pm fleet nodes` asks `history=0&view=live` — 2.1 MB → ~65 KB on the
  prod bucket of 2026-10-07 (2 MB of it was `history`, which a node with
  `fleet:read` received and never read). Without the query the answer is
  unchanged; the generated method keeps its signature and takes it in
  `options.query`. `view=live` keeps `module` on each process row (the
  feed's frames too): `3pm fleet apps` prints it.
- **Link traffic by category** (lot 3 of `rd/logs-bytes-counter/PLAN.md`,
  no schema change): the agent's `traffic` section (agent ≥ the one of
  lot 2) is read at every store tick of a collected node through
  `NodeState::traffic` + `traffic_view` and stored as
  `threepm_link_bytes_total` / `threepm_link_messages_total`
  `{cluster, node, link_dir, link_kind}` — the closed vocabulary only
  (`Traffic::known`, ≤ 30 pairs), an unknown kind dropped and warned once
  per node. Catalog `link.bytes` / `link.messages` (counters, `rate`),
  `link_dir` / `link_kind` as `Custom` labels on those lines only (no
  reserved word). The node card gains a "Link traffic" panel
  (`queryMetrics`, `rate(…[1m])` by direction and kind, state kinds
  "est.", `rx_bps` beside). An older agent sends none: nothing stored.

### Fixed
- **Logs ingest, review of 2026-10-07** (no schema change): dropping
  what the catalog refused is linear (a set of the refused templates,
  not a scan of the list for each record: 50 000 templates, 20 000
  refused went from 4.65 s to 26 ms in a debug build) and runs off the
  async runtime (`spawn_blocking`), so a batch of tiny `tpl` records
  against a full catalog no longer blocks a tokio worker for seconds; a
  body read then refused `503` (the fleet's 32 slots taken) is charged
  to the node's budget at its compressed size, so it is not uploaded
  again for free; that `503`'s `retry-after` is 2 s, plus 1 s per 32
  batches turned away in the last 10 s window (a slot freed does not
  reset it), 10 s at most (was
  1 s, which brought a reconnecting fleet back all together); a new
  source at the 1 000-source cap never evicts a source the same batch
  writes counts, lines or drops under (the keep set was its declared
  sources and its templates' only).
- **Logs, review of 2026-10-06** (no schema change, still V22):
  `logsQuery` asks its counts five nodes at a time without a `node`
  filter (VictoriaMetrics refuses a query over
  `-search.maxUniqueTimeseries`), a refused chunk is
  `truncated.counts`, not a failed answer; the log catalog is purged of
  rows not named for 30 days (hourly) and a node at its bound evicts its
  least recently seen source/template instead of refusing every new one
  forever (a refusal left is said in the answer, `refused`, and logged);
  a template counted is marked seen once a day; a change of seats writes
  the seats that moved, in the batch's one transaction (was ~4 000 row
  writes for one new template); `filters.tpl` needs `node` and `src`
  (`400`); records with a control character in an id or label are
  skipped (also line/paragraph separators, zero-width and bidirectional
  marks); at most 32 batches stored at once fleet-wide, taken once the
  body is read (`503` + `retry-after`), a body slower than 30 s a `408`;
  eviction never takes a row the batch names, the `refused` list is the
  link's `LogsAccepted.refused`; count chunks run 4 at once; parameter `distinct` is merged over the whole range.
- **Insights log rules**: count queries scoped per node and template,
  failures and truncations counted (`partial_rules`), chronic kernel
  errors are `inventaire` unless ×3 their usual rate, `log_new_error`
  skips templates of a miner epoch younger than a day (state reset; the
  epoch is read from any series of the node, so an old quiet source still
  fires); "chronic" needs every day of the history there is, 3 to 7.

### Added
- **Logs query** (`logsQuery`, `POST /api/buckets/{id}/logs/query`, the
  owner's, like `eventsQuery`): `{range ≤ 30 d, filters? {node, src,
  owner, sev (a word, a list, or `warning+`), tpl}, text?, limits?
  {templates ≤ 500 (200), lines ≤ 1000 (200)}}` → `{range, sources,
  templates, lines, truncated}`. Sources from the catalog with the lines
  counted, shipped and dropped over the range and their templates;
  templates worst severity first then by lines, text and first/last seen
  from the catalog, parameter stats merged (`distinct` the largest, `top`
  the most frequent), the unlabelled ones as one `tpl: null` "(other
  templates)" per source and severity; shipped lines newest first. One
  instant VictoriaMetrics query and three VictoriaLogs queries (tenant
  `0:1`) per request; bounds in `limits.logsQuery`.
- **Log ingest** (`pushLogs`, `POST /api/buckets/{id}/logs`, the
  agent's own route, `link-protocol.md` §6b): a node's batches of logs,
  gzip NDJSON, its fleet bearer, any active node of the bucket. With
  `--events-url` the `welcome` now says `logs {apps, host, files}`;
  without it the route answers `503 ERR_STORE_DISABLED`. Refused before
  the body is read: `content-length` missing or over 1 MiB (`413`), a
  batch of the node already in flight or the node over its budget
  (128 KiB/s inflated, 4 MiB burst: `429` + `retry-after`); a body that
  inflates past 4 MiB is a `413`. Records that do not read, are of an
  unknown kind, break a bound or name an undeclared source are skipped
  and counted (`{accepted, skipped}`); a retried batch (same `run`,
  `seq` not past the last) is `{duplicate: true}` and stored once, and
  a line is stored once per `(node, src, pos)` (last 50 000 per node, in
  memory). Sources and templates in SQLite (migration V21:
  `log_sources`, `log_templates`; 1 000 sources and 50 000 templates per
  node), counts as `threepm_log_messages{cluster,node,src,tpl,sev}` (the
  node's 2 000 busiest templates labelled, the rest `tpl="other"`) and
  drops as `threepm_log_dropped{…,src,reason}` in VictoriaMetrics, lines
  and parameter stats (`kind=line|param`) in VictoriaLogs tenant `0:1`,
  stream `cluster,node,src` — events stay in `0:0`. A store that does not
  take the batch is a `503 ERR_STORE_UNAVAILABLE`; the retry writes only
  the half still owed, under the labels first chosen. The template seats
  survive a restart (migration V22, `log_templates.seated`); a template
  id reused by another template (another severity, no word in common)
  is first seen again. A body that is not gzip or does not open with
  `batch` is a 400. Counted in `GET /api/v1/info` → `store.logs`.
- **Sessions only for the caller's own key** (with 3pm's agent of the
  same release): the agent's new refusal code `key_not_authorized` (the
  caller's `cli.key` is not in the node's own `authorized_keys`, the
  message naming the fingerprint and the `3pm link authorize` to type)
  settles a session as `err.code` `forbidden`, keeping `key_not_authorized`
  in `result.code`; an older backend stored it as `unknown`, so deploy
  this one before the agents. `nodeShell` takes `record` (the caller's choice, `true` only with `--record`, absent from an older 3pm = yes): it only
  lowers the bucket's `record_shells`, never raises it, and the
  effective value goes back in the `CallView` before the handshake that
  binds it. The request the node receives is bound to the session's
  handshake: a backend that changes argv, `--user`, `host:port` or
  `record` on the way gets no session. Proven against the real agent by
  `tests/it/real_agent_access.rs` (a key the node does not admit, a
  rewritten request, an `-R` connection taken by another key, no
  `authorized_keys`, a refused burst).
- **Insights read the logs** (`insights/apps/log_rules.py`): four
  rules on what the nodes ship, each finding its templates by their
  shipped lines in VictoriaLogs tenant `0:1` (7 days) and summing their
  counts in `threepm_log_messages`. `log_segfault` (kernel `segfault at`
  / `general protection` ≥ 3 in 1 h on a node), `log_unit_failed`
  (systemd `Failed with result` in 24 h, naming the unit; one alone is
  `inventory`), `log_kernel_errors` (kernel `err`/`crit` lines over 100
  in 1 h, e.g. ACPI or `brcmfmac`), `log_new_error` (an `err`/`crit`
  template with lines in the last hour and none in the 7 days before,
  on a node that shipped logs a day ago; its text from the node's `new`
  line). A store query that fails or answers garbage skips that rule
  for the pass, not the whole rules pass.
- **`insights` / `insight`** (`GET /api/buckets/{id}/insights[/{incident}]`):
  the incidents the insights apps rank for a bucket, relayed from their
  `triage` (`--insights-url`, `THREEPM_BACKEND_INSIGHTS_URL`; contract
  `GET /api/v1/incidents?cluster=` version 1), this bucket's cluster
  only, cached 30 s. Text for an LLM by default, JSON with `?format=json`;
  each incident's store name linked to the catalog and a `queryMetrics`
  body to draw it, no MetricsQL served. Filters `tier`, `node`,
  `min_score`, `limit`, `known`. Fleet-wide incidents name their
  machines (`hosts`) and draw them; an unknown incident id is a 404 that
  never reaches triage; a malformed incident is left out; the URL is
  checked at start. New codes 409 `ERR_INSIGHTS_DISABLED`,
  503 `ERR_INSIGHTS_UNAVAILABLE`. Open to a node's bearer with
  `fleet:read`. `src/insights/`, `src/http/api_insights.rs`.
- **`nodeListen`** (`POST …/nodes/{node}/listen {bind, port, e2e}`,
  `3pm fleet ssh -R`): one long `AgentRequest::Listen` on two sealed
  `pipe` legs (`e2e` required, else 422; `bind` empty = the node's
  loopback, `port` 0 = the node's pick). The node says `bound`, then
  `accepted {conn, peer}` per connection; the caller's keepalive every
  20 s keeps the quiet legs alive (no sweeper change). A non-loopback
  `bind` is the node's own `[shell] gateway_ports`, never this
  backend's. Guarded exactly as `nodeForward` (`3pm:forward`,
  `ERR_FORWARD_NOT_ALLOWED`, no cross-site `Bearer`); never recorded;
  a shell's call for every sweep. Audit: `open_requested` / `refused`
  with `bind` and `port`, `ended`. `src/http/api_listen.rs`.
- **`nodeForward {accepted: {listen, conn}, e2e: true}`**, with no
  `host`/`port` (422 otherwise): the node takes over connection `conn`
  of the caller's listen instead of dialing. **One 404**
  (`ERR_NOT_FOUND`, `no running listen <id> of yours on node …`) unless
  `listen` is a `listen` call on this node, placed by this caller, still
  running — the same words for each case, so a guessed id tells nothing
  of another caller's listen; `3pm:forward` is checked before it. Audit:
  `listen` and `conn` in place of `forward`. `e2e` on `nodeForward`
  is passed to the agent's `forward` (`-W`, `ssh`'s `ProxyCommand`, is
  a plain sealed forward: nothing new here).
- **End-to-end `3pm fleet ssh`: an opaque relay.** `nodeShell` and
  `nodeForward` take `e2e: true`: the legs carry Noise records the
  backend relays without reading, never tees, never records. Refused
  at once with 409 `ERR_NODE_NO_E2E` (naming the agent's version) for a
  node whose session did not announce `shell_e2e`. The request carries
  `e2e` and `record` (the bucket's `record_shells`) to the agent.
- **Recordings made by the node.** For a sealed session the agent
  uploads the asciicast (`putCallArtifact`, cap `RECORDING_MAX_BYTES`
  for `3pm:shell`) and names it in `exited {recording}`; the backend
  claims it as the call's artifact (`shell::kept_recording`) and the
  `ended` line names it — or says `recording: lost` when the agent
  died before sending it. A sealed session may go LAN direct even when
  the bucket records shells.
- **A copy is never recorded.** An `exec` whose argv is one of `3pm
  fleet cp`'s fixed scripts, byte for byte
  (`threepm_link::cp::is_cp_transfer`), asks `record: false` and is
  not teed in clear either: a download's screen is the file. The audit
  keeps its command line and paths; its `ended` line names no
  recording, `lost` included. The agent applies the same predicate.
- **Audit: `e2e` and `cli_key`** on `open_requested` and the agent's
  `open` (the caller's key fingerprint).
- **Shell legs may go LAN direct.** `nodeShell` and `nodeForward` take
  `direct: true` (the caller takes its legs through its own agent):
  between two neighbours both set to `direct: all`, the answer names
  the target (`peer`), the target's ticket the caller, and a `pipe`
  leg the writer reports `direct` leaves the relay like a `live` one
  (kept hidden, fed by `blob_stat`). Never for a session in clear
  that is recorded (`record_shells` on, the default): its screen is
  read off the relay. An end-to-end session, recorded by its node, may.
  A forward, never recorded, goes direct whenever its caller asks.
- **Session recording** (`record_shells`, on by default; schema V18):
  the screen leg of every shell, command and attach session is copied,
  off the relay through a bounded queue that never waits, into an
  asciicast v2 artifact of its call (`ext` `cast`, action `3pm:shell`),
  64 MiB at most, a marker on a cap or a gap; never the keyboard leg,
  never a forward. Its id is on the backend's `ended` audit line
  (`recording`). `setBucketSettings` takes `record_shells`; settings
  page toggle "Record shell sessions".
- **`nodeForward`** (`POST …/nodes/{node}/forward {host, port}`, `3pm
  fleet ssh -L`): one `AgentRequest::Forward` per local connection, on
  two `pipe` legs carrying raw bytes (`END` = half-close). Guarded like
  `nodeShell` but by its own `callers` action `3pm:forward` and its own
  403 `ERR_FORWARD_NOT_ALLOWED`; closed to a cross-site `Bearer`.
  Audited (`open_requested` / `refused` with `forward = host:port`,
  `ended`); a shell's call everywhere (`is_shell`, the SQL sweeps).
- **`nodeShell` takes `user`** (`3pm fleet ssh -l <user>`): the shell
  or the command placed as `run_as {user, req}` (422 for a name that is
  not a plain login, or with `attach`); `user` on the `open_requested`
  and `refused` audit lines. A `run_as` call is a shell everywhere a
  shell is found (its legs, its view, the sweeps).
- **`nodeShell` attaches to a `tty` app** (`3pm fleet attach <node>
  <app>`): the body takes `attach {app, instance?, mode?, read_only?,
  peek?}` (exclusive with `command`, 422 otherwise) and places
  `AgentRequest::Attach`. Its calls are a shell's everywhere
  (`is_shell`, the SQL sweeps); `open_requested` and `refused` carry
  `app`; the agent's new `attach` audit line is stored with `app`,
  `instance`, `mode`, `read_only`.
- **`nodeShell` runs one command** (`3pm fleet ssh <node> -- <cmd>`):
  the body takes `command` (its words: 1 to 256, 16 KiB in all, no NUL,
  the first not empty, else 422) and `pty`, and places `AgentRequest::Exec`
  instead of `Shell`. The calls of an `exec` are a shell's everywhere
  a shell's are found (the legs hung up on a refusal, a session whose
  legs are gone brought forward, a reconnect leaving it in flight);
  `open_requested` and `refused` carry `command` (the line the target's
  shell is given, `threepm_link::exec_line`: `cat '>' /etc/x`, never
  the words space-joined, which read `>` as a redirection), and the agent's
  `exec` audit event is stored (`event: exec`, `command`, `pty`).
  `MAX_BODY_BYTES` of the route grows from 1 KiB to 64 KiB.
- **`nodeShell`: an interactive shell on a node** (`POST
  /api/buckets/{id}/nodes/{node}/shell`, body `{term, cols, rows}`, the
  server side of `3pm fleet ssh`). Opens a pair of `pipe` legs charged
  to the caller and places `AgentRequest::Shell` with `from` (the
  call's `placed_by`) and `from_hostname`; the answer is the `CallView`
  (`input.blob` the keyboard, `output.blob` the screen, `live`
  counters while it runs). Same gate and scope as `nodeAction` (the
  account, or a node's bearer with `fleet:invoke`), but a node needs the
  target's `callers` empty or naming `3pm:shell` with `app` `-` — else
  403 with a new code, `ERR_SHELL_NOT_ALLOWED` — and a `Bearer` from
  another site never opens it (`guard::BEARER_CLOSED`); `term` is 1–64
  characters of `[A-Za-z0-9._+-]` (422), the size clamped; refused 409
  while the backend stops. `nodeCall` with `op: shell` is a 400 that
  names this route. The reply `exited` settles the call (`ok` on a
  zero status, else `err` `failed` saying the code or the signal), and
  may come on a **later** link connection: one node's link closing no
  longer settles its in-flight shells (a restart still does); a shell
  whose two legs the relay has forgotten gets a one-minute deadline
  instead (`db::calls_shell`). **Audit** (`kind = shell` in the events
  store): the backend writes `open_requested` when the call is placed
  (`refused`, with why and no `call`, for an attempt it turns away once
  it knows the target) and `ended` when it settles — on the agent's
  answer, its refusal included, and on a `lost` / `unacked` settled by
  the sweeper or a restart; a shell the agent refuses has its two legs
  ended at once (`Registry::hang_up`), not at the relay's idle minute; the
  agent's `AgentEvent::Shell` events become `open`, `line`, `screen`,
  `close` lines, none of them rate-limited, and none of them on the
  bucket's feed (a node with `fleet:read` follows it). `eventsQuery`
  takes a `call` filter. Behind nginx the shell needs the block of
  FEATURES "Behind nginx" as it is: `proxy_http_version 1.1`, request
  buffering off and timeouts over a minute, or a quiet shell is cut.
- **`pipe` blobs: a stream that loses nothing** (the transport of the
  coming `3pm fleet ssh`). A third blob mode beside `file` and `live`:
  framed like `live`, but its window is trimmed only behind its reader
  and its `PUT` stops being read once 256 KiB ahead
  (`PIPE_WINDOW_BYTES`), so TCP carries the back pressure home instead
  of the window dropping messages (a message bigger than the window is
  let through to its end, or writer and reader would wait on each
  other). One reader, from the first byte,
  once (a second one is 409), memory only, reserved like a `live` leg.
  Opened in pairs by `Registry::open_pipe_session` (both legs charged
  to the caller, or neither). The sweeper: a leg whose one reader left
  is over on the next tick and takes its running twin (a twin that has
  finished is spared, so the end of the screen is delivered); a session
  one of whose legs is parked on its window is waiting, not idle, read
  or not (a reader that never came: 10 min); otherwise the `live`
  minute; never past a day. A given-up pipe cuts its readers at once
  and answers its `PUT` 408 even while the client sends nothing. At
  the door a pipe refuses another site (`Bearer` included), a `?k=`
  (`viewBlob` mints none: 403), `as=mjpeg` (422), and the bucket's
  account when it is not one of its two ends; it never goes direct. A
  backend that stops ends every open pipe, so a session's requests
  close instead of holding the graceful shutdown. `nodeShell` opens
  them (above); `live` and `file` blobs are unchanged.
- **Network traffic per app.** A 3pm node's per-app network figures
  (its daemon's meter, Linux and macOS) are stored:
  `threepm_process_net_received_bytes_total` / `…_sent_…` per app
  (`process.net.received` / `.sent`) and
  `threepm_subprocess_net_received_bytes_total` / `…_sent_…` per tree
  group, accumulated like the other group counters
  (`subprocess.net.received` / `.sent`). The feed's records carry
  `net` (`RedactedNet`: `r`, `t`, `rb`, `tb`, `pa`) and the tree lines
  `net` (`NetUsage`), both in `api.d.ts`. No dashboard display yet.
- **Top processes of a node.** A 3pm node's `top_processes` (the
  heaviest processes of the host by CPU, memory and network, on by
  default) is served live: `nodeHost` and the dashboard feed's host
  carry the table — unlike `extended`, which the feed drops. No series
  is stored per process (pids churn: unbounded cardinality); one
  freshness series, `threepm_host_sampled_top_at_seconds`
  (`system.sample.top`), says when the table last landed. Names are
  cleaned and cut again on the way in, and a table of more than 30 rows
  is dropped whole. `HostTopProcess` is in `api.d.ts`, with
  `footprint_bytes` (macOS nodes: the memory top is ranked on it, not
  on `mem_bytes`; Linux nodes keep the RSS) — live only, no series.
- **Extended Monitoring stored and queryable.** What a 3pm node with
  `[hostmetrics] extended` (on by default) sends is persisted:
  - **Host families:** one series per family, `threepm_hostx_<family>_total`
    (counter family) or `threepm_hostx_<family>` (gauge; an `Unknown`
    kind is stored as a gauge), `field` = the kernel's spelling, the row
    under the label that already means it (`core`, `iface`, `blockdev`,
    `array`; `cpuidle` `<core>/<state>` → `core` + `row`) or a generic
    `row`. A family, field or row off the grammar (name
    `[a-z][a-z0-9_]*` ≤ 48 bytes not ending in `_total`, values ≤ 64
    bytes without control characters) and non-finite values are
    skipped, never mangled. Rendered label blocks are cached per
    (row, field) and evicted like the others.
  - **Catalog 231 → 291 lines:** 42 `host.ext.*` (one per family, the
    three eBPF short-lived-process families included, each saying where
    its fields are documented), 16 `subprocess.*` (faults, context
    switches, disk and logical I/O, syscalls, threads, fds, fd usage,
    PSS, footprint, swap — present only where the platform gives them),
    `process.memory.pss` (the tree's PSS, absent unless every member's
    was read) and `system.sample.extended` (freshness,
    `threepm_host_sampled_extended_at_seconds`). `field` and `row` are
    `Custom` labels like `bat` — an application's own `field` label
    still works — and the inventory lists neither.
  - **Slow-tier counters rate over a floor:** a catalog line may carry
    `min_window_ms`; `host.ext.cpuidle` (read every 5 min) rates over at
    least 10 minutes, capped at the query's range.
  - **The live feed carries no `extended`** (~40 KB a Linux node at
    every tick, drawn by no dashboard): `nodeHost` and `queryMetrics`
    serve it; the store reads the full host.
  - Measured with a real 3pm on edgecr4ft.net (2026-09-29): a node with
    the mode on writes 2 436 series vs 663 off (+1 228 `hostx`,
    +~380–545 per-process series seen in a 2-min window, mostly a
    forking test app's children (~220 at steady state); +50–80
    `shortlived*` with a root agent); its link costs 777 B/s vs 529 B/s. No schema change; a
    node without the mode, or an older agent, writes what it did.
- **The install + link line.** `createToken` and `bucketToken` answer
  `install_line` beside `paste_line`: `curl -fsSL
  https://3pm.edgecr4ft.net/install.sh | sh -s -- link <cluster> <token>
  <server>`, which installs 3pm on a new host and links it (re-run, it
  updates and keeps the link). The bucket page shows it under *Link a
  host* with the bucket's permanent token (hidden when that token
  cannot be read); the settings page and a minted token show both lines.
- **Bucket settings: approval of new hosts can be turned off.** A page
  per bucket, `/b/{cluster_id}/settings` (linked from the bucket page),
  with one box: *Require approval for new hosts*. On — the default, and
  what every bucket did — a host linked with a reusable token is
  `pending` until `approveNode`. Off, every host that links with a
  valid token of the bucket is active at once: the token is the whole
  gate, which is what enrolling a batch of hosts with one reusable
  token wants. Turning it off approves the hosts already waiting, in
  the same transaction, each announced on the feed as a `node` event;
  turning it back on sends nobody back to the queue.
  - **New route `POST /api/buckets/{id}/settings`**
    (`setBucketSettings`), body `{ "approval": boolean }`, answers the
    `BucketView`. The owner's, guarded like every write; a body that
    names no setting, a wrong type or an unknown field is a 400.
  - `BucketView` gains `approval` (`getBucket`, `listBuckets`, the
    feed's `init`). A node reading the bucket with its own bearer does
    not get it: whether the gate is open is the owner's to know.
  - Schema 14: `buckets.approval`, default 1.
- **Bucket setting `require_access_rights`.** A box on the settings
  page, *Require nodes to specify access rights*, **on by default** —
  what every bucket did: a node asks of its bucket only what
  `setNodeAccess` granted it. Off, every **active** node of the bucket
  is treated as holding `fleet:invoke` and `fleet:list` (a target named
  by hostname is found in the bucket's list) for as long as it stays
  off — nothing to grant node by node, and a node that enrolls tomorrow
  talks to the others at once. Never `fleet:read`: hosts, metrics and
  the catalog of actions stay a grant of their own. The rows are
  untouched (`NodeView.scopes` still shows what the owner granted), so
  turning it back on takes back exactly what it gave, and both ways are
  felt on a node's very next request, with nothing to relink. A
  target's `callers` rules still narrow who may call it.
  `setBucketSettings` takes `{ approval?, require_access_rights? }` —
  a setting left out is unchanged, a body naming none is a 400 — and
  `BucketView` gains `require_access_rights` (the owner's, like
  `approval`: a node's bearer does not read it). Schema 16 added
  it as `open_calls` (default 0) and was deployed under that name;
  schema 17 renames it to `buckets.require_access_rights` (default 1)
  and carries each bucket's choice over, inverted.
  - A caller that holds no grant of its own is told its call settled
    (`call_result`) like any other: the setting is read from the
    database at that moment, not from a bucket row as old as the
    target's session.
- **A permanent enrollment token per bucket.** The settings page shows
  one `3pm link …` line that is always the same: the token never
  expires, enrolls any number of hosts, and no route renews or revokes
  it — what a provisioning script (`tabloid link --all`) wants to hold.
  **New route `GET /api/buckets/{id}/enroll-token`** (`bucketToken`,
  the owner's; never a node's bearer): `{ token_id, token, paste_line,
  created_at }`, created on the first read and identical at every read
  after. It is not pre-approved, like any reusable token: with
  `approval` on its hosts wait, with `approval` off it is the whole
  gate. `listTokens` shows it with `permanent: true` (never its
  secret) and `revokeToken` refuses it, 409 `ERR_CONFLICT`.
  - **Its secret is stored in clear** (`tokens.secret`) — the only
    token for which that is true, because it has to be read back;
    every other token is still kept as its hash alone. Whoever reads
    the database can enroll hosts into the bucket.
  - Schema 15: `tokens.permanent`, `tokens.secret`, one permanent token
    per bucket (a partial unique index).
- **A node can be deleted.** `Delete`, next to `Revoke` / `Approve` on
  the rows of the bucket page — every node but one that is **online**
  (deleting it would cut a working link its agent cannot mend by
  itself: revoke it, or wait for it to go stale) — after a
  confirmation, and **new route
  `DELETE /api/buckets/{id}/nodes/{node}`** (`deleteNode`, the owner's,
  `204`; `404` for a node the bucket does not know). The row goes, and
  with it its calls and its artifacts — rows and files; the calls still
  in flight on it are settled first, so whoever placed one is told at
  once. Its bearer is dropped, its live session is closed 4401
  `ERR_NODE_UNKNOWN`, and the hub forgets what it sent. Unlike a
  revoke, nothing is held against the key: the host may enroll again,
  as a new node. What the store recorded (metrics, events) is left to
  the store's retention.
  - **The feed says `reset`** — no new event kind. The hub publishes
    it and every subscriber's stream ends on it, exactly as for a
    subscriber that lagged: the client comes back for a fresh `init`,
    where a deleted node is sure not to be. A page written before
    nodes could be deleted follows it too.

### Changed
- **`require_access_rights` off now also opens `fleet:read`**: every
  node of such a bucket reads the others' hosts, metrics and catalog of
  actions, on top of listing and calling them (`db::effective_scopes`).
  A node that may call every action of every other gained little by
  being kept from reading them, and a caller (the `lan` module) checks
  what a peer declares before calling it. Turn the requirement back on
  to keep the fleet's map a grant of its own.
- **LAN direct on by default**: a node's `direct` setting defaults to
  `all` (a call's files **and** its live legs go agent to agent between
  LAN neighbours) instead of `files`. New nodes enroll at `all`; migration
  V20 turns every node stored at `files` into `all` (`off` is kept) —
  including a node an operator had set to `files` on purpose, which
  cannot be told from the old default: set it back with `setNodeAccess
  { "direct": "files" }`. An unknown or corrupt stored word reads as
  `files` (fail-safe). A live leg gone direct has no Watch preview
  ("direct · no preview"): an operator who wants the old behaviour sets
  `files` again the same way.
- **Process-tree series are stored by group, not by pid.** The
  `threepm_subprocess_*` series are keyed by `(app, instance, ns,
  proc)`: one set per executable name in an instance's tree, whatever
  the number of pids. The `pid` label is gone — a kiosk's shell loop
  (`sleep 3` in a loop) made ~17 000 pids a day per tablet, 708 000 of
  the 735 000 series created on 2026-10-01, and past VictoriaMetrics'
  daily cap every node loses its new series. Gauges are summed over the
  members (`pss`, `footprint`, `swap`, `threads`, `open_fds` only when
  every member gives them); `detached` is now a count; the ten
  `*_total` counters are accumulated per group by the backend, so they
  never go down when a member exits (a member is `(pid, start)`; one
  already running at the group's first sight is a baseline, so a
  backend restart reads as a counter reset; a group unseen for a minute
  starts again at 0). **Removed:** `subprocess.start_time`
  (`threepm_subprocess_start_time_seconds`) and `subprocess.fd_limit`
  (`threepm_subprocess_fd_limit`). **Added:** `subprocess.procs`
  (`threepm_subprocess_procs`, the members of the group) and
  `subprocess.fds.usage_max` (`threepm_subprocess_fd_usage_ratio_max`,
  the highest open fds / soft limit among the members). `pid` is no
  longer a query label: a query naming it gets a 400 (``label `pid` is
  neither built in … nor a valid custom label name``); it stays a
  reserved label key for applications. The old per-pid series stay in
  the store until retention (30 days): a range that spans the deploy
  shows the per-pid series before it and the group series after. The per-member detail stays live
  (`nodeProcessTree`, the tree panel). New module
  `store/tree_groups.rs`.
- **The link is at version 5: the file an action returns is uploaded,
  not sent as frames.** `blob_chunk` is gone from the wire, and with it
  the whole receiving half here — the per-session blobs on disk, their
  bounds and their sweeps (≈ 800 lines). Same flag day as link 4:
  **update the nodes first, this backend last.**
  - **New route `PUT /api/buckets/{id}/calls/{call}/artifacts`**
    (`putCallArtifact`, agent only). The answering node uploads the
    file's **raw** bytes with the bearer of its `welcome`, before the
    reply that names it, and gets back `201 {artifact_id, bytes,
    sha256}`. The bytes are gzipped and hashed (over the raw file) as
    they stream into the store, and the row is written **pending**: no
    read shows it until a reply claims it. Headers: `content-length`
    (required), `x-3pm-ext`, `x-3pm-instance`.
    Refused before a byte of the body is read — 404 (no such call, or
    one that has settled), 403 (a bearer that is not the call's
    **target**, an account), 400 (a header missing, an extension that
    is none, an instance the call never reached), 409 (a second file
    for one instance, a seventeenth for the call), 413 (`content-length`
    past what the action declared it may answer — `output.max_bytes`,
    bounded by 256 MiB; **32 MiB** when it declares none, the cap the
    chunks had — or past what the bucket still has room for, below).
    The refusal is answered **without reading the body**, so an agent
    whose file cannot be kept learns so on the first round trip instead
    of pushing 32 MiB for it. A body longer than it declared, or one
    that stops short, keeps nothing; neither does one whose call
    settled while it was arriving (the row is written under the very
    condition its reply claims it under, so a file nothing can claim is
    never a row).
  - **A bucket's artifact bytes are bounded before the write, not only
    after it.** `UPLOAD_BYTES_PER_BUCKET_MAX` (2 GiB, twice retention's
    1 GiB) refuses an upload the bucket has no room for. The per-file
    cap is the **node's own** declaration, so without this a node
    declaring `o: 256 MiB` and answering a sixteen-instance call writes
    4 GiB between two retention passes, and again on the next call.
    Schema **V13** brings back V8's covering index for the sum.
  - Half-written files cannot outlive their upload: the writing handle
    removes its `.tmp` when it is dropped unfinished, and a start
    sweeps `artifacts/**/*.tmp` for whatever a killed run left (every
    deploy, before this, leaked one per upload in flight).
  - **Claiming.** An outcome's `file` has to name an upload this call
    is holding, from this instance, with the size and digest the upload
    measured; anything else fails that outcome and removes the upload it
    named. The claim writes the outcome's app, source and action on the
    row — the upload only knew what the call's request said. Whatever
    is still pending when the call settles (`replied`, `unacked`,
    `lost`, abandoned with its link) is removed, file then row.
  - **One outcome type in both directions** (`Outcome` replaces
    `RemoteOutcome` and `SettledOutcome`): the node that placed a call
    now hears the **`file`** on its `call_result` too, and fetches it
    with its own bearer (`artifactFile`, which already admitted a node
    party to the call). `Payload` is gone — a `data` is inline or it is
    nothing: past the reply's 256 KiB of gzip the agent fails that
    outcome `too_large` rather than shipping bytes.
  - Schema **V12**: `artifacts.pending`. Rows written before it were
    stored by the reply itself, so they are claimed by definition.
  - A generated agent-only client method (`enroll`, `link`,
    `putCallArtifact`) now declares the route's path arguments before
    throwing: a method's arity is part of the contract.

- **The link is at version 4, and its sub-protocol is `3pm.link`.**
  One flag day, three changes to how the wire is typed and read —
  nothing of what it carries moved. A link-3 agent is refused at the
  upgrade (400: the sub-protocol no longer matches) or at the handshake
  (4426 `ERR_LINK_VERSION`), so **update the nodes first, this backend
  last**, then restart the agents refused during the window.
  - **The handshake is three frames**: `challenge` → `hello` →
    `welcome`. `proof` is gone: `hello.sig` is the Ed25519 signature
    over the challenge's nonce and the node id of the `Authorization`
    header, verified against the key on that row — the header is a
    claim and this is what makes it true. `hello` no longer carries the
    agent's name (a constant) or its node id (the header's, bound by
    the signature).
  - **`welcome.pace`** replaces `metrics_every_s` / `trees_every_s` /
    `state_every_s`, which had three meanings of "absent":
    `{state_s, metrics_s, trees: {mode}}`, every field mandatory, every
    period floored at 1 s on decode. The three flags are unchanged and
    are folded into it in one place (`Config::pace`): `--trees-every-s
    0` is `{"mode":"shape"}`, `n` is `{"mode":"figures","every_s":n}`.
  - **One enum per direction and a two-step decode.** The backend
    matches `AgentFrame` exhaustively (`ws::session::dispatch`,
    `calls::frames`) instead of ignoring a `_` arm, and `decode_agent`
    answers `Inbound`: a tag this build does not know is skipped, a
    **body** it cannot read is skipped and counted — neither ends an
    established session, where a frame body used to close it 4400. Only
    a message that is not a JSON object with a string `t`, or one over
    the cap, still does. **Before** the `hello` the peer has proved
    nothing (the `Authorization` header is a claim over two public
    ids), so there it is stricter: at most 8 such frames are skipped —
    said at `debug!`, counted nowhere — and the ninth closes the
    connection 4400.
  - **A `hello` whose nonce the store no longer holds is refused**
    `ERR_NONCE_INVALID` (4401). The challenge nonce was already taken
    once per connection; its verdict was dropped on the floor, so the
    code the spec documents could not be emitted. A nonce that lapsed
    or was swept while the handshake ran now ends it, whatever the
    signature verified against.
- **`GET /api/v1/info` gains `link: {frames_invalid, crc_full_mismatch}`**
  (`Info.link`, `LinkInfo` in `api.d.ts`): agent frames this backend
  accepted and could not act on, since it started. `frames_invalid`
  counts the bodies above; `crc_full_mismatch` the sessions in which a
  full state section failed its checksum — accepted as before, checks
  off for the session, but now said once at `error!` **with the section
  name** instead of a warning that named none, and counted. Both `0`
  means the fleet and this build agree on the wire.
- **Typed identifiers on the link.** `BlobId`, `CallId` and `Sha256`
  (`threepm-link`) replace the backend's own `blobs::BlobId` and every
  ad-hoc check of a blob name, a call id or a digest: the grammar is
  enforced by the type when the frame is decoded, so a `blob_chunk`
  naming `../etc/x`, a `call_result` with a digest of the wrong shape
  or an `output` that is not 64 hex characters never becomes a frame at
  all. The link's own `LinkInput` and `OutputBlob` replace
  `threepm_protocol::calls::{ActionInput, OutputRef}` where a call
  crosses the link — neither carries the `path` or the `token` that
  belong to the receiving host — and `call_result.results` carries
  `SettledOutcome`, which has no `file` and no blob `data`: what the
  backend cannot deliver cannot be typed. `blobs::BlobId::generate()` →
  `blobs::new_blob_id()`.
- **Enrollment is one `POST /api/v1/enroll`, and a node is no longer
  given a certificate.** The 3-frame WebSocket dialogue
  (`enroll_challenge` → `enroll` → `enrolled`) is gone: an agent posts
  `{cluster_id, token, node_pubkey, hostname, os, arch, agent_version,
  protocol, link_min, link_max, proof}` and reads back `{node_id, state,
  link, server_time_ms}`, or an ordinary `{"error":{code,message}}` with
  the code's status — 400 `ERR_BAD_REQUEST`, 401 `ERR_TOKEN_INVALID` /
  `_EXPIRED` / `_EXHAUSTED` / `ERR_PROOF_INVALID`, 403
  `ERR_NODE_REVOKED`, 426 `ERR_LINK_VERSION`, 429 `ERR_RATE_LIMITED`.
  `proof` no longer carries a server nonce (`3pm-enroll-v2\0<token_id>\0
  <pubkey>`): what bounds a replay is the token's use count, which was
  always true. The checks run in the same order as before and the use is
  still taken in the node row's transaction, so a failure never burns a
  one-use token and two racing hosts never both win the last use.
  `link` is the highest version the request's range shares with the
  server's, and `server_time_ms` is what the agent checks its clock
  against before it writes anything. The per-address budget is taken
  before a byte of the body is read, and the route is capped at 8 KiB.
- **The link upgrade names the node instead of presenting a
  certificate**: `Authorization: 3pm-node <cluster-id>:<node-id>`
  (`threepm_link::LinkAuth`). Both ids are public, so the header is a
  claim, not a credential — the key on the row is what the proof frame
  is verified against, as it already was. A row that is not there is
  401 `ERR_NODE_UNKNOWN` (new code, replacing `ERR_CERT_INVALID`);
  pending and revoked are 403 as before.
- **`nodes.caps` is what the host declared**, not what the backend
  granted: `hello.caps` (the owner's `link.toml` allow-list) is written
  to the row at every handshake, verbatim, and shown in `NodeView.caps`
  as the list it is — an empty one means the node accepts nothing, or
  has not linked since it enrolled. The backend grants nothing, and
  reads nothing into an empty list.
- The link is at **version 3** (`info.link_min = link_max = 3`): an
  agent speaking link 2 is refused, at enrollment with 426 and at the
  handshake with 4426, and must be updated. Nodes enrolled under link 2
  keep their id and key and link again without re-enrolling.

### Removed
- **The per-bucket certificate authority, node certificates and their
  serials.** `crypto/ca.rs`, `crypto/verify.rs` and `ws/issue.rs` are
  gone, with the re-issue in `welcome` (which no longer carries `cert`
  or `accepted_caps`), `--cert-ttl-days` /
  `THREEPM_BACKEND_CERT_TTL_DAYS` (the flag no longer parses),
  `createBucket`'s CA, and the `cert_serial` / `cert_not_after` /
  `cert_expired` fields of `NodeView` — with the dashboard's "Cert
  until" column. Schema **V11** drops `buckets.{ca_pub, ca_key,
  next_serial}` and `nodes.{cert_json, cert_serial, cert_not_before,
  cert_not_after}`; `node_id` and `pubkey` stay, which is all a
  handshake reads.
- The enrollment nonce store (`AppState.enroll_nonces`): there is no
  enrollment nonce any more. `crypto/nonce.rs` stays for the link
  handshake, `NonceLease` with it — the link's challenge nonce is now
  leased for the life of the connection, so an upgrade that never
  answers gives it back when its task ends instead of at the 60 s TTL.
- **A call's output is no longer copied into the artifact store.** The
  answer a target pushes to the call's output blob stays that blob:
  `getBlob` reads it until the relay's retention takes it, and nothing
  of it is kept for thirty days or counted against the bucket's
  gigabyte of artifacts. Gone with the copy: `output_artifact_id` (on
  the outcome, `getCall` and the feed's `call` events), the backend's
  SHA-256 check of those bytes — a well-formed digest that does not
  match is now relayed as the target announced it, for the caller to
  check when it pulls the blob (`output.{bytes,sha256,ext}` on the call,
  unchanged) —, the "too big" and "bucket full" cases, and the
  `artifacts_bucket_bytes` index (schema V9 drops it). An outcome that
  names another blob, a malformed digest or extension, or a second
  claim on the call's one output still fails. Files an action leaves
  and `data` past 256 KiB are still artifacts; `identity` rows already
  stored are still served by `artifactFile` until retention takes them.
  `calls/adopt.rs` → `calls/output.rs` (the reference check only).
- **The four older node-state streams, and link version 1.** `telemetry`,
  `host_patch`, `app_metrics` and `trees` frames are no longer read, nor
  `resync`, `trees_request` and `host_resync` sent: the `state` stream
  (below) is the only way a node's state comes in, and
  `threepm_link::NodeReader` the only patch reader. Gone with them: the
  per-stream readers and gap detectors (`NodeHost::apply` and its size
  bound, `NodeApps::apply`, `NodeTrees::apply`, the telemetry
  heuristics that asked for a full patch or a tree after a 5 s grace,
  the 100-per-frame caps), three of the session's four resync timers,
  `last_full` and the store's fallback on it, the `full` / `merged` /
  `tick` answers of `nodeHost.identity` (always `"patched"` now; the
  field stays), `TelemetrySnapshot.procs_dropped`, and
  `--host-patch-every-s` / `THREEPM_BACKEND_HOST_PATCH_EVERY_S` (the
  flag no longer parses). `--state-every-s 0`, which meant "the older
  streams", is refused at start: `1` at least. The link is at
  **version 2** (`info.link_min = link_max = 2`): an agent older than
  the `state` stream offers link 1 and is refused at the handshake
  with `ERR_LINK_VERSION` (close 4426) until its node runs `3pm update`
  (and `3pm restart agent`); a frame of a removed stream, should one
  come, is an unknown tag — skipped, never an error. The feed's frame
  types (`TelemetrySnapshot`, `FullPolicy`, `strip_static`) are the
  backend's own now (`hub::feed_snapshot`), no longer wire types. An
  instance's metrics and tree leave with their section (`gone`, or the
  agent restarted without it) and — a safety net — with their row of
  the process table, absent two frames in a row behind a daemon that
  answers; a state or a tree refused (caps, label grammar)
  drops what was held for the instance instead of leaving it behind its
  section. `hub/{host,apps,trees,node_state,registry_state}.rs`,
  `ws/{session,resync}.rs`; `tests/it/host_patch_e2e.rs` deleted,
  `app_metrics_e2e`, `trees_e2e`, `host_e2e` ported to `state` frames
  (`tests/it/support/state.rs`), link 1 refused in `dashboard_e2e` and
  `ws::handshake`.
- **Dead link code: `renew` / `cert` frames, `hello.last_seq`,
  `welcome.session`, the TLS pin.** None of it did anything: `renew`
  only counted as a sign of life and `cert` was never sent, `last_seq`
  was read and dropped, the session id was minted and used nowhere (the
  backend numbers its sessions itself), and no pin was ever set — so
  `tls_pin` leaves `GET /api/v1/info` (`Info` in `api.d.ts`) and the
  pasted line carries the bare cluster id, `3pm link 3pmc_<id>
  3pmk_<token> ws(s)://…`; `Advertise::tls` / `url` lose their `served`
  argument (a bare `host:port` is `ws://`). A certificate is renewed by
  `welcome.cert` at the handshake, and only there. A `renew` frame from
  an older agent is an unknown tag: skipped, counted as liveness.
  **Update the agents first**: an agent ≤ 0.5.0 needs `welcome.session`
  to decode the welcome, so it cannot hold a link to this backend.

### Fixed
- **A blob `PUT` refused mid-body answers its status rather than a
  reset.** A refusal found before the body (404, 413 on the declared
  length, 429) already read a bounded part of the rest before answering,
  so the sender could read it; one found while the body was arriving —
  over the blob's length (422), its bucket's quota, given up on (408) —
  answered at once. It now drains like the others: up to 16 MiB or 2 s,
  then `Connection: close`.
- **`viewBlob`'s `?k=` is documented as what it is: reusable for its
  minute, not single-use.** The module said "once"; a `<video>` asks
  for the same URL again for every `Range` it needs, so the token reads
  its one blob as often as it is presented within its sixty seconds, and
  a test now holds that.
- **An artifact upload no longer compresses on the server's async
  workers.** `putCallArtifact` and shell recordings gzipped and hashed
  every piece of the body on the task reading it: up to 256 MiB of gzip
  holding a runtime worker, and every task queued behind it. The bytes
  are now gathered into 256 KiB batches, each hashed and deflated on the
  blocking pool; the stored `.gz`, the digest and the answer are
  unchanged.
- **A blob written a few bytes at a time costs what it holds.** Every
  piece of an HTTP body was kept as a chunk of its own, holding on to
  the buffer it was read into, and a window counts payload bytes: a body
  sent a byte at a time made its 2 MiB window cost ~110 MiB. Writes
  under 1 KiB are now copied onto the window's last chunk while it stays
  under 1 KiB (a `live` stream's only within the same millisecond, so it
  still ages as before), which keeps a window's real cost within a tenth
  of what it counts.
- **A node that placed a call hears its `call_result` while it stays
  linked, even through a slow socket.** A result its session's queue
  had no room for waited five seconds and was dropped: the caller,
  still connected, only learned its call was over from its next
  `welcome` — after its own timeout, or never if it never relinked. The
  result now waits for a slot as long as the session lives (one stuck
  sending ends within `--silence-ms`, and its next `welcome` carries the
  call as before), bounded to 64 results and 8 MiB per caller node and
  64 MiB over all of them (`calls::held`); past that it is left to the
  `welcome`. Agents already handle `call_result`: nothing changes on
  their side.
- **A call placed as its node's session ends is refused, not left
  `sent` for its whole timeout.** A route read the session, stored the
  call, then handed its frame over: when the session ended in between,
  the frame went into a queue nobody read any more and the call, missed
  by the session's abandon, waited for its deadline (a day for a
  session). A session that ends now closes its queue before it leaves
  the hub (`ws::teardown`): a frame handed after that is refused (`409
  ERR_NODE_OFFLINE`, the call `refused`, or `unacked` when the abandon
  got to it first), and one queued before it is settled `unacked`.
- **A reply sent just before the agent closes is no longer settled
  `lost`.** Each `ack` and `reply` is stored by a task of its own, and a
  session that ended abandoned its calls right after reading the
  `Close`: a reply read first, still being claimed, lost to the abandon
  and its call was settled `lost` though the node had answered it. A
  closing session now waits for its call frames' tasks (2 s at most)
  before it abandons anything.
- **A node whose stored `callers` cannot be read is warned about once,
  not on every read.** The row is read on every request that names the
  node, so the warning repeated with each of them; it now names the
  node and is said once per node until the backend restarts.
- **The backend's memory no longer climbs for half an hour under
  load.** A fleet bench (calls, `fleet cp`, streams and shells between
  two nodes) took the backend from 25 to 57 MB over 30 minutes before it
  levelled off. Two bounded caches were filling: each bucket's feed log
  kept its last 4 096 events whatever their size — a node's `telemetry`
  is ~11 KB a second, so ~45 MB for a two-node bucket (16 MB of it in
  the bench's first 11 minutes) — and every relayed answer over 2 MiB
  kept its last 2 MiB in memory for its 15-minute retention, though no
  reader could ever be served from it. The feed log now also stops at
  8 MiB of events per bucket (a page reconnecting past it is told to
  `reset`, as past 4 096 events), and a `file` blob whose first byte is
  gone, or a `pipe` whose reader has been, lets its window go as soon
  as it has settled with nobody reading. The blob sweeper also drops an
  owner's creation instants once they are out of the rate's window.
- **A node's `state` frames are applied on its own lock, not the
  hub's.** The patches, the CRCs, the typed decodes and the telemetry
  rebuilt for a frame held the hub's one mutex, shared by every node,
  feed and route: a node sending large frames slowed everybody down,
  within its work budget. Each node's state and its host, metrics and
  trees now sit behind a lock of their own (`hub::node_side`); the hub's
  is taken only to touch the node before and to publish what the frame
  moved after. Readers (`nodeAppMetrics`, `nodeProcessTree`,
  `nodeHost`, the store's collect) clone the node's pointer under the
  hub's lock and read it after letting go, so the two locks are never
  held together; the hub-wide work budget is a leaf lock of its own.
  Two sessions of a node crossing publish only the later frame, and a
  frame landing on a node revoked meanwhile publishes nothing.
- **LAN direct no longer dials a neighbour through the `lan` overlay.**
  A node running the `lan` module reported its `3pmlan0` address
  (`10.77.x`) among its private addresses, first in interface order, so
  a LAN neighbour's direct leg went into the lan's own fleet sessions
  (4 Mbit/s per peer) instead of the host's network: 0.48 MB/s measured
  between two tablets on one Wi-Fi. `3pmlan` interfaces are now left
  out like container bridges, from the direct candidates and from the
  nodes table's `private` column.
- **`putCallArtifact`: uploads in flight share the bucket's budget, and
  have a clock.** Each upload checked the bucket's stored bytes alone,
  so sixteen arriving at once each fitted the 2 GiB budget on their own.
  An upload now reserves its `content-length` against the bucket, beside
  the stored files and every upload still arriving (413 past it), until
  its row is written; a second upload of the same instance of a call
  while the first arrives is a 409. The body is given up (408, nothing
  kept) at the call's deadline and after 60 s without a byte, so a node
  that opens an upload and goes quiet holds nothing.
- **A call a node placed that expires is told to that node.** The
  sweeper settled it `unacked` / `lost` for the feed only; the node now
  gets its `call_result` (or its next `welcome`), as for a call a closed
  link took — not its own deadline, a day away for a session.
- **Blob disk budget leaks on the relay's error paths.** A blob forgotten
  while its `PUT` still ran (its call abandoned or swept) released its
  file but left its writer going: the next piece the ring dropped opened
  a new file, counted against `--blob-disk-mb` until the process
  restarted. It is now expired, and its writer stops at its next piece
  (408). And a spill whose file could not be created, whose blocking
  task died, or whose run went while the bytes were being appended now
  gives back the budget it had taken for them.
- **Reconnects without a key can no longer park the fleet's agents.**
  Both ids a link upgrade names are public, and a full handshake-nonce
  store answered 429 `ERR_RATE_LIMITED` — which every agent takes as
  "back off for up to 15 minutes". The upgrade now takes a slot of its
  client address before anything else (`--link-pending-per-ip`, 64; an
  IPv6 client by its /64, enrollment's budget too) and gives it back once
  the handshake is over; past it, and when the nonce store is full, the
  answer is **503 `ERR_LINK_BUSY`** (was 429): a code no agent knows, on
  a 503, so every agent retries within its transient backoff (1–60 s).
  Behind a reverse proxy `--trust-proxy` is now required in practice,
  or the whole fleet shares the proxy's 64 slots.
- **One node can no longer hold the hub with its `state` frames.** A
  frame was applied patch by patch under the hub's one lock, so a frame
  repeating a 4 MiB section a thousand times — or a stream of them —
  made the hub walk gigabytes of JSON while every other node, route and
  feed waited. Now only the last patch of each section of a frame
  applies (the copies are counted in `dropped`); each section's size is
  kept exact by re-measuring the one section a patch moved (no more
  re-measure of the whole node, up to 32 MiB a frame); and the work a
  frame causes — the process table rebuilt for every frame, heartbeats
  included, each patch, and each read of a whole section it causes — is
  taken from a per-node budget (3 × 32 MiB a second, a burst of 4×) and
  then from one shared by every node (256 MiB a second). A frame over
  either is refused but for its `gone`s, its sections asked again in
  full, the node warned once a session; an honest node at every cap
  never comes near it. The pending `state_resync` names are capped at
  the sections a node may hold. And a session runs at most 32 `ack` /
  `reply` tasks at once: past them it stops reading its socket until one
  is done (an agent flooding unknown acks waits on itself), and its
  silence clock stops meanwhile, so a slow database never closes an
  honest session.
- **An unreadable `callers` column closes the node instead of opening
  it.** `nodes.callers` was decoded like `scopes`: anything that did not
  parse (a rule with a field this build does not know, a hand-edited
  row) read as the empty list — which for `callers` means *every node
  holding `fleet:invoke` may call every action*, `3pm:shell` included.
  It now reads as the documented deny rule `{"from":"*","app":"-"}`
  alone (`db::access::decode_callers`, a warning in the log): valid, so
  the Access panel still shows it and a `setNodeAccess` of the scopes
  alone writes it back; the owner reopens the node by writing the list.
- **A failing stream call no longer fills its bucket's blob quota.**
  Each `stream: true` call held its two `live` legs' full reservation
  (10 MiB each) until the idle sweep a minute later, so a caller retrying
  a failing stream action in a loop (prod 0.50: the `lan` module, ~1000
  calls) filled the 4 GiB quota and every `3pm fleet ssh` got 429
  `ERR_QUOTA`. Now a stream call answered as a failure (an `err`, or no
  outcome `ok`) or past its deadline gives every open leg back at once —
  charge and creation; a `PUT` still sending ends 408, a reader sees the
  end — and a link that drops gives back only the legs nothing touched
  (a session gone LAN-direct, or still carrying bytes, goes on). A
  finished output leg a caller is still draining keeps its tail: a
  handler that streamed then failed is read to its end. A call that ends
  well keeps its legs as before (`calls/legs.rs`). And an open `live` leg
  is charged what it holds — a 256 KiB floor plus its window and `INIT`,
  in 256 KiB steps — instead of 10 MiB from its opening; each write
  reserves what it may add before landing, atomically on the ledger, and
  a bucket with no room left drops the window's oldest messages first
  (counted as dropped or unwatched, like a full window), then refuses
  the write 429. Opening a blob checks and posts its reservation in the
  same step, so a growing leg cannot take the room it was admitted on.
  A shell's `pipe` legs keep their full reservation (`blobs/blob_charge.rs`,
  `blobs/quota.rs`).
- **A quiet pipe leg no longer cuts its streaming twin.** The sweeper
  gave up a `pipe` leg written once and then idle 60 s, and took its
  twin with it: a `-L` download whose client sent nothing after its
  request was cut at a minute. An idle leg now lives while its twin
  carries bytes (relayed, or reported direct with a counter that
  grows: a direct leg's reports alone, which come every 5 s whatever
  it carries, excuse nothing). Agents ≤ 0.45 that drop
  a forward's keyboard `GET` after its END still lose the tail through
  `reader_left`: no grace (a drained reader and a dead one look alike).
- **A recording's stop reason shows in `getCall`.** `summary` was
  stored as a bare sentence that `getCall` could not parse; it is now
  JSON, and a recording row stored before reads its bare sentence as
  that string (no migration). A node's recording that stopped early sends its reason as a
  code (`x-3pm-stopped`: `cap`, `gap`, `gap_end`), turned into the
  backend's own sentence — never the node's text.
- **`getCall` documents the agent's refusal code** (`result.code`:
  `e2e_required`, `e2e_broken`) kept beside `err.code`; the CLI reads
  it to tell the channel's refusals (exit 255).
- **A node's `agent_version` follows the agent, not its enrollment.**
  The column was written by `POST /api/v1/enroll` and never again, and
  an agent is updated without enrolling again: the bucket page showed
  `0.1.0` for a fleet running 0.23.0. `hello` already carries the
  version (it was only logged); `welcome_frame` now writes it to the
  row when it differs, as it does for `caps`, before the view the feed
  announces is read. A node shows its real version from its next link.
- **`last_seen_at` is the node's last frame, not the start of its
  session.** The row is written when a session attaches and when it
  goes, and the view served that value alone: a node online for two
  hours and sending 300 B/s read "2h ago". The hub's `last_frame_at` is
  now layered on the view (`NodeLive::layer`, so `listNodes`,
  `getBucket` and the feed's `node` events alike), the later of the two
  winning; an offline node, or a backend just restarted, still shows
  the row's value.
- **A blob gives its owner's open slot back when it settles, not
  fifteen minutes later.** The per-owner count of open blobs was only
  decremented when the registry forgot a blob, so a relayed file held
  its slot through its whole retention (`BLOB_TTL_MS`): a caller that
  downloaded forty outputs in a quarter of an hour was refused
  ("too many blobs open on this node") with nothing in flight. The count
  now lives on the blob ledger and drops at `finish` / `expire`, or at
  `forget` for a blob that never settled; it bounds transfers in flight
  only. Bounds raised with it: 100 open blobs per owner (was 40), 200
  opened per minute (was 32).
- **A call's unused output blob gives its reservation back as the call
  settles.** An output blob reserves 256 MiB of its caller's share of
  the bucket's quota (half of `--blob-quota-mb`: 8 outputs under the
  default 4 GiB), and a call whose handler failed before pushing its
  file (`EACCES` on a screen capture) held it until the 60 s idle sweep:
  two dashboard pages on one token, five captures each, was ten
  reservations in a minute and a burst of `ERR_QUOTA`. The blob is now
  forgotten the moment nothing can write it — a reply in which no
  outcome claimed it (an error, inline data only, a refused
  reference), a reply to a call settled otherwise meanwhile, a call
  abandoned with its closed link, and a call past its deadline — and a
  blob that carried nothing gives its creation back to the per-minute
  count as well. A session's `live` return leg is left to its session,
  and a blob someone is reading to the sweeper. `calls/output.rs::release`.
- **A call whose link closed no longer waits out its deadline.** An agent
  aborts the calls of the connection that delivered them and replays
  nothing, but the backend kept their rows `sent` / `acked` until the
  sweeper's deadline — a day for a stream session. After a backend
  restart, or when the target's link fell, the caller's `io.open`
  session never learned its call was over (a camera stream stayed dead
  until its sender was restarted). The calls a closed session carried
  are now settled at once, `unacked` or `lost`, with
  `result.err = {code: "lost", message}`: when the node's session ends
  (up to the moment a newer session of the node attached), and at start
  for every call the previous run left in flight. A linked caller hears
  its `call_result` right away, one that is not finds the call in its
  next `welcome.pending_calls`, and the feed hears the `call` event. A
  reply still being stored (claimed less than 60 s ago) is left to
  settle its call.
- **What a node's `state` stream left behind on the paths no test took**
  (review of the `state` reader, before its first release).
  *A daemon that stopped answering without restarting* made the agent
  send an empty table and — rightly — no `gone`; the backend aged every
  instance out after two such frames, and one whose section never moves
  (an instance that only declares actions, a tree in shape-only mode)
  never came back: the agent believed it delivered. Instances are no
  longer aged on a table that names no daemon, and an instance aged out
  takes its section with it (the next patch is a gap, answered whole).
  *An agent that restarted* never says `gone` for what went away while
  it was down: the sections of its former incarnation stayed in the
  reader for ever — a dead `host` still fed the store, and, 404
  sections later, every new one was refused without a word. A frame of
  another `gen` now forgets what it did not bring again, a 405th
  section is warned once a session. *A `host` section dropped over the
  size bound* left the typed host behind it. *A gap usually cost two
  full sections*: the patches an agent had cut before it read
  `state_resync` noted the section again; one that came in full
  meanwhile is no longer asked (`Resync::settle`). Also: one budget for
  all the sections of a node (32 MiB, next to 4 MiB a section), a
  revoked node keeps its view and none of its state, the process table
  is decoded when it moves instead of at every frame (heartbeats
  included), patches are measured without being written, the reader's
  refusals are what `telemetry.dropped` /
  `threepm_telemetry_dropped_total` count (they read a constant 0),
  `FullPolicy` is one function (`feed_snapshot::full_due`).
  `hub/{node_state,registry_state,apps,trees,registry,host,feed_snapshot}.rs`,
  `ws/{resync,session}.rs`; tests: `hub::node_state`,
  `hub::registry_tests`, `ws::resync`,
  `state_e2e::a_reconnection_keeps_the_node_and_a_restart_leaves_nothing_behind`.
- **A page served elsewhere can run a custom action.** The cross-site
  guard refused every guarded route to a request carrying another
  origin *whatever credential it brought*, so a dashboard on its own
  origin could log in and read metrics but got 403 `ERR_FORBIDDEN` on
  `nodeAction`, `getCall` and each mutation, and needed a server-side
  proxy to place a call. The guard is about an **ambient** credential
  (the session cookie); an `Authorization: Bearer …` is put on the
  request by the page that holds the token, never by the browser, so it
  now opens the guard from any origin on a route behind an account
  (`guard::Opens`, next to the existing `?k=` rule) and `auth` answers
  for the token — 401 for a bad one, never a fallback on the cookie.
  Still refused cross-site: the session cookie, `Basic` (a browser
  replays cached `Basic` credentials by itself), and `register` /
  `login` whatever they carry. `http/guard.rs`,
  `tests/it/cors_e2e.rs::a_bearer_writes_from_another_origin`,
  `tests/it/auth_e2e.rs`.
- Serving a call no longer needs `fleet:invoke`. `getBlob` and `putBlob`
  admit a node that is party to the blob (the call's writer or reader)
  whatever its row grants — membership was already the authorisation
  (`Endpoint::party_admits_a_node`, `auth::node_gate`), and a blob that
  is not the node's still reads as a 404. A target could not read the
  bytes sent to it without also being granted the right to call every
  node of its bucket.
- A blob that carried nothing gives its creation back to its owner's
  per-minute rate when it is forgotten (`blobs::Registry::forget`), so a
  call refused by a rule, by an absent node or by its target does not
  spend one of the thirty-two a minute. One byte written or one reader
  attached and it counts, as before.

### Added
- **Bytes between LAN neighbours.** When a node places a call carrying
  bytes on another node of its bucket and the two linked from the same
  public address, each announced a LAN listener (`hello.direct_port`)
  and each has a private address in its host, both tickets name the
  other end: `peer` (node id, public key, at most eight private
  addresses) on the placing node's answer and on the target's `call`
  frame. The two agents may then move the blob between themselves; the
  blobs stay open here as the fallback. The writer's `path_report` says
  which way it went: `direct` forgets the relay's blob at once (refused
  when the relay already carried a byte of it or has a reader on it),
  `relay` is only recorded; `path` on `getCall` and on the feed's `call`
  events. Per node, `setNodeAccess.direct` = `off` | `files` (default) |
  `all` (live legs too, both ends), shown on the node view; `--direct
  on|off` (`THREEPM_BACKEND_DIRECT`, default on) for the whole backend.
  Schema V10: `nodes.direct`, `calls.path`. An account's call, a call
  without bytes and a node calling itself never get a peer.
- **Live sessions between LAN neighbours.** With `direct: "all"` on both
  nodes, a session's legs go agent to agent like a call's file. On its
  writer's `path_report`, a leg gone direct stays in the relay's registry
  but is served to nobody (`404`) and reserves nothing — no quota, no
  open slot, its creation given back once, however often the report is
  repeated — so its session's twin rules read it like any other leg. Its
  counters and its liveness come from the writer's `blob_stat` frames
  (every 5 s, the latest figure kept, never going back): `live` on the
  call keeps counting, with `live.direct: true`, and the dashboard shows
  "direct · no preview" instead of the Watch button — the relay has no
  picture to serve. Silent for 15 s, the leg is over — an end and an
  abandon read the same — and it ends alone: its twin, if it went by the
  relay and is still writing, runs on (a short question, a long answer);
  one idle or never started ends with it.
- **A call's output blob reserves what the action declared.** The
  target's `output.max_bytes` (`io.action(name, {output: {max_bytes}})`,
  published as `AppAction.o`) is what the output blob declares, and so
  what it holds against its caller's share of the quota — the node's
  published word, never a size the caller sends, which could reserve
  less than the handler writes. Over what the call reaches the largest
  declaration wins; one that names none, a value outside 1..=256 MiB,
  or an action the node never published reserves 256 MiB, as before
  (`calls/output.rs::reservation`, `place.rs::declared_actions` shared
  with `declared_timeout`). The relay holds the handler to it: a `PUT`
  past the declared size is cut. `AppActionView.output_max_bytes`
  (`nodeAppMetrics`, `nodeActions`) shows the bound; the JS client is
  regenerated.
- **One patch reader: the node's whole state arrives as `state`
  frames.** The backend asks its agents (`welcome.state_every_s`,
  `--state-every-s` / `THREEPM_BACKEND_STATE_EVERY_S`, default `1`) for
  one frame a tick carrying what moved in
  the node — host, process table, application metrics, process trees —
  and rebuilds it with one `threepm_link::NodeReader` per node
  (`hub::node_state::NodeState`). A patch off its revision, of another
  incarnation or with a bad checksum is answered `state_resync {secs}`
  naming the section (at most once a second), the rest of the frame
  applied; a section past 4 MiB, or past 404 sections, is dropped and
  whole again at its next patch. What the frame moved goes, typed, to
  the holders every consumer already reads, and every frame — a
  heartbeat included — publishes the `telemetry` the feed always
  carried: the feed, `nodeHost`, `nodeAppMetrics`, the trees, the rings
  and the store keep their shapes. (The streams it replaces are removed
  in this release: see "Removed".)
- **The host arrives as a diff.** A node's host is rebuilt from what
  moved in it — the `host` section of its `state` frames (first
  delivered as a `host_patch` stream of its own with
  `--host-patch-every-s`, removed again in this release: see "Removed")
  — and held typed by
  `hub::host::NodeHost`. The store, the node view's addresses and
  accounts, `nodeHost` and the feed read the rebuilt host; the feed's
  `telemetry` keeps its shape (`host_full` / `host_tick` put in by the
  backend). `nodeHost.identity` is `"patched"`.
- **The feed takes a query** (`GET /api/buckets/{id}/events`,
  `http::feed_query`): `history=0` sends `init.history` empty,
  `match=<text>` keeps the nodes whose hostname contains `<text>` (in
  `init`, and on the feed as `node` events move them in or out),
  `view=live` cuts each `telemetry` frame to the host's gauges and a
  short row per process. Applied to `init`, the replay and the live
  stream after the audience's projection; without a query the feed is
  passed through untouched, and a value it does not know is 400. The
  generated clients take it after the handlers — `events(id, handlers,
  query)`, `follow(id, { query })` — so existing callers read the same,
  and send `history: 0` unless told otherwise. Measured on a bucket of
  nine hosts at one frame a second: 108 KB/s for the whole feed, 14 KB/s
  for `?history=0&match=envision&view=live`, 1.1 KB/s once nginx gzips
  `text/event-stream` (FEATURES.md, « Behind nginx »).
- **Batteries are stored.** `HostMetrics` grew a `batteries` section and
  an `on_ac` flag; both now reach VictoriaMetrics. Twelve gauges per
  pack under a new `bat` label (its sysfs name, `BAT0`):
  `system.battery.charge` (ratio), `.energy` / `.energy_full` /
  `.energy_design` (watt-hours), `.health` — the last two divided,
  **not clamped**, because a fresh pack legitimately reads above 1 —
  `.power` (watts, a magnitude), `.voltage` (volts), `.cycles`,
  `.temperature`, `.present`, and two codes for the two strings:
  `.state` is sysfs `status` as a **nominal** code (0 unknown, 1
  charging, 2 discharging, 3 not charging, 4 full, 5 a status this
  backend has no name for, the same `…_other` idiom as the process and
  TCP state maps) and `.level` is `capacity_level` as an **ordinal**
  (1 critical … 5 full). `system.on_ac` is node-scoped, not a pack's:
  one host, one answer, even with two packs on a dock. Absent stays
  absent throughout — no `status` file writes no state, an `Unknown`
  level writes nothing rather than rank itself below `Critical`, a
  missing sensor writes no temperature, and a full pack drawing
  `Some(0.0)` W writes a real 0. `technology` and `model` are not
  stored: a chemistry and a part number never move, and they are not
  worth the one label slot left under `MAX_LABELS`. New units
  `watt_hours`, `volts` and `state`; the catalog is 231 lines.
  `store/host/rows.rs`, `store/prefix.rs` (batteries are name-keyed and
  evicted like the mounts), `query/catalog/lines/host_rows.rs`,
  `store/host/tests_rows.rs`.
- `bat` is a `Custom` query label, not a built-in: a built-in must also
  be reserved against an application's own labels
  (`every_builtin_label_is_reserved`), and that reserved list mirrors
  the daemon's by hand — reserving a word here alone would have the
  store drop an application label the daemon had accepted.
  `QueryLabel::Custom` now holds a `Cow<'static, str>` so a static
  catalog line can name one in a `const`.
- **`just api` writes a third client: the Node SDK's.** The same route
  table, generated as CommonJS into the sibling checkout
  (`../3pm/packages/node/lib/fleet/api.{js,d.ts}`, published as
  `require('@3pm/api/fleet')`) — and skipped, with a message, when
  there is no `3pm` beside this one. One template feeds both: it marks
  its browser-only blocks (`//{{browser}}` … `//{{/browser}}` — the
  token prompt, the `EventSource` reader) and the Node build cuts them,
  so the browser client and the Node one cannot drift.
  `--check` covers the two new files like the other three.
- **A token prompt in the generated client.** `await connect({ baseUrl
  })` returns a ready `Client`, asking the viewer for an account token
  (`3pmu_…`) only when it has none remembered for this page — so a
  front page of your own, served from anywhere, needs no login code of
  its own. The card is drawn in a shadow root (the host page's CSS
  cannot deform it, nor its own leak out), follows the viewer's
  light/dark preference, and takes Enter to submit, Escape or a click
  outside to dismiss (rejecting with a `TokenPromptCancelled` error).
  Nothing is stored until the backend has accepted the token: the
  `3pmu_` prefix is checked for a useful message, then one `listBuckets`
  call — the cheapest authenticated read, and unguarded, so it answers
  cross-origin — decides. It is remembered under
  `3pm.token:<location.pathname>` (`connect({ namespace })` overrides
  the key), every `localStorage` access wrapped so a private window
  prompts each time rather than breaking. A 401 later on clears the
  token, reopens the card and retries the call that met it;
  `api.forgetToken()` clears it on purpose. `new Client(…)` is
  unchanged and no DOM code runs at import.
- The generated client is **browser-only** and now says so: `connect()`
  throws a message naming the SDK's fleet client in a runtime with no
  `document`. Nothing that worked under Node was removed — the claim of
  Node support was.
- **CORS, open on every route.** The backend sent no `Access-Control-*`
  header at all: a page on another origin could only reach it through a
  proxy that added them. A `CorsLayer` now wraps the whole router
  (`http::cors`, applied outermost in `http::router`): any origin, any
  method, the request headers a preflight asks for, response headers
  exposed with `*`, and a ten-minute preflight cache.
  `Access-Control-Allow-Headers` deliberately mirrors the preflight
  rather than sending `*` — the Fetch spec excludes `Authorization`
  from that wildcard, so `*` would have failed every `Bearer` call
  while looking permitted. No `Access-Control-Allow-Credentials`: it is
  forbidden with a wildcard origin, and the session cookie is
  `SameSite=Strict; HttpOnly`, so a cross-origin caller is bearer-token
  only by design. The cross-site guard is unchanged and still 403s
  `register`, `login` and every mutation carrying another origin —
  because the layer is *above* it, that 403 now carries the CORS header
  and reads as a 403 with `ERR_FORBIDDEN` instead of an opaque CORS
  failure, and a preflight is answered 200 instead of meeting the
  guard or the router's 405. `tests/it/cors_e2e.rs`.
- Two "Link two machines" recipes in `GET /api/v1/routes`: the
  `setNodeAccess` body that lets a node place calls (`scopes`), and the
  one that lets it call an app's actions on another node (`callers`),
  checked by the recipes test like the metrics ones. FEATURES "Node
  calls" shows the same thing with `curl`.
- `NodeView.user` and `NodeView.sessions`: the account a node's daemon
  runs as, and the accounts logged in on it, read off the node's last
  complete host snapshot (`hub::addrs`, beside the addresses) and
  layered onto the view — never stored, and never read from a tick,
  which does not carry `user` at all. A change of either announces the
  node on the feed the way a moved address does. `user` goes out with
  `fleet:list`, beside `peer_ip` and `private_ips`; **`sessions` needs
  `fleet:read`** — who is logged in on someone else's box is the one
  thing on a node view that is about people — and a node without that
  scope gets no key rather than an empty list, on its own row as on a
  neighbour's. The owner reads both. A node whose agent predates the
  fields carries neither. The dashboard's card prints `user …` in the
  line that already carries `public …` and `private …`.
- **Live streams.** `nodeAction` takes `stream: true` and opens a
  session of two `live` blobs, one each way, carrying framed messages
  (`KEY`/`END`/`INIT`, 1 MiB each at most, refused on their header
  rather than buffered). The window forgets whole messages, keeps the
  stream's `INIT` outside it for whoever joins, sends a viewer that
  falls behind on to the newest `KEY`, and never hands anyone half a
  message. `CallView.live` and the feed's `call` events count
  `bytes_in`, `bytes_out` and `dropped_out` from memory (`dropped_out`
  is what a viewer that cannot keep up costs: what the window forgets
  before anyone has connected is not one); a session ends on a minute's
  silence either way — a leg that has not started is excused while the
  other is alive — or after twenty-four hours, and reserves ten
  megabytes of its bucket's quota per leg.
  `CallInputView.bytes` is now optional and `CallInputView.live` says
  which it is.
- `viewBlob` (`POST /api/buckets/{id}/blobs/{blob}/view`): the owner
  mints the sixty-second `?k=` URL an `<img src>` can carry, with
  `{as: "mjpeg"}` for the rendering. `getBlob?as=mjpeg` serves a stream
  as `multipart/x-mixed-replace`, one `image/jpeg` part per message
  behind a fresh sixteen-byte boundary, `nosniff` and a `sandbox` CSP;
  a message that is not a whole JPEG is dropped without ending the
  response. The dashboard's call card shows a session's counters and
  its picture.
- **Node identity outside the WebSocket.** Every `welcome` mints the
  node a bearer for the HTTP routes (`3pma_…`, `Welcome.fleet_token`),
  valid for its session plus a ten-minute grace, kept in memory as
  `sha256(token) → nodes.id` and nowhere else. Fourteen routes accept
  it, each under a fleet scope granted per node by the bucket's owner —
  `fleet:list` (the bucket, its nodes, the feed, the per-server and
  per-app inventories), `fleet:read` (metrics, host, process trees, the
  catalogue of actions), `fleet:invoke` (the call routes). Default:
  nothing. Every request re-reads the node's row, so granting a scope
  and taking one away are both felt on the node's next request, with
  nothing to reconnect.
- `setNodeAccess` (`POST /api/buckets/{id}/nodes/{node}/access`): the
  owner grants the scopes and writes who may place a call on the node
  (`callers`, wildcards allowed except on an action of the `3pm:`
  prefix; each field at most 64 characters, at most 64 rules, unknown
  keys refused so a typo cannot widen a rule). Each list it names is
  replaced; the answer is the node's `NodeView`, which also goes out on
  the feed.
- `NodeView.scopes` and `NodeView.callers`, and `node` on every route of
  `GET /api/v1/routes`, `assets/api.json` and the generated client: the
  scope a node's bearer needs, next to `auth`.
- The dashboard's **Access** panel per node, and a badge per scope in the
  node table.
- Schema V5: `nodes.scopes`, `nodes.callers`, `calls.placed_by`, and an
  index on `calls(state, deadline_at)` — the sweeper scanned every call
  of every bucket once a second.
- Retention on `calls`: 30 days, swept with the artifacts.
- Blob relay: `putBlob` / `getBlob` (`PUT` / `GET
  /api/buckets/{id}/blobs/{blob}`), an in-memory registry, a 2 MiB
  ring per blob spilling into a single file only while a reader is
  behind it, byte-exact tailing, back pressure by window, quotas per
  node, per bucket and for the disk, sixty-second `?k=` URL tokens,
  and the codes `ERR_TIMEOUT` (408), `ERR_CONFLICT` (409),
  `ERR_TOO_LARGE` (413), `ERR_UNPROCESSABLE` (422) and `ERR_QUOTA`
  (429). Schema V6.
- **Node calls.** A node places a call on another node of its bucket
  with its own bearer: `fleet:invoke` on the caller **and** a `callers`
  rule on the target, which are granted apart. `nodeAction` takes
  `input` (`{bytes, sha256?, ext?}`), `output: true` and `from_app`, and
  answers `input.blob` / `output.blob` on the `CallView` — one blob each
  way, authorised by membership of the call and by nothing else, so
  neither application ever speaks to the network. `placed_by` (on
  `CallView`, on the feed's `call` and `artifact` events, in the row)
  reads `node:<3pmn_…>/<app>`, with `-` for a call that named no
  application. A call with an input names its `instance` (422
  otherwise), `deadline_at` allows for the transfer as well as the
  timeout (`transfer_budget_ms`), and `stream: true` is refused until
  the streams land. The target's answer, once its SHA-256 is checked
  here, becomes an artifact stored `encoding: identity`
  (`output_artifact_id` on the outcome, served by `artifactFile` without
  `Content-Encoding`); one too big to keep stays a blob. The caller is
  told on its own link (`call_result`, a bounded send, its scope
  re-read first), and `Welcome.pending_calls` hands it the calls it
  placed that settled while it was away — the last 24 hours, at most
  256, newest first, served by the index `calls_placed`.
- `CallView.result.ok` (the action ran: `res === "triggered"`, never
  whether every instance succeeded) and `CallView.output.{bytes,
  sha256, ext}`, filled in from the outcome that claimed the blob once
  the call is terminal: what lets the side pulling the answer check it
  against a digest instead of trusting the relay. The outcome keeps the
  reference it was given (`results[].output`) next to
  `output_artifact_id`, which the feed now names too.
- Schema V7: `calls_placed` on `calls(bucket_id, placed_by, sent_at)`,
  and `pending_calls` matches a half-open range on `placed_by` rather
  than `instr` so the index serves the filter and the order — every
  `welcome` scanned the bucket's thirty days of calls before, which is
  exactly what a reconnection storm does at once.
- **Node addresses** — the node view gains `peer_ip` (where the last
  link came from, the forwarded client behind `--trust-proxy`) and
  `private_ips` (the private addresses of the node's interfaces from
  its last `host_full`: RFC 1918, CGNAT, IPv6 ULA; container and VM
  bridges left out). A `node` event announces a change of either. The
  dashboard prints them next to each card's hostname (`public … ·
  private …`).
- **Behind a TLS proxy** — `--advertise` takes a URL whose scheme wins
  (`--advertise wss://fleet.example.com` → the pasted line ends with
  `wss://fleet.example.com:443`); a bare `host:port` keeps the old rule.
  An invalid URL stops the start, and outside `--dev` a clear-text
  advertise is warned about (agents refuse it to a remote host without
  `--insecure`). `--trust-proxy <ip>[,<ip>…]` believes those peers'
  `X-Forwarded-For` (rightmost untrusted hop; an unreadable one keeps
  the peer): the per-address enrollment budget and the link logs key on
  the real client instead of the proxy.
- **Accounts: login by token** — no password, no email. `POST
  /api/auth/register` (`register`) creates an account and returns its
  token once (`3pmu_<id>_<secret>`, only the SHA-256 stored) with a
  session cookie; `POST /api/auth/login` trades a token for a session
  (`3pm_session`: `HttpOnly`, `SameSite=Strict`, 30 days sliding,
  `Secure` with `--cookie-secure`), `POST /api/auth/logout` closes it
  and keeps the token; `GET /api/auth/me` (name, tokens without their
  secrets, bucket count), `POST /api/auth/tokens` (another token, shown
  once), `DELETE /api/auth/tokens/{token_id}` (closes the sessions it
  opened; never the last one, 409 `ERR_LAST_TOKEN`). Scripts send the
  token as `Authorization: Bearer` (or Basic, the token as the
  password: `curl -u :$TOKEN`). Every bucket belongs to an account
  (migration V4: `users`, `user_tokens`, `sessions`, `buckets.user_id`)
  and every route under `/api/buckets` wants one: 401
  `ERR_UNAUTHORIZED` without, 404 on another account's bucket (the
  answer a missing bucket gets). `--registration closed|open`
  (`THREEPM_BACKEND_REGISTRATION`): closed by default, the first
  account only (then 403 `ERR_REGISTRATION_CLOSED`), which adopts
  every bucket without an owner; `--dev` means open. The route table
  carries `auth` (`public` / `user`) into the descriptor, the
  generated map and the LLM index. Dashboard: a login card (paste a
  token, or create an account and copy its token), the name and "Log
  out" in the header, "Account tokens" on `/`. The generated client's
  option is `token`; with one, the feed is read through `fetch`
  (`EventSource` sends no header).
- **Remote actions** — `POST /api/buckets/{id}/nodes/{node}/actions`
  (`nodeAction`) runs an action an app declared through its SDK
  (`io.action`), on every instance of the app or on one, under the
  caller's `timeout_ms` or else the timeout the app declared (30 s when
  none); `POST …/nodes/{node}/calls` (`nodeCall`) sends `restart`,
  `stop` or `ping` (an `action` is refused, 400: use `nodeAction`); `GET
  …/calls/{call}` (`getCall`) follows a call from `sent` to `replied` —
  or `unacked`, `refused`, `lost` (new state: acknowledged, never
  answered by the deadline; the sweeper settles both) — with each
  instance's outcome and the handler's JSON. A node offline answers 409,
  a node whose queue is full 503 `ERR_NODE_BUSY` (the call is recorded
  `refused`). The feed announces every step (`call` events: state, ids,
  timings, each instance's ok and error code) and every file
  (`artifact` events: ids and sizes) — never params, results, messages
  or summaries, which only `getCall` and `listArtifacts` serve. Commands
  are now executed: the agent answered `unavailable` until this step.
- **Artifacts** — a result past the call's 256 KiB inline budget, or the
  file an action left (`3pm:profile`: a `.cpuprofile` / `.heapprofile`),
  crosses the link as `blob_chunk` frames, assembled on disk under
  `<data-dir>/tmp` (cleared at startup; a blob at most 64 chunks and
  32 MiB, a session up to 128 blobs open at once — 4 calls in flight ×
  16 instances × data and file — and 256 MiB of them together, so a
  fan-out or several profiles at once land every file), checked (count,
  size, SHA-256) and stored under `--data-dir` (default `<db>.d/`),
  never in SQLite; `GET /api/buckets/{id}/artifacts` (`listArtifacts`,
  `?node=`, `?app=`; the generated client takes them as
  `listArtifacts(clusterId, {node, app})`) lists them with a profile's
  hot frames (`summary`, computed just after the call is replied:
  `null` until it lands, usually within seconds), `GET
  …/artifacts/{artifact}/file`
  (`artifactFile`) streams the gzip with `Content-Encoding: gzip` and
  `Cache-Control: no-store`. Kept 30 days and 1 GB per bucket.
- **Dashboard** — an app that declares actions has them in a row
  right under it in the node card's process table (under its CPU and
  memory, as `3pm ls` lists it): its buttons per source (`CPU profile
  10 s`, `Heap profile 30 s` for `3pm:profile`, with a countdown; a
  params field for the rest). A click opens the **Action results**
  dialog on the call, followed to its end (fetched with `getCall` as
  the feed reports each step, JSON folded, "waiting for the node…"
  until then; the last 10 calls stay). The process table is kept by
  app id across frames, so the row and what is typed in it survive
  the one-second refresh. An "Artifacts" panel per node lists the top
  5 frames, raw and gzip sizes and downloads. The buttons and the
  artifacts panel run as the account that owns the bucket.
- **`GET /api/buckets/{id}/nodes/{node}/actions`** (`nodeActions`) —
  the actions the node's apps declare, per instance and source (name,
  description, timeout), nothing of the metrics: what the action block
  polls every 3 s for every card that is not offline, so the bar no
  longer depends on the "Custom metrics" panel being open.
- **`GET /api/buckets/{id}/nodes/{node}/host`** (`nodeHost`) — the
  host as it is now. A tick blanks the host's identity on the link
  (hostname, OS, kernel, arch, boot time, CPU brand and counts, totals,
  interface names, MACs and IPs, mount points, GPU models: 5 KiB a
  second saved), and only the first frame, an identity change and one
  frame a minute carry it whole (`host_full`); the route puts the last
  `host_full`'s identity back under the last tick's figures
  (`threepm_link::merge_static`, rows paired by position) and says
  where it came from (`identity`: `full`, `merged`, `tick`). 404 before
  the node's first frame.
- **Live JSON** — a button per node in the node table and a `JSON`
  button per app in each card open a dialog with the node's documents
  as the API serves them, refreshed every second: the host now
  (`nodeHost`, what the node button opens on), the last telemetry
  frame as the feed sent it, `nodeAppMetrics`, `nodeProcessTrees`,
  `nodeActions`, or one app composed from the three (its process
  record, its custom metrics, its tree); a selector narrows any of them
  to one app. The document is a collapsible tree (json-formatter-js
  2.5.23, MIT, vendored under `assets/vendor/`, loaded on the page —
  nothing from a CDN): the first level open, `►` to unfold, a hover
  preview, syntax colours, and what the reader unfolded kept across
  the one-second repaints; a **search** box cuts the document down to
  the keys and values holding the text, each with its parent whole
  and the keys above it, unfolded, the hit highlighted; **text** shows
  the raw JSON, a **stop /
  resume** button freezes the document to study it (a change of source
  or app loads that document once and stays frozen), and copy.
- **Events and Metric keys fold and filter** — both lists sit behind
  their title (closed by default, the count next to it) with a filter
  box: words, every one of which must appear in the line (a key's id,
  kind, unit, labels, source and description are matched); the count
  says what is shown of the whole.
- **SDK guide** on the bucket page (`assets/pages/guide.js`): how an
  app sends custom metrics, events and actions through `@3pm/api`
  (`io.gauge` / `counter` / `histogram` / `metric`, `io.event` /
  `notifyError`, `io.action` with a JSON result, parameters, a file
  result, a local-only one), how it is profiled (`3pm profile`, the
  buttons here, the API, opening a `.cpuprofile`), and the `3pm`
  commands to know (`start`, `ls --tree`, `show`, `events -f app`,
  `trigger`, `profile`, `link`) — every example naming this bucket, its
  first online node and its first app, coloured by highlight.js 11.12
  (BSD-3, vendored under `assets/vendor/hljs/` with the `bash`,
  `javascript` and `json` grammars — nothing from a CDN).
- **The action routes are guarded** — `nodeAction`, `nodeCall`,
  `getCall`, `listArtifacts` and `artifactFile` refuse cross-site
  requests, like every mutating route, and answer only the account
  that owns the bucket: they run code on the nodes and serve what it
  produced.
- **`proc:action` capability** — granted by default at enrollment;
  migration V3 grants it to every node still holding the earlier
  default (the three), and `welcome` re-issues the certificate of a
  node whose row and certificate disagree, so the capability is live at
  the next hello, not at the certificate's expiry.
- Flags: `--data-dir` (`THREEPM_BACKEND_DATA_DIR`), `--call-margin-ms`
  (`THREEPM_BACKEND_CALL_MARGIN_MS`, 15 000: over an action's timeout,
  before a call is settled here).
- `nodeAppMetrics`: each source lists the `actions` it exposes
  (`{name, description?, timeout_ms?}`).
- Events store: `kind: action` lines (the node's `proc_action` audit:
  app, instance, source, call, action, origin, ok, err_code, duration;
  a trigger the daemon refused as a whole comes without `err_code` nor,
  sometimes, `source`, and reads `failed`).

### Fixed
- **A call the target's agent refused no longer reaches the caller as a
  success.** An agent past its four commands in flight answers
  `unavailable`; the backend settled that as `replied` with no outcome
  and pushed `call_result` with `err: None`, so the caller read
  `{ok: true, results: []}` while `getCall` said `ok: false`. The error
  now travels in `Link::CallResult.err` and in `getCall`'s `result.err`
  (`calls/answer.rs::error_of`: `unavailable` → `busy`, retryable).
- **A settled call reaches its caller before the dashboard hears of it**,
  and in fewer database trips: `calls/settle.rs` split into `settle` and
  `announce`, claim + row + settle + calling node in one transaction
  (`db/scope_settle.rs::claim_reply`), bucket + node in one trip when a
  node places a call (`Db::scope_and_node`), and an inline result is
  relayed as the agent encoded it instead of decoded and re-encoded.
- **A finished `live` session gives its blobs back at once.** Both legs
  used to stay in the relay for 15 minutes after the session ended
  (`BLOB_TTL_MS`, meant for a `file` answer somebody may still fetch),
  each still counting against the node's 40 open blobs
  (`BLOBS_PER_NODE`) and its bucket reservation: twenty sessions in a
  row and the node read `429 ERR_QUOTA` "too many blobs open on this
  node" for a quarter of an hour. A session is now forgotten on the
  sweeper's next tick once both legs are over and nobody is still
  draining either one (`blobs::sweep::retention`); its `live` counters
  on `getCall` go with it. `file` blobs keep their retention.
- **One `live` stream was capped at about 4 MB/s over a link with
  latency.** Every accepted socket had its send buffer set to 128 KiB,
  to keep a relayed frame's latency at 20 ms instead of 800. Setting
  `SO_SNDBUF` does more than choose a size: it turns Linux's send buffer
  auto-tuning off for the life of the socket, pinning the bytes that may
  be in flight at twice what was asked. 256 KiB over the 60 ms round
  trip of a fleet link is 4.37 MB/s — and 4 MB/s of an offered 5 is what
  a hundred 50 KiB images a second measured under `tc netem delay 30ms`,
  with 18 % of them thrown away by the window behind a queue that stood
  at 200 ms. On Linux the buffer is now the kernel's to size and
  `TCP_NOTSENT_LOWAT` alone keeps the queue shallow, which is what it is
  for: it bounds what is queued **unsent**, not what is in flight.
  Measured end to end under the same `netem`: p50 216 → **61.0 ms** at a
  hundred images a second (p90 61.3, floor 60), 261 frames lost → 10 of
  1 485, and nothing lost at thirty. On the socket, 2 626 560 bytes of
  send buffer where there were 262 144 — up to about 43 MB/s a stream
  where the path permits it, a bound read off the socket rather than
  measured. Where
  there is no such knob (macOS, the BSDs, Windows) `SO_SNDBUF` is
  unchanged and so is the latency it bought. Nothing on the wire, in the
  API or in `--live-window-ms` moves; the agent's own outgoing socket
  carries the same change.
- **A `live` viewer the window ran past was left mid-message.** The
  reader was handed every piece from its position to the newest whole
  message and gave them to the socket one at a time, so between two of
  them its position was wherever a *write* happened to end — the middle
  of a message, as soon as a write is a TCP segment rather than a frame.
  Being sent on to the last `KEY` from there put half a message on the
  wire ahead of a whole one, and the decoder at the other end read four
  bytes of payload as a length: a session died on `frame of 2122219134
  bytes, at most 1048576` and never resynchronised. `?as=mjpeg` was cut
  the same way and is fixed with it: a browser watching a leg was handed
  half a message and its multipart reader lost the stream from there.
  Not seen on a loopback, where one write happens to be one message;
  under 30 ms of delay each way, where one message is thirty segments,
  it is a matter of minutes.

  A stream is now served one **message** at a time
  (`Ring::served_end_from`), so a reader that has drained what it was
  handed stands on a boundary, and the jump is held until then. Two
  things follow. A reader's backlog is now a message (1 MiB at most)
  where it could be a whole window (8 MiB), so a slow viewer costs the
  relay less memory, not more. And a reader is handed one piece per
  message rather than one per write — which on this chain is the same
  thing: an agent sends one body frame per message, so a chunk of the
  `PUT` carries one message and the window already held one per write.
  Measured end to end at 2 KiB and five hundred messages a second,
  latency and CPU do not move (p99 880 µs against 867, process CPU
  2410 ms against 2430, two runs each). The extra piece is only ever
  paid for by a writer that packs several messages into one chunk, and
  it costs that writer's reader 549 ns a message against 13. What the
  window drops and where a viewer rejoins are unchanged, and a 50 KiB
  frame at a hundred a second across two agents still crosses a bare
  loopback at 1.5 ms p99.

### Changed
- **The backend takes 3pm's release version.** `just publish` in
  `3pm-rd` writes it into `Cargo.toml` and into the version the
  generated clients carry (`assets/api.{js,json}`, the Node SDK's
  `lib/fleet/api.js`), so the backend, 3pm, the agent and the SDKs of
  one release share one number (was 0.1.0 whatever the release).
  `/api/v1/routes?version=` answers that number.
- **`fleet:invoke` is enough to place a call.** An empty `callers`
  list on the target used to accept nobody; it now accepts every node of
  the bucket holding `fleet:invoke`, for every action, `3pm:profile`
  included. `callers` only narrows: once it holds a rule, a call goes
  through when a rule admits it (and a `3pm:` action still has to be
  named in full there). Existing nodes with no rule become callable by
  every node holding the scope; `{"from": "*", "app": "-"}` closes one.
- **A node may call itself through the fleet.** `nodeAction` aimed at
  the caller's own node used to answer `422` ("a node does not call
  itself through the fleet"); it is now judged like any other caller
  (`fleet:invoke`, then the target's `callers`), and the bytes loop
  through the backend. Two applications of one host can hold a `live`
  session this way, which the daemon never opens locally
  (`tests/it/real_agent_streams_loopback.rs`: one daemon, one agent).
- `getBlob` is behind the cross-site guard, like every other read of
  what a call produced: a request whose `Origin` or `Sec-Fetch-Site`
  names another site reads 403 — **unless** it carries a `?k=` view
  token, which is a minted credential and not an ambient one, and is
  what an `<img src>` has. A browser tag and a `curl` are unaffected;
  the session cookie driven from a sister host of the same registrable
  domain no longer is.
- A node may hold at most **half** of its bucket's blob quota
  (`--blob-quota-mb`): past its share it reads `429 ERR_QUOTA` and its
  neighbours keep their room, where one blob of `--blob-max-mb` used to
  take the lot. `viewBlob` tokens are capped per bucket (64) as well as
  for the process, so one account's minting no longer expires another's.
- The dashboard's pages carry `Content-Security-Policy: frame-ancestors
  'none'` and every asset carries `X-Content-Type-Options: nosniff`; the
  `events` feed carries `X-Accel-Buffering: no`, which D10 always asked
  of it. A blob that was given up on now **fails** its readers' bodies
  instead of ending them cleanly, so a viewer can tell a cut stream from
  a finished one; and a `PUT` refused before its body is read gets its
  status back rather than a connection reset.
- Two scopes, two answers to the same routes: with `fleet:list` a node
  reads every node's identity and its process table and its own grant;
  `fleet:read` adds the hosts, the daemons, the per-process history and
  the applications' `emit` events. It never reads a neighbour's grant.
  `getBucket`, `listNodes` and the feed (`init`, `node`, `telemetry`)
  are projected by one rule (`http::Audience`); an account reads
  everything, as before.
- A node's own calls only, everywhere they are told: `getCall` answers
  404 and `artifactFile` 404 for a call its node did not place,
  `listArtifacts` leaves them out, and the live feed drops their `call`
  and `artifact` events — the fleet's own state (`init`, `node`,
  `telemetry`, `emit`) goes through as before. The owner of the bucket
  still sees all of it.
- `Caller` carries `account: Option<Account>`: a node is an identity of
  its own, not a stand-in for the owner.
- The route table is assembled from one module per resource
  (`api/surface/table/`), concatenated at compile time — the count is
  asserted during const evaluation, so a group cannot grow unnoticed.
- Three socket options on every accepted connection, the same three the
  agent already set on its own outgoing one: `TCP_NODELAY`, so a small
  last write no longer waits on an ACK (up to 40 ms a hop); `SO_SNDBUF`
  at 128 KiB and, on Linux, `TCP_NOTSENT_LOWAT` at 64 KiB, so the
  kernel's invisible send queue stays shallow and a relayed frame's
  latency is bounded by the link rather than by the backlog. Setting
  `SO_SNDBUF` explicitly also turns Linux's auto-tuning off for that
  socket, which caps one connection at roughly 8 MB/s over a 30 ms
  link: the deliberate trade of D11, now made on both ends of the
  fleet's data path.
- **`BucketDetail.advertise` carries its scheme** — `ws://127.0.0.1:9616`
  or `wss://fleet.example.com:443`, the address the pasted line ends
  with (it was the bare `host:port`); the bucket page shows it as is.
- **Breaking: `--admin-token` is removed** (with
  `THREEPM_BACKEND_ADMIN_TOKEN`, `BucketDetail.admin_token_set` and
  `http::admin::ENFORCE_TOKEN`): accounts replace it, and passing the
  flag now fails at startup instead of leaving an operator believing
  the backend is protected. `http/admin.rs` is `http/guard.rs`, the
  cross-site refusal alone. Reads under `/api/buckets` (lists, the
  feed, metrics) are no longer open: they want the owner's login too.
- `calls.state` admits `lost` (the table is rebuilt by migration V3, rows
  kept).
- **`GET /api/buckets/{id}/metrics/inventory`** (`metricsInventory`):
  what a dashboard or an LLM starts from, in one call, shaped for few
  tokens. `tokens` comes first: `{ claude, o200k }`, what the answer
  costs an LLM under Claude's tokenizer (estimated from its bytes with a
  ratio measured per shape) and OpenAI's `o200k_base` (exact) — the
  same pricing as `GET /api/v1/routes`.
  `metric_groups` are the catalog lines the store holds at least one
  series for over the last two UTC days (a histogram by its `_count`),
  grouped by the labels they carry, with nothing empty or derivable: no
  `source` (the `app.` prefix says it), no `default_time` equal to the
  kind's, no empty unit or description; an application line lists the
  apps whose series carry it (`apps`). `label_values` are what every
  other label of those lines holds (`window`, `mount`, `iface`, `src`,
  an application's own labels…; not `node` / `app` / `instance` / `ns`,
  which `hosts` gives, nor `pid`), at most 50 values each, `more`
  counting the rest. `hosts` are every enrolled node
  with its processes (`{ app, instance, ns? }`, read from `process.up`,
  label values ready for a query's `filters`). The text opens with an
  `about:` line saying how the lists join: every series carries `node`
  = a node_id, and `hosts` names it. It answers TOON-style
  `text/plain` by default — a header per list with its count and
  fields, a row per entry — and the JSON with `?format=json` or an
  `Accept` naming `application/json` (the generated client's);
  `?detail=index` leaves the descriptions out. On a real bucket (3
  hosts, 15 processes, 10 labels' values): 14 873 tokens as JSON, 9 732
  as an index, 12 436 and 7 657 as text (before `label_values` and
  `apps`, 13 282 as JSON against 20 807 for the same content in
  `metricsCatalog`'s shape with a flat process list). The stored names
  and the processes are read side by side
  (`/api/v1/label/__name__/values`, `/api/v1/series`), then each listed
  label's values and one series read of every application series but a
  histogram's buckets (none without an application line), all scoped
  twice and cached a minute; 400 on an
  unknown parameter, 409 without `--metrics-url`, 503 when the engine
  is down.
- **`GET /api/buckets/{id}/metrics/servers`** (`metricsByServer`): the
  inventory by server — which metric is stored on which server, for
  which app, where `metricsInventory` lists the lines of the whole
  bucket at once (a Linux-only line, or an app's line on one host only,
  looked available everywhere). `{ tokens, metric_groups, servers }`:
  `metric_groups` describe the placed lines once, as the inventory
  does (without `apps`); `servers` is keyed by node_id, each `{
  hostname, os, arch, state, metrics?, apps? }` — `metrics` the ids of
  the host's own lines stored there, `apps` keyed by app name with the
  ids its series carry on that server (`process.*`, `subprocess.*`,
  `app.*`), catalog order, nothing empty; a node only the store knows
  comes last, `state: "unknown"`. Present means a series over the last
  24 hours. Text by default (`servers:` then `<app>[<n>]: <ids>`
  lines), JSON when asked, `?detail=index` as the inventory. One
  instant query answers for the bucket, `count by (__name__, node, app)
  (last_over_time({cluster=…}[1d]))` — `last_over_time` because
  `present_over_time` drops the series name — scoped twice, cached a
  minute: 0.1 s for 586 rows on the dev bucket. On it (3 servers, 12
  apps across them): 17 731 Claude tokens as text, 12 530 as an index,
  20 052 and 14 427 as JSON. `MetricsReader` gains `placements`.
- **`GET /api/buckets/{id}/metrics/apps`** (`metricsByApp`): the same
  placements read app first — each app with the servers it has series
  on and, on each, the ids those series are: `{ tokens, metric_groups,
  hosts, apps }`, `apps` keyed by app name (sorted), each keyed by
  node_id in the order of `hosts` (`{ node_id, hostname, os, arch,
  state }`, every enrolled node, then a node only the store knows).
  Only an app's lines (`process.*`, `subprocess.*`, `app.*`) are
  described and listed; a host's own lines are left to
  `metricsByServer`. Text by default (`hosts[n]{…}:` rows, then
  `apps:` with `<node_id>[<n>]: <ids>` lines), JSON when asked,
  `?detail=index`; no new read — the instant query and its cache are
  `metricsByServer`'s. On the dev bucket (11 apps on 3 servers): 4 983
  Claude tokens as text, 4 031 as an index, 5 281 and 4 240 as JSON.
  The bucket page links both (**by server**, **by app**, next to
  **inventory**).
- **Process trees: every process of an app, live and stored.** The
  agent sends each app's process tree (3pm's `3pm ls --tree`) on the
  model of the application metrics — the shape when its membership
  moves, the figures every `--trees-every-s` seconds (1 by default,
  `0` for the shape only; `welcome.trees_every_s` on the wire) — and
  the backend keeps the last one per instance and incarnation. Two
  routes read it: `GET /api/buckets/{id}/nodes/{node}/trees`
  (`nodeProcessTrees`: every instance with a tree, each with its lines
  and the folded rows `3pm ls --tree` draws, computed by the same code,
  plus the `refused` count) and `GET …/procs/{proc}/tree`
  (`nodeProcessTree`: one instance, 404 while the backend holds no
  tree for it — running alone, or before its first tree). The
  dashboard's process table writes `+N` after an app with other
  processes (`proc_count` on the record) — a click opens the card's
  new **Process trees** panel, which polls the node once a second and
  draws the folded tree (chains on one row, twins as `×N`, `⚠` on a
  process `3pm stop` cannot reach), then every process. The store
  writes one series set per member — `threepm_subprocess_cpu_ratio`,
  `_memory_rss_bytes`, `_start_time_seconds`, `_detached` on the app's
  labels plus `proc` and `pid` — and the count as
  `threepm_process_procs`; the catalog lists them (`subprocess.*`,
  `process.procs`) and `proc` / `pid` join the reserved label keys.
  Nothing goes on the feed. An instance the telemetry shows with more
  than one process and no tree for its incarnation after 5 s is asked
  for one (`trees_request`, once a second per node at most, and again
  every 5 s while it stays missing — except one the node has no room
  for, already holding 100 trees: it is not asked for a tree it would
  refuse, and is asked as soon as a place frees up, never more of them
  at once than there are free places, a request in flight holding its
  place for 5 s), and so is one
  running alone while
  its tree is still held — the agent then sends it empty and the
  backend drops it — and one whose held tree is of another incarnation
  past the grace (a restarted daemon numbers its gens from 1 again); a
  frame and a node are held to 100 trees, a tree to 256 lines and
  64-byte names, and a tree for an instance the telemetry does not list
  is refused — each counted; an empty tree takes no place and is never
  refused for the node cap (a full node would otherwise count one per
  new instance at every session start); member names are cleaned again on the way
  in (control characters, bidi overrides, line separators blanked). A
  tree alone over the message cap is left out by the agent and asked
  again through the telemetry, like an application-metrics patch. The
  store writes a member's
  figures only within a minute of the frame that carried them, so
  `--trees-every-s 0` yields a point per member after each shape frame
  rather than a frozen value.
  `ProcessTreesView`, `ProcessTreeView`, `TreeProc` and `CompactRow`
  in the client's types, `proc_count?` on `RedactedRecord`.
- **`GET /api/v1/routes` documents the query routes well enough to
  write one.** A route whose body is a small language of its own now
  carries `details` next to its one-line `doc`: for `queryMetrics`,
  what every field of `range` / `queries` / `formulas` / `options`
  means, which time function goes with which metric kind, what the
  answer holds, and a runnable example body; for `eventsQuery`, the
  same in short. A new top-level `limits` block gives the numeric
  bounds by route name — range, points, series (and their defaults),
  queries, formulas, filters, values, value length, body size —
  straight from `query::validate::Limits`, the constants the validator
  refuses on, so they cannot drift from the code the way the comments
  had. The step floor is not among them on purpose: it is the store's
  resolution, and `step_floor_from` points at `store.interval_ms` of
  `GET /api/v1/info`. `details` is spliced into `assets/api.d.ts` as
  JSDoc (`assets/api.js` keeps the one-liner), and the client example
  the descriptor ships gains a metrics and events section. Two tests
  hold it: `limits` is keyed by routes that take a body, and each
  documented example is parsed and run through the very validation its
  handler applies — an example that rots fails the build.
- **The route descriptor says what it costs to read, and reads well to
  an LLM.** `GET /api/v1/routes` answers text by default — the JSON
  table with `?format=json`, or for an `Accept` naming
  `application/json`, which the generated client sends: an index of
  one TOON-style row per route a caller can use (name, method, path,
  body, answer, admin, doc), the routes that are not for an LLM
  (pages, the client module, the agent's WebSockets) as a short list,
  and how to read more. `?route=<name>` is one route in full: its
  `details`, its bounds, and the TypeScript of its body and answer with
  every type they name, JSDoc included, read from the generated
  `api.d.ts` — the shapes an LLM had no way to see from the table. The
  client example stays in the JSON. Every answer opens with its price,
  `tokens: { claude, o200k }`: `o200k` counted exactly with OpenAI's
  `o200k_base` (`tiktoken-rs`, ranks embedded), `claude` estimated from
  the bytes with a ratio measured per shape through Anthropic's
  `count_tokens` API — Claude counts about 1.5× more than `o200k` here,
  so neither stands in for the other. Priced at a fixed point (the
  field included), once per process; `src/llm.rs` holds the counting,
  the TOON cell and the negotiation for every LLM-facing answer. The
  index also gives `metricsInventory`, `queryMetrics` and `eventsQuery`
  whole, their types once — what a dashboard starts from, in the first
  read — after a `## Recipes` section: a body for each common question
  (the value now, a total over a period, the top N, a percentage by
  formula, a counter's rate, one host by name, one label value, the
  error events) and how to read its answer, then the rules no type says
  (legends carry `node` = a node_id, no value predicate, a bool gauge
  reads per instance, an `app.*` metric may come from several apps).
  Every recipe body is run through its route's validation by a test.
  The table costs 8 497 Claude tokens as JSON, 10 741 as the index
  (1 876 for its lists alone), 750 to 3 900 per route in full.

- **Every numeric field of `HostMetrics` is persisted.** The store used
  to write 14 host series out of the ~370 numbers a host agent
  collects; it now writes all of them, and the catalog grew from 49
  entries to 212. New families, one module each under `store/host/`:
  the CPU per logical core and by state (`core` label, the state in the
  name), the kernel counters, the `/proc/meminfo` decomposition, the
  page-fault counters, the VFS cache, the entropy pool, macOS's memory
  compressor, PSI (`window` = 10/60/300 seconds, as the load averages
  use it in minutes), processes and TCP connections by state, the TCP
  and socket counters, conntrack, file handles, the daemon's cgroup,
  the NTP offset, then one series set per network interface (`iface`),
  block device (`blockdev`), temperature sensor (`sensor`), GPU (`gpu`,
  by index) and RAID array (`array`), and how fresh each sampling tier
  is. Six built-in query labels come with it — `core`, `iface`,
  `blockdev`, `sensor`, `gpu`, `array` — reserved in 3pm first, so an
  application cannot claim one and lose its own series.

  Nothing changed on the wire: every number already rode each tick.
  What a tick blanks (`strip_static`) is read from the last full
  snapshot alone, and a test holds it — a stripped tick must write
  exactly what its full snapshot writes.

  The per-interface counters are new names
  (`threepm_host_interface_*`); the summed `threepm_host_network_*`
  pair is untouched, so no existing chart moves. Mounts are no longer
  capped at the eight largest and no longer filtered on `removable` /
  `read_only`: `system.disk.*` now covers every mount point, so a
  `sum by (node)` over it will read higher than before.

  `max_series` goes from 100/500 to 200/2 000 and the label-values cap
  from 1 000 to 5 000 — a chart of one host family across a fleet needs
  the room — and the writer keeps four requests in flight instead of
  two, since a node now fills several times more of a body.
- **3pm's own metrics, persisted.** A node's `HostMetrics.3pmDaemon`
  (what the host agent measures of the 3pm daemon and of itself, 3pm
  `dfc5ada`) is written as 27 series on the node's labels:
  `threepm_daemon_{cpu_ratio,memory_rss_bytes,open_fds,threads}` and
  `threepm_host_agent_{…}` for the two processes, and the daemon's
  counters — apps by status, restarts, timers, bus subscribers, log
  sinks/rotations/bytes, `RLIMIT_NOFILE`, handoffs, agent respawns.
  The role and the status are in the name, not in a label, so nothing
  competes with an application's own `role`/`status`. Catalog ids
  `daemon.*` / `host_agent.*`, in `GET …/metrics/catalog` and the
  dashboard's table. A tick the daemon did not answer (mid-`3pm update`)
  skips the counters, never the process rows. The JavaScript client's
  `HostMetrics` type gains the `3pmDaemon` section.

### Internal
- The integration tests are one binary, `tests/it` (a module per former
  file, `support` compiled once), instead of forty-three: a change to
  the library relinks one test binary. `just test` and `just e2e` run
  under `cargo nextest` (`.config/nextest.toml`, the real-agent tests
  four at a time), and `just test` takes a filter
  (`just test 'test(/^auth_e2e::/)'`). Dev builds keep line tables only
  and build dependencies without debug info at `opt-level = 1`.
- Fleet RPC, simplification pass, no behaviour change: the two tests of
  a parked writer share one mount, the two linked nodes of the test
  support read a `call` frame through one function, and the unread
  `Registry::disk_crossings` is gone (the counter it wrapped is still
  read through `counters.crossings()`).

### Changed
- **The process series are those of the app's whole process tree.** A
  node running a 3pm with process trees reports an app's `cpu_percent`
  and `mem_bytes` as the sums over the spawned pid and every descendant
  (an `npm run dev` that starts Electron and its helpers goes from
  `0% / 71mb` to `62% / 3.0gb`); the agent relays them as before, so
  `threepm_process_cpu_ratio` and `threepm_process_memory_rss_bytes`,
  the fleet table, the history rings and every chart on them carry the
  tree's totals without a change here. Memory is the members' RSS
  summed, an upper bound when they share pages. Only the catalog says
  so: the two descriptions (`GET …/metrics/catalog`, the **Metric keys**
  table) name the tree instead of "one process". A node on an older 3pm
  keeps reporting the spawned pid alone; the two look the same on the
  wire.

### Fixed
- The five routes still documented as needing `--admin-token`
  (`nodeAction`, `nodeCall`, `getCall`, `listArtifacts`, `artifactFile`),
  the same claim in the feed's own documentation (which also said the
  feed needed no token at all), and the last mention of the flag in the
  dashboard: it was removed with the accounts.
- `options.max_series` was documented as "≤ 500 per result, default
  100" in `assets/api.d.ts` and in `query/mod.rs`; it has been 1–2 000
  with a default of 200 since the host-metrics work. Both now say so,
  and `limits.queryMetrics` serves the numbers from the validator
  itself.
- The step arithmetic the new `details` and the `.d.ts` quote said "30
  days fits at `30m`" and "`5m` fits about 7 days" — true only at
  `max_points: 2000`, not at the default 600, where the same 30 days
  draws at `2h` and `5m` reaches two days (and 7 days at `5m` is 2 018
  points, past 2 000 as well). Corrected, and pinned by
  `the_documented_step_arithmetic_is_what_the_validator_does`: prose is
  the one part of the descriptor no generator checks.
- A query on an application counter compiled the series without the
  `_total` the store appends (`rate(threepm_app_x{…})` found nothing);
  the dynamic catalog now names a counter `threepm_app_<name>_total`,
  and a row loaded with the old name is rewritten at the next flush.
  Found on the SAW rig (plan §8): `app.host.processed_callbacks`.

### Added
- The bucket page ends with **Metric keys**: every key the bucket
  holds from `metricsCatalog` — kind, unit, default time function,
  labels and origin (`static` / `app`) — applications first, counted
  next to the title, asked again every 30 s while the tab is visible.
  Rendering and escaping in `tests/dashboard_render.rs`.
- `tests/real_agent.rs` gains the chain for real (plan §10): an
  application on the Node SDK (`@3pm/api` from `../3pm/packages/node`)
  under a real `3pm`, the real agent linked, this backend writing to a
  fake engine — the labelled counter and the histogram's `vmrange`
  buckets in the store's lines, the catalog, `nodeAppMetrics`, a
  quantile query explained, the event in the store and back through
  `eventsQuery`. Run by `just e2e`; skipped without `node`.
- Application metrics (step D of
  `3pm-app-metrics-plan.md` in `~/3pm-rd-doc-2026-09-09.tgz`): the
  agent's `app_metrics` patches are applied per instance in the hub
  (`threepm_link::patch::apply`, keyed by process id and incarnation),
  a patch off its base or a telemetry record ahead of the state for 5 s
  is answered with `resync` (coalesced, one frame a second per node);
  the store writes `threepm_app_<name>` / `_total` /
  `_bucket{vmrange}` + `_count` + `_sum` with `app`, `instance`, `ns`,
  `src` and the app's labels re-validated; `GET
  /api/buckets/{id}/nodes/{node}/app-metrics` (`nodeAppMetrics`)
  serves the live state; the catalog gains the bucket's `app.<name>`
  lines (learnt at each flush, persisted in the new `app_metrics`
  table — schema V2 — 5 000 per bucket, purged after 30 days) with
  `MetricKind::Histogram`, custom labels, `time.fn: "quantile"`
  (`histogram_quantile` over `sum(rate(_bucket)) by (vmrange, …)`),
  `source` on every catalog line, and `options.explain` →
  `results[].expr`. `welcome.metrics_every_s` from
  `--app-metrics-every-s` (1). `info.store.catalog_over_cap`.
- Application events (`AgentEvent::App`) become log lines: `kind:
  "app"`, `_time` = the app's clock (bounded to ± 5 min of the
  backend's), `received_at`, `app`, `instance`, `src`, `event_name`,
  `level`, `severity_number`, the attributes flat under `a.`,
  `exception.type` / `message` / `stacktrace`, `fingerprint`, `repeat`.
- `POST /api/buckets/{id}/events/query` (`eventsQuery`): the persisted
  events, newest first — a range, exact filters, a limit — compiled to
  LogsQL in the store (`store/victoria/logsql.rs`), read on
  `/select/logsql/query`; 409 without `--events-url`. The writer's
  event buffer is bounded (20 000, drops counted); a node may emit 100
  application events a second (burst 500), the feed forwards 50 a
  second per bucket (`info.feed_app_events_dropped`).
- Dashboard: a "Custom metrics" panel per node card (polls
  `nodeAppMetrics` every second while open) and the "Events" list
  completed from `eventsQuery`; application events described with
  their level, name, repeat and exception. `tests/dashboard_render.rs`
  runs the panels under `node` with `<img onerror>` in every app
  string.

### Changed
- `validate` takes the bucket's catalog (`&dyn Lookup`); `SpecPlan.metric`
  is owned; `QueryLabel` is no longer `Copy` (`Src`, `Custom(name)`);
  `MetricsReader` gains `explain`; `SampleSink` takes a `&str` name and
  a `text` method for blocks rendered with `vmrange`; `store::spawn`
  takes the catalog's `Persist`; `Hub::telemetry` returns the instances
  to resync.

- The metrics query is cached and coalesced: the engine's answer to
  each spec is kept for a step (5 s to 60 s) under `--store-query-cache-mb`
  (64; `0` turns it off), keyed by the request itself, so the same chart
  refreshed within a step by any number of viewers is one request to the
  engine — and identical requests in flight wait for the first one's
  answer instead of leaving too (`src/store/cache.rs`). Label values are
  cached a minute. `GET /api/v1/info` → `store.cache` reports entries,
  bytes, hits, misses, coalesced.
- Every JSON route is gzipped when the client asks (`Accept-Encoding:
  gzip`; `tower-http`, fastest level): a full chart answer, tens of MB
  of numbers, shrinks about tenfold. The feed and the sockets never are.
- Charts read the persisted metrics through `POST
  /api/buckets/{id}/metrics/query` (`queryMetrics(clusterId, body)` in
  the client): a JSON query — a range and a step, queries naming a
  catalog metric with filters, a time function and an aggregation,
  formulas over their refs (`A / B * 100`), options (`max_points`,
  `max_series`, `sort`, `fill`, `mode: summary`) — validated against
  the catalog (unknown field, unknown metric, a label the metric lacks,
  `rate` on a gauge, a bad regex, a range past 31 days… are 400
  `ERR_BAD_REQUEST` naming the field) and compiled in the backend,
  scoped to the bucket twice (a matcher and `extra_label`), the engine
  asked for the `max_series + 1` best series by `sort` over the whole
  range; the answer is one `times[]` and numeric `values[][]` per
  series with a summary. `step` is a duration string or milliseconds.
  `GET …/metrics/catalog` lists what a query may name (OpenTelemetry-
  style ids, kind, unit, labels, default time function);
  `GET …/metrics/labels/{label}/values` the values a label takes.
  `--store-query-concurrency` (8) bounds the chart queries in flight,
  whatever the number of engine requests each fans out to; 409
  `ERR_STORE_DISABLED` without `--metrics-url`, 503
  `ERR_STORE_UNAVAILABLE` when the engine is down, busy or slow. The
  engine's query language never crosses the API: `src/query/` is
  engine-agnostic (a test holds it) and `src/store/victoria/` compiles
  the plan.
- The telemetry is persisted: `--metrics-url` (VictoriaMetrics) and
  `--events-url` (VictoriaLogs) name the engines, `--store-interval-ms`
  (5 s) the stored resolution. Every interval the store takes the last
  frame of every node out of the hub (one lock, pointer clones —
  nothing runs per frame) and writes one sample per series as gzipped
  Prometheus text: `threepm_process_*` (cpu ratio, rss, up, restarts,
  start time) with `cluster`, `node`, `app`, `instance`, `ns`;
  `threepm_host_*` (cpu, load, memory, swap, processes, uptime,
  network counters summed over the physical interfaces, the largest
  mounts); `threepm_node_info` / `threepm_node_up`. Process lifecycle
  events go to the logs engine as NDJSON. Retries 1 s / 2 s / 4 s on
  429, 5xx, timeouts and connection errors; two flushes queued at
  most, the oldest dropped whole; a hung engine never holds the
  shutdown. Without the flags nothing changes. `GET /api/v1/info.store`
  shows the counters (`sent`, `dropped`, `errors`, `retries`,
  `last_error_kind`, `last_flush_at`).
- A JavaScript client for the whole API, generated from the backend's
  own route table. `src/api/surface.rs` declares every route once
  (`name`, method, path, `kind`, `guarded`, argument names, body and
  response types, doc); `http::router` mounts exactly that table, and
  `just api` writes `assets/api.js` (one ES module: a method per
  endpoint, `ApiError` carrying the backend's `ERR_*`, and a
  `Subscription` that reconnects on its own and turns a `reset` into a
  fresh `init`), `assets/api.d.ts` and `assets/api.json`. Same module
  in a browser and in Node — `new Client({baseUrl, adminToken})`, with
  `fetch` / `EventSource` taken from the global scope or passed in.
  `just check` fails when what is committed no longer matches the
  table.
- `GET /api/v1/routes[?version=]`: that table as data, live. `version`
  defaults to `latest`; `latest` and the backend's own version answer
  the same table, any other is a 404 naming the version this build
  does serve. `Client.discover()` reads it and follows that backend's
  paths from then on, so a page built against one version keeps
  working against another. The document also carries `client {url,
  types, descriptor, example}` — where the module and its types are
  (relative, so they resolve behind any proxy) and a runnable snippet,
  so fetching this one document is enough to get started.
- `GET /api/v1/client.js`: the client under the version of the API it
  calls — the URL to import from elsewhere. `/assets/api.js` serves
  the same bytes; `.d.ts` files are now served as `text/plain` instead
  of `application/octet-stream`.
- Both dashboard pages carry an `API routes` / `JS client` nav.
- The client tells a page where its live data is. `Client.follow(
  clusterId, {signal, onError})` is the SSE feed as an async iterable
  of `[kind, payload]` (leaving the loop or aborting the signal closes
  it; a `reset` is an item followed by a fresh `init`), over a generic
  `Client.iterate(name, args, options)`. Generated like the rest: an
  `sse` entry of the table names its iterator twin (`iter`), and the
  twin takes the feed's own arguments. `docs/examples/follow.mjs`.
- The client knows which backend it was generated from: `VERSION` is
  exported, `discover()` compares it with the answering backend's
  `version` and reports a difference to `onVersionMismatch(backend,
  client, api)` — a `console.warn` by default, `null` to silence, any
  function to show it (the dashboard appends `client generated for
  <version>` to its header). `client.versionMismatch` says it after
  the fact. The client follows the backend's table either way.
- Requests can be abandoned: every generated method, `call()` and
  `discover()` take `options` last, and `options.signal` (an
  `AbortSignal`) cancels the request — it rejects with `fetch`'s
  `AbortError`, not an `ApiError`. `request()` takes the same
  `{signal, retry}` in place of its bare `retry` flag.
- The two agent WebSockets (`enroll`, `link`) say what they are:
  their generated methods throw an `AgentOnlyError` naming the
  sub-protocol and the feed to read instead, and their route docs
  point at `events`.
- The descriptor groups the live surface: `realtime {feed: {route,
  transport: "sse", eventKinds, iter}, agent: {routes, transport:
  "websocket", subprotocol, min, max, specPath}}` points into `routes`
  by name (the table itself stays whole), replacing the top-level
  `link` and `eventKinds`. `specPath` is a workspace path, not a URL —
  the backend serves no copy of the protocol doc. The client exports
  the same block as `REALTIME`, and `LINK` is its agent half.
- The two ways around the route table are closed:
  `no_route_is_mounted_outside_the_table` refuses a route mounted by
  hand in `http::router` (it would be absent from `/api/v1/routes` and
  from the client, and no other test would see it — they all walk the
  table), and `every_type_the_table_names_is_declared` refuses a
  `returns` / `body` naming a type `types.tmpl.d.ts` does not declare
  (nothing here compiles TypeScript, so a typo would otherwise only
  break in a consumer's editor). The rule itself moved into
  `CLAUDE.md`, next to the other non-negotiable ones.
- `tests/api_surface.rs` holds the table to the running server: every
  entry is mounted on the method it declares (an undeclared verb
  answers 405), every `guarded` entry answers 401 without the admin
  token and no unguarded one does, `/api/v1/routes` matches
  `assets/api.json`, the generated files are current, and every field
  the API serializes is declared in `api.d.ts`.

### Changed
- The store renders the label block of a series once, at first sight,
  and keeps it across flushes (`src/store/prefix.rs`: per node, per
  process, per mount, per load window; unseen for twelve flushes, it
  goes) — a flush is one `push_str` per line instead of a sorted label
  set per process; and the bodies of a flush are gzipped side by side
  when there are several. `just bench`, 10 000 nodes: render 146 → 68 ms,
  render + gzip 193 → 84 ms.
- Release builds use one codegen unit and thin LTO: the hot loops (the
  store's render, the engine's answer parsed) swung by 20 % between
  unrelated changes with the default partitioning; pinned, and the
  benchmarks measure the code rather than the luck.
- The engine's `query_range` answer is parsed borrowed: label names and
  values point into the body, the points go straight to numbers through
  a visitor (`store/victoria/points.rs`) — no string per point where a
  full answer (500 series × 2 000 points) used to allocate a million.
  `parse_matrix` on that answer: 51 ms → 33 ms (`just bench`,
  `query/parse_matrix`); the rest is serde_json reading 29 MB of JSON.
- A body the JSON layer refuses on `POST /api/buckets` and `POST
  /api/buckets/{id}/tokens` is a 400 `ERR_BAD_REQUEST` in the API's
  error shape (the one sentence axum has about it), not axum's own 422
  text — the same answer the metrics query already gave. An absent body
  on `/tokens` is still the defaults.
- A time function's `window` takes milliseconds as well as `1m`, like
  `range.step`. A series' `summary` (what `sort` ranks by) is over the
  points the engine returned, before `fill`, so the `max_series` cut
  here is exactly the engine's; `count` is those points.
- The store's shutdown counts every sample: the sends in flight finish
  within the 2 s budget while the queue and one last collect go out with
  one attempt each; whatever the budget cuts off is counted `dropped`
  (before, an abandoned request's samples were in neither `sent` nor
  `dropped`). The writer's freshness is a flag per node, set by a
  `telemetry` frame and cleared by the collect, instead of comparing
  the frame's arrival time with the writer's own clock — a frame
  recorded between the two readings can no longer be missed.
- The link logs name the agent's address: `node <id> (<hostname>)
  linked from <ip>:<port>: <agent> <version> boot <boot_id>` when a
  session opens, the same `<ip>:<port>` on every line that ends one
  (left, silence, refusal, ordered close), and `link refused from
  <ip>:<port>` when the certificate never gets past the upgrade.
- The dashboard is written on the generated client and driven by the
  route table it serves. `assets/app.js` imports the client from
  `/api/v1/client.js`, calls `discover()` (`GET /api/v1/routes`) and
  hands the client to the page; from then on every path comes from
  that table by name — bucket links (`api.url("bucketPage", id)`), the
  bucket page's own argument read back from the `bucketPage` template
  (`pathArgs`), the feed opened under the name `realtime.feed.route`
  gives, the `API routes` / `JS client` / `types` links re-pointed at
  `client.{descriptor,url,types}`. No URL and no `fetch` of its own;
  both pages load it with `<script type="module">`. Split into
  `ui.js` (an escaping `html` template tag, the formatters, `reason`,
  `pathArgs`), `pages/index.js`, `pages/bucket.js` (the feed, tokens,
  approve / revoke, the event log) and `pages/fleet.js` (`Fleet`: the
  nodes' rows and cards, patched one node at a time). A `reset` no
  longer reloads the page — the client reopens the feed and redraws
  from the new `init`. The header names the backend that answered
  (`3pm-backend <version>`, from the descriptor) on both pages.
- The dashboard is dark: `assets/style.css` keeps the same tokens with
  a near-black ground, `color-scheme: dark`, and status colours
  lightened to stay legible on it.

### Added
- Per-node receive traffic. Every WebSocket message from an agent is
  counted in payload bytes as it arrives (deflated frames, pings and
  pongs; before inflation): `NodeView` gains `rx_bps` — the average
  over the last 5 complete seconds — and `rx_bytes`, the total since
  the backend started (across sessions of the node). The sweeper rolls
  the rate once a second and publishes a `node` event when it moves;
  `GET /api/buckets/{id}`, `GET …/nodes` and the SSE `init` carry both.
  The bucket page shows an `Rx/s` column in the nodes table (the total
  on hover) and `rx …/s` on each node card.

### Changed
- Extracted from the 3pm repository (`3pm-backend/` there, history
  kept) into its own repository. The shared crates are path
  dependencies on a sibling checkout (`../3pm/crates/…`); `just
  backend` / `just backend-dev` became `just check` + `just test` /
  `just dev`, and the real-agent test runs with `just e2e`.

### Added (while inside the 3pm repository)
- Skeleton, SQLite layer, bucket CA, tokens, nonces, `GET
  /api/v1/enroll`, buckets/tokens API with the pasted line.
- Ingestion: `GET /api/v1/link` admission, handshake machine, session
  task, hub (registry, presence `online → stale → offline`, rings,
  per-bucket event log, sweeper), SSE feed, `GET /api/buckets/{id}`.
- First dashboard (embedded `assets/`), node API (`GET /nodes`, `POST
  /approve`, `POST /revoke`), `--admin-token` and the cross-site guard,
  enrollment rate limits, header timeout, 1 MiB frame caps.

</document>

<document url="https://3pm.edgecr4ft.net/docs/internals/hostagent-tiers.md" title="Host metrics tiers (design)">

# Host metrics — tiered sampling (audit + proposed architecture)

Status: **implemented** (`hostmetrics/{tiers,fast,medium,slow,top}.rs`,
`hostmetrics/extended/`, `hostagent/sampler.rs`). §1 is the audit that motivated the split; §2–3
describe the shipped design (the *as-built* notes flag where it departs
from the original proposal). Companion to `hostagent.md` and
`metrics.md`.

## 1. Audit of `collect()` — the shape before the split

Historical: the whole-snapshot `collect()` this section measured is gone
from `hostmetrics.rs`, replaced by the tier collectors of §3. Back then
one `collect()` call gathered everything, every 10 s, on one blocking
thread. Its wall time (~200–700 ms) was dominated by two *waits*, not by
work, and the cheapest gauges (CPU %, RAM) were refreshed no faster than
the most expensive probe (NTP round trip, `nvidia-smi`).

Costs below are per call, measured on this M4 Max (macOS 15) for the
subprocesses and estimated from the syscalls involved elsewhere.
"Volatility" is how fast the value meaningfully changes.

| # | Collector (code) | What it feeds | Cost / hazard | Volatility |
|---|---|---|---|---|
| 1 | `sleep(MINIMUM_CPU_UPDATE_INTERVAL)` + `refresh_cpu_usage` | `global_cpu_percent`, `per_core_percent` | **200 ms fixed wait** — CPU% is a delta and `System` is rebuilt each call, so the warmup window is paid every time | high |
| 2 | `start/finish_cpu_sampling` (`/proc/stat` ×2, `systemstat` on macOS) | `cpu_states`, `kernel` | 2 × <1 ms reads, but rides on wait #1 | high |
| 3 | `sysinfo::System::new_all()` | `processes_count`, `process_states`, `cpu_brand`, cores, `mem_*`, `swap_*`, `cgroup_limits` | **Enumerates every process** (`/proc/<pid>/stat` walk, `proc_pidinfo` on macOS): 20–300 ms depending on process count; the only reason it is here is the process-state histogram | mem: high; processes: medium |
| 4 | `query_ntp_offset_ms` (thread) | `ntp_offset_ms` | UDP to `pool.ntp.org`, **up to 500 ms**, network egress every 10 s from every host of a fleet | very low (drift is minutes-scale) |
| 5 | `gpu::read_gpus` (thread) | `gpus` | `nvidia-smi` subprocess 20–100 ms (2 s deadline; also keeps an idle NVIDIA GPU out of its low-power state), `ioreg` ~30 ms, DRM sysfs <1 ms | gauges: medium; identity: static |
| 6 | `Disks::new_with_refreshed_list` + `statvfs` per mount | `disks` (space, inodes, read/write totals) | `getfsstat`/`/proc/mounts` + one `statvfs` per mount, ~1–5 ms; **a stalled NFS/FUSE mount blocks the whole collect** | space: low; I/O counters: high |
| 7 | `Networks::new_with_refreshed_list` | `networks` (counters, mac, ips) | `getifaddrs` + `/proc/net/dev`, ~1 ms | counters: high; mac/ips: static |
| 8 | `Components::new_with_refreshed_list` | `sensors` | `/sys/class/hwmon` walk (dozens of files) or SMC/IOKit on macOS, 5–50 ms | low |
| 9 | Linux `/proc` one-file readers: `stat`, `vmstat`, `meminfo`, `net/snmp`, `net/netstat`, `sys/fs/file-nr`, `pressure/*`, `net/sockstat`, `sys/net/netfilter/*`, `sys/fs/{dentry-state,inode-nr}`, `random/entropy_avail`, cgroup files | `paging`, `mem_detail`, `tcp_counters`, `file_handles`, `pressure`, `sockets`, `conntrack`, `vfs_cache`, `entropy_avail`, `cgroup` | <1 ms each, ~5 ms total | high |
| 10 | `read_tcp_states` (`/proc/net/tcp{,6}` full walk) / `listeners::get_all()` on macOS | `tcp_states` | **O(sockets)**: a busy server with 100 k connections is a multi-MB read under a kernel lock, 20–200 ms; macOS `listeners` enumerates every pid's fds | medium |
| 11 | `read_raid` (`/proc/mdstat`) | `raid` | <1 ms | very low |
| 12 | `block_devices_of` (`/proc/diskstats`) | `block_devices` | <1 ms | high |
| 13 | macOS `vm_stat`, `netstat -s -p tcp` subprocesses | `mem_compressor`, `tcp_counters` | fork+exec, ~15–20 ms each | medium |
| 14 | `System::host_name/name/long_os_version/kernel_version/cpu_arch/boot_time` | identity fields | `uname`, `/etc/os-release`, `sw_vers` plist parse: ~1–10 ms | **never changes** |

Findings:

- **F1 — the cheap, volatile core is hostage to the slow tail.** CPU %,
  load, RAM, PSI, NIC/disk counters cost ~5 ms combined but are only
  refreshed every 10 s because the same tick also waits for NTP,
  `nvidia-smi` and a process walk. Lowering `THREEPM_HOSTMETRICS_INTERVAL_MS`
  today drags the expensive probes down with it (the doc even warns that
  below ~1 s "the collection itself dominates").
- **F2 — the 200 ms CPU warmup is an artefact of statelessness.** A
  periodic sampler already *has* a window: the tick. Keeping the
  `sysinfo::System` / previous `CpuTicks` across ticks removes the sleep
  entirely (same trick `rates.rs` already applies to byte counters).
- **F3 — throughput resolution is 10 s.** `rx_bytes_per_s` on the `ls`
  line is an average over the last 10 s; pm2 users expect ~1 s.
- **F4 — a hung collector stalls every metric.** One deadlocked
  `nvidia-smi` (2 s deadline) or a stalled NFS `statvfs` (no deadline)
  delays CPU/RAM freshness with it. The blast wall isolates the daemon,
  not the metrics from each other.
- **F5 — fleet noise.** One NTP packet per host every 10 s is 8 640
  UDP probes/day/host for a value that drifts over hours.
- **F6 — static identity is re-derived 8 640 times a day** (#14, GPU
  model/driver, NIC mac/ips, disk inventory).

## 2. Proposed tiers

Assignment is by **cost first, volatility second**: anything under a few
ms with no hazard goes fast regardless of volatility (it is free); anything
that walks, forks, or touches the network goes slower.

| Tier | Default period | Env knob | Fields |
|---|---|---|---|
| **T0 identity** | rides the slow ticker (first tick at start) | — | `hostname`, `os_*`, `kernel_version`, `arch`, `boot_time_epoch_s`, `cpu_brand`, `physical_cores`, `logical_cpus`, `user` (the account the daemon runs as: `getuid` → passwd, `None` where there is no passwd database) |
| **T1 fast** | **1 s** | `THREEPM_HOSTMETRICS_INTERVAL_MS` (keeps its name: it *is* "the" sampling rate users tune) | `global_cpu_percent`, `per_core_percent`, `cpu_states`, `cpu_frequency_mhz`, `per_core_freq_mhz`, `kernel`, `load_avg`, `uptime_s`, `mem_*`, `swap_*`, `paging`, `mem_detail`, `pressure`, `file_handles`, `sockets`, `tcp_counters` (Linux `/proc` path only), `conntrack`, `vfs_cache`, `entropy_avail`, `cgroup`, NIC counters + `*_per_s`, `block_devices`, disk `read/written_bytes_total` + `*_per_s` |
| **T2 medium** | **10 s** | `THREEPM_HOSTMETRICS_MEDIUM_MS` | `processes_count`, `process_states` (the process walk — on Linux, since 2026-10, T6's walk when it is younger than a medium period: one walk of the table every 5 s instead of three every 10 s), `tcp_states` (socket table walk), `sensors`, `gpus` (identity + gauges — one tool invocation), disk rows: identity + `available_bytes`/`inodes_*` (statvfs), `raid`, `batteries` + `on_ac` (the `/sys/class/power_supply` walk, Linux; a handful of small sysfs reads per supply), macOS `mem_compressor` + `tcp_counters` (subprocesses), `sessions` (the accounts logged in — one utmp walk through libc, sorted, deduplicated, capped at 64; empty where there is no utmp) |
| **T3 slow** | **300 s** | `THREEPM_HOSTMETRICS_SLOW_MS` | `ntp_offset_ms`; T0 identity refresh; with Extended Monitoring on, the `extended` families `cpuidle`, `cpufreq`, `thermal` (§3.7) |
| **T4 3pm** | = medium | (rides `THREEPM_HOSTMETRICS_MEDIUM_MS`) | `3pmDaemon`: the daemon's and the agent's own cpu/RSS/uptime/fds/threads (two `sysinfo` process refreshes + a few `/proc` reads, ~1 ms) and the daemon's `Internals` over `rpc.sock` (one connect + ping + request, bounded 2 s; on the daemon side `fill_io_gauges` walks `/dev/fd` and stats three files — cheap at 10 s, noise at 1 s, hence never on the fast period). Its own loop (`hostagent/selfmetrics.rs`), so a daemon busy with a handoff never delays T2's walks, and never part of the first-snapshot gate |
| **T5 extended** | **10 s** | `THREEPM_HOSTMETRICS_EXTENDED_MS`; exists only with `[hostmetrics] extended` on (`THREEPM_HOSTMETRICS_EXTENDED`, default on) | `extended`: the kernel's own counters by family (§3.7), Linux; an empty map elsewhere. Never part of the first-snapshot gate |
| **T6 top** | **5 s** | `THREEPM_HOSTMETRICS_TOP_MS`; exists only with `[hostmetrics] top_processes` on (`THREEPM_HOSTMETRICS_TOP_PROCESSES`, default on) | `top_processes`: the 10 largest processes of the host by CPU, by memory (footprint on macOS, RSS on Linux) and by network rate, one union table sorted by pid (§3.8). A walk or a `ps`, plus one fork: ~35–45 ms a tick; macOS also keeps one `top` running for the footprint, ~1.8 % of a core at 5 s. Never part of the first-snapshot gate |

*As built:* NIC `mac`/`ips` stay in T1, but since 2026-10 on Linux
they are no longer re-read every tick: the counters come from one
`/proc/net/dev` read a tick, the addresses from a `sysinfo` refresh
(seven sysfs files per interface and two `getifaddrs`) every 10 s and
at once when an interface appears — an address change on an existing
interface shows within 10 s. Per-core frequencies are read from
`scaling_cur_freq` files the agent keeps open (one `pread` a core a
tick; the agent holds one descriptor per core). Block devices and the
disks' I/O counters come from one `/proc/diskstats` read a tick (it
was read twice, beside `/proc/mounts` and `/dev/disk/by-id`), the
mounts they are joined to being re-listed every 10 s — the period of
T2's disk rows. GPU identity
stays with the gauges in T2 (one `nvidia-smi`/`ioreg` run instead of
two). Disk rows are T2 (`statvfs` hazard) and receive their I/O
counters and rates from T1 by `mount_point` at assembly time.

Clamps: fast ≥ 200 ms (today's `MIN_INTERVAL_MS`), medium ≥ fast,
slow ≥ medium, extended ≥ fast, top ≥ fast; a knob set below its floor is raised with a warning in
`hostagent.log`, never rejected.

Effect on the `ls` host line (all its segments are T1 except the CPU
sensor temperature and GPU, which are T2): cpu/ram/load/throughput refresh
every second, throughput segments appear after 2 s instead of 20 s.

Budget: T1 at 1 s costs ~2–5 ms CPU per tick (no sleep, no fork) — 20–50 ms
per 10 s, i.e. the same order as today's single 10 s collect, for 10×
the freshness. T2 unchanged (~50–150 ms per 10 s). T3 removes 29 of every
30 NTP probes.

## 3. Architecture

### 3.1 Tier collectors

```
hostmetrics/
  mod.rs          the collectors' module; no whole-snapshot entry point (the agent assembles)
  tiers.rs        IdentityPart / FastPart / MediumPart / SlowPart / ExtendedPart / TopPart; Parts; assemble()
  fast.rs         FastCollector { sys, disks, nets: sysinfo handles; CpuWindow }  ← stateful
  medium.rs       MediumCollector { Linux: procwalk::Walker; elsewhere: sysinfo process table, tasks off }  ← stateful
  procwalk.rs     Linux: one `/proc/<pid>/stat` read per process into a kept buffer (T2's histogram, T6's table)
  slow.rs         fn collect_slow(extended) -> SlowPart, fn collect_identity() -> IdentityPart
  top.rs          TopCollector (two readings → the table), TopTier (reader + collector)  ← stateful (§3.8)
  top/            macos.rs (`ps`, `nettop`), linux.rs (`procwalk` walk, `ss` fallback), sockets.rs (`sock_diag`): readers + parsers
  extended/       Extended Monitoring (§3.7): parse.rs (shapes), table.rs (sources), mod.rs (collect),
                  ebpf.rs + ebpf/shortlived.bpf.{c,o} (short-lived processes, Linux, root)
  linux.rs / macos.rs / gpu.rs   unchanged pure parsers + readers
```

- `HostMetrics` (wire) stays the single flat struct — **no wire change for
  the shape**. Internally it is assembled from its parts:
  `HostMetrics = identity ⊕ fast ⊕ medium ⊕ slow ⊕ 3pm ⊕ extended ⊕ top`
  (the last three optional, never waited for) by a pure
  `assemble(&Parts) -> HostMetrics` in `tiers.rs`, unit-tested with
  partial inputs (a part still `None` → its fields absent/default,
  exactly like today's "absent, never faked").
- Cross-tier joins are by key: NIC by `name`, disk by `mount_point`, GPU
  by index within vendor — the same keys `rates.rs` already uses.
- `FastCollector` is **stateful** (F2): it owns a `sysinfo::System`
  refreshed with `refresh_specifics(cpu + memory)` only (never
  `new_all()`), and the previous `/proc/stat` ticks. CPU % and
  `cpu_states` are deltas over the tick window; the very first tick keeps a
  one-off 200 ms warmup so the first sample is never `None` on cpu (the
  `ls` line hides absent segments — an empty first line would look like a
  regression). Disk I/O counters on macOS come from
  `Disks::refresh_specifics(DiskRefreshKind::nothing().with_io_usage())`
  (no `statvfs`); on Linux from `/proc/diskstats` which T1 already reads.
- `rates.rs` generalises from `(prev: &HostMetrics, cur: &mut HostMetrics)`
  to `(prev: &FastPart, cur: &mut FastPart)` — same tests, window is now
  the fast tick.

### 3.2 Sampler: one ticker per tier, one cache

```
sampler::run(cache, Collectors, Intervals, token)
   ├─ task identity: interval(slow)   → spawn_blocking(collect_identity()) → cache.set_identity(part)
   ├─ task fast:     interval(fast)   → spawn_blocking(fast.collect())     → cache.set_fast(part)   (rates vs the previous fast part)
   ├─ task medium:   interval(medium) → spawn_blocking(medium.collect())   → cache.set_medium(part)
   ├─ task slow:     interval(slow)   → spawn_blocking(collect_slow(ext))  → cache.set_slow(part)
   ├─ task extended: interval(extended), only when Some → spawn_blocking(collect_extended() + ebpf::collect(shared)) → cache.set_extended(map)
   └─ task top:      interval(top), only when Some → spawn_blocking(TopTier::collect()) → Some: cache.set_top(part); None: nothing
every set_*: stamp the part, store it, bump the version; a read (`latest`) assembles(parts)
```

- `Cache` serves `latest` and `wait_first` to `server.rs`. *As built
  since 2026-10:* the snapshot is assembled when it is read, not when a
  part lands — an assembly deep-copies every part (the extended map is
  ~1 200 values) and the fast tier lands one a second, read or not. A
  `watch::Sender<u64>` version, bumped by each landing, is what readers
  wait on. The first snapshot is servable once **identity, fast and
  medium** have each landed (all first ticks fire immediately; the slow
  tier's NTP probe is not waited for). Collectors are `FnMut` boxes built by a
  factory so the fast one can keep state and be rebuilt after a panic.
- Each tier has its own `MissedTickBehavior::Delay` ticker and its own
  `spawn_blocking`; a wedged T2 (`nvidia-smi`, NFS) never delays T1 (F4).
  A panic in one tier keeps that tier's previous part and leaves the
  others live — extend
  `a_panicking_collector_keeps_the_previous_part_and_is_rebuilt`
  (`hostagent/sampler.rs`, and its twin in `hostagent/selfmetrics.rs`)
  per tier.
- Overlap guard: a tier whose collection is still running when its next
  tick fires skips that tick (Delay already gives this).

### 3.3 Freshness on the wire (additive, protocol v1)

`collected_at_epoch_ms` keeps meaning "age of the newest data" = the T1
stamp, so `3pm hostmetrics`'s `sampled` row and existing consumers are
unchanged. One optional field was added:

```json
"sampled_at_epoch_ms": { "fast": 1787490168000, "medium": 1787490160000, "slow": 1787489900000 }
```

(`Option<HostSampledAt>` whose members are each optional — a tier that
has not run yet is simply absent; an `extended` member dates the
Extended Monitoring tier when it is on (its slow families are the `slow`
stamp's); a `top` member dates the top-processes table (§3.8): when
the last table published landed in the cache — up to the network fork's
2 s after its reading — and, since a tick with nothing to publish leaves
it alone, how a reader tells a stale table; the whole object is absent from a
pre-tier daemon; frozen in `tests/fixtures/wire/v1/`.) `rate_window_ms`
stays and now reports the fast window (~1000). The `hostmetrics`
renderer adds `sampled (medium)` / `sampled (slow)` rows only when that
tier lags the fast stamp by more than 2 s.

### 3.4 Config surface

| Knob | Default | Floor |
|---|---|---|
| `THREEPM_HOSTMETRICS_INTERVAL_MS` | `1000` (was `10000`) | `200` |
| `THREEPM_HOSTMETRICS_MEDIUM_MS` | `10000` | `= fast` |
| `THREEPM_HOSTMETRICS_SLOW_MS` | `300000` | `= medium` |
| `THREEPM_HOSTMETRICS_EXTENDED_MS` | `10000` | `= fast` |
| `THREEPM_HOSTMETRICS_EXTENDED` / `[hostmetrics] extended` | on | — (off: no T5, no slow-tier walks, `extended` absent) |
| `THREEPM_HOSTMETRICS_TOP_MS` | `5000` | `= fast` |
| `THREEPM_HOSTMETRICS_TOP_PROCESSES` / `[hostmetrics] top_processes` | on | — (off: no T6, no fork, no walk, `top_processes` and `sampled_at_epoch_ms.top` absent) |
| `THREEPM_NTP_SERVER` | unchanged | — |

Changing the default of the existing knob is a behaviour change worth a
CHANGELOG line; the e2e harness that lowers it keeps working (it lowers
the fast tier, which is what the tests observe).

### 3.5 Tests (per CLAUDE.md checklist)

- `tiers.rs`: `assemble` with every subset of parts present (proptest over
  the 2⁴ combinations + random keys: a NIC present in fast but not identity
  still yields its counters with empty `mac`/`ips`).
- `fast.rs`: second tick yields cpu without sleeping (paused tokio time +
  a fake `CpuTicks` source); first tick warmup exactly once.
- `sampler.rs`: `start_paused` component test asserting tick counts per
  tier over a simulated 60 s (60 / 6 / 1 with a 10 s slow for the test),
  plus tier isolation (a T2 collector that blocks 5 s does not delay T1
  publishes).
- e2e: `ls` shows throughput segments within 3 s of daemon spawn;
  `hostmetrics --json` carries `sampled_at_epoch_ms` with
  `fast ≥ medium ≥ slow`; `THREEPM_HOSTMETRICS_MEDIUM_MS` below fast is
  clamped (log line asserted).
- Snapshots: `hostmetrics` render with and without the new ages.

### 3.6 As built

Shipped in one change set: parts + `assemble()`, the stateful
`FastCollector`, the multi-ticker sampler, the three knobs,
`sampled_at_epoch_ms`, and the doc updates (`hostagent.md`,
`metrics.md`, `protocol.md`, OV `3pm-docs-archived/architecture` §8.2, FEATURES, CHANGELOG).
T4 (`3pmDaemon`) followed as its own change set: `hostmetrics/threepm.rs`
(OS half), `hostagent/selfmetrics.rs` (the loop and the `rpc.sock` pull),
`Parts.threepm` landing the wire section as is. T5 (§3.7) and T6 (§3.8)
each came behind an option the daemon resolves once per image and hands
to every agent incarnation in the option's own env var.

### 3.7 Extended Monitoring (T5 + the slow tier's share)

`HostMetrics.extended` is a map of families (`HostExtended`), not typed
fields: the kernel names the values and varies them between versions.
`hostmetrics/extended/table.rs` has one row per kernel file — family,
path, shape (`parse.rs`), kind rule, skip list; `mod.rs` walks a table.
Adding a family is adding a row.

- **Two tables, two tiers.** `EXTENDED` on T5; `SLOW` (the three sysfs
  walks with a file per core per state or per zone: `cpuidle`, `cpufreq`,
  `thermal`) on T3. `assemble` merges the two into one map. With the
  mode on the map is present on every platform (macOS: no family); off,
  no T5 task exists, the slow tier opens nothing more, the map is absent.
- **Kinds.** A family is all counters (cumulative: since boot for the
  kernel's files, since the agent's eBPF load for `shortlived*`) or all
  gauges, for good. A file mixing both is split at the source:
  `vmstat` / `vmstat_gauge` (`nr_*` levels but seven `nr_*` counters,
  `workingset_nodes`), `snmp` / `snmp_gauge` (settings, `CurrEstab`),
  `pressure` / `pressure_gauge` (`avg*` vs `total`), `ksm` /
  `ksm_gauge` (`full_scans`, `pages_scanned`, `pages_skipped` count),
  `hwmon` / `hwmon_gauge` (`energy*_input` counts), `btrfs` /
  `btrfs_gauge` (`reclaim_*` count), `zfs_arc` / `zfs_arc_gauge`
  (every `*hit*`, `*misses*`, `evict_l2_*` and the named events), and
  `wireless`, `sctp` alike. `diskstats` is all counters (its one level,
  in-flight I/O, is `block_devices`').
- **Never twice.** Each source lists what a typed field already carries
  (`/proc/stat`'s all-cores line and scalars, the six sysinfo NIC
  counters, systemstat's nine `diskstats` columns, `meminfo` totals and
  `mem_detail`, `vmstat` paging, the twelve `tcp_counters`, `sockets`,
  `vfs_cache`, cpu 0's frequency, hwmon `temp*_input/_crit/_highest`, an
  amdgpu chip's `power1_average` (`gpus[].power_w`), a thermal zone's
  `temp` when an hwmon chip bears its type's name — or always, when no
  hwmon chip has a temperature and sysinfo reads the zones — and the
  three DRM gauges). Nor one family twice: vmstat's `nr_*` levels that
  `/proc/meminfo` prints (`nr_free_pages` = `MemFree`, the LRU lists,
  slab, `nr_dirty`, `nr_shmem`, THP, `nr_kernel_stack`, …) and every
  `nr_zone_*` go; `nr_file_pages`, `nr_isolated_*`, `nr_zspages`,
  `nr_free_cma` and the dirty thresholds stay.
- **Throwaway rows.** Loop devices and ramdisks (`diskstats`), veth
  pairs (`netdev`, `wireless`) are not sent: a series set per snap or
  container start that nobody reads.
  Fully typed files are not in the table (`file-nr`, `inode-nr`,
  `entropy_avail`, `uptime`, netfilter, `power_supply`).
- **Values.** Counters are whole numbers (the node state quantizes a
  fraction to 3 digits, which would break a rate); only `kB` → bytes is
  converted. No NaN or infinity, no name starting with `~`.
- **Availability, read by read.** A missing file: the family is absent,
  nothing said (no ZFS here). A refusal (`EACCES`/`EPERM`):
  `unavailable[family] = "permission"`, the family absent — or partial,
  for a sysfs walk where one attribute refused (a write-only attribute
  such as `emul_temp` does not count). Any other failure: `"error"` and a
  debug log. `EOPNOTSUPP` counts as missing (PSI compiled in, `psi=0`
  at boot). Never an euid check. A walker that panics (the slow share
  runs beside the NTP probe) costs only its families that tick, listed
  as `"error"`.
- **Rows.** `""` for a one-row family; else the kernel's key: core index
  (`cpu`, `cpufreq`), `<core>/<C-state>` (`cpuidle`), device, interface,
  protocol (`snmp`: `Ip`, `Tcp`…), `<resource>_<scope>` (`pressure`),
  sysfs entry (`thermal_zoneN`, `cooling_deviceN`, `cardN`), hwmon
  chip name (`nvme`, `coretemp`; `nvme_1` for a second `nvme`, by hwmon
  index — `hwmonN` follows probe order, the name does not),
  `<dev>/<port>` (InfiniBand), `<uuid>/<group>` (btrfs).
- **Cost (edgecr4ft.net, 5.4, 32 cores, 2026-09-29, debug build):** T5
  median 9.4 ms, 20 families / 941 series; the slow share 7.5 ms, 3
  families / 287 series (`cpuidle` 192). Families there: `cpu` 320,
  `netstat` 133, `vmstat` 87 + 9, `snmp6` 87, `snmp` 77 + 7,
  `diskstats` 72, `netdev` 60, `meminfo` 38, the rest under 15 each.
  On the link (same host, 2026-09-29, a node with the mode on beside one
  with it off, same 4 apps, 14 min): 777 against 529 B/s up, +248 B/s
  for host families and per-process `ext` together — a state patch
  carries only the leaves that moved, deflated; ~30 B/s more with the
  eBPF families (root agent).
- **Short-lived processes (eBPF, `ebpf.rs`).** What no tier sees: on
  edgecr4ft 84 % of the processes that exit live under a second.
  `ebpf/shortlived.bpf.c` hooks `sched:sched_process_exec` / `exit` and
  keeps `totals`, `by_comm` (name → count, total and max lifetime) and
  `starts` (pid → exec time, cleaned by each exit, `max_entries` fixed:
  full, it loses lifetimes, never node memory). Only a process whose exec
  was seen has a lifetime and a name: an exit without one — forked
  without exec (Postgres backends, prefork workers), started before the
  load, or its exec refused by a full `starts` — is `exits_without_exec`
  and appears in the totals only. A full map is counted, not silent:
  `starts_full` (execs `starts` refused), `by_comm_full` (exits
  `by_comm` refused). No CO-RE — tracepoint
  arguments only — so one committed object (`shortlived.bpf.o`, bpfel,
  8 352 bytes, rebuilt by `just bpf`, reproducible) serves every kernel
  and architecture, with or without BTF; embedded with
  `include_bytes_aligned!`, loaded by `aya` (a Linux-only dependency).
  Loaded on T5's first tick, once per agent: the state is shared by
  every collector T5 builds, so a rebuild after a panic neither reloads
  nor attaches twice. Before aya, `CapEff` (`/proc/self/status`): without
  `CAP_SYS_ADMIN`, or `CAP_BPF` + `CAP_PERFMON`, the answer is `not_root`
  and nothing is loaded — aya's loader parses the kernel's BTF
  (`/sys/kernel/btf/vmlinux`, 4.5 MB) before anything can fail; an
  unreadable `CapEff` tries (a deliberate exception, 2026-09-29, to "try,
  don't guess", plan §4; the uid is still never consulted). The check only rules out: the capabilities
  are necessary, not sufficient. Attaching also reads the tracepoint's id
  from tracefs (`/sys/kernel/tracing/events/…/id`, `0700` root on
  Ubuntu: uid 0 or `CAP_DAC_READ_SEARCH`), Ubuntu's
  `perf_event_paranoid = 3` may refuse `perf_event_open` without
  `CAP_SYS_ADMIN`, and before Linux 5.11 the maps and the program count
  against `RLIMIT_MEMLOCK`: on `EPERM` aya 0.14 raises it to unlimited
  and retries once (`with_raised_rlimit_retry`), which takes
  `CAP_SYS_RESOURCE` or an unlimited hard limit, so a capable but
  limited agent is reported `not_root`. The try says which. The load
  passes `btf(None)` (no CO-RE):
  one BTF parse instead of `Ebpf::load`'s two, the one left being
  `EbpfLoader::new`'s own, which aya 0.14 cannot skip. The agent's
  allocator is pinned against what that parse frees: `docs/hostagent.md`,
  "Memory". Refused:
  `unavailable.shortlived` = `not_root` (the `CapEff` check, `EPERM`,
  `EACCES` outside the verifier), `bpf_disabled` (`ENOSYS`) or
  `load_failed` (the verifier, a missing tracepoint: warn log), and no
  retry before the next agent start; the log line of a refusal quotes
  `unprivileged_bpf_disabled`, which classifies nothing (tracepoint
  programs need privilege whatever it says). Each T5 tick: `totals` →
  `shortlived` (counters, one row: `execs`, `exits`, `exits_under_1s`,
  `lifetime_ns`, `lifetime_under_1s_ns`, `exits_without_exec`,
  `by_comm_full`, `starts_full`, since the load); `by_comm` drained (keys listed, then each read and
  deleted), the window's 10 names with most exits kept, the rest — and a
  name that cannot be a row: empty, a control character, `~…`, `other`
  itself — summed under `other`: `shortlived_by_comm` (counters `exits`,
  `lifetime_ns`, running totals kept agent side) and
  `shortlived_by_comm_max` (gauge `max_ns`, the window's). A row appears
  only in the windows it made; its running total survives 30 windows
  without it, then goes (a later return is a counter reset): at most
  11 × 31 totals held. A name's first exit of the window inserts with
  `BPF_NOEXIST` (two first exits on two CPUs add up instead of
  overwriting); an exit racing the drain of its name is lost (no
  lookup-and-delete on a hash map before 5.14); `totals` loses none.
  Verified 2026-09-29: edgecr4ft (5.4, BTF) and a tablet (6.19, no BTF)
  as root, `true` × 200 → exactly 200 execs / exits / under 1 s and a
  `true` row of 200; as `ubuntu`/`envision`, `not_root`.
- **Tests.** Fixtures are `/proc` and `/sys` as captured read-only on
  edgecr4ft (5.4) and nirmata (6.8) under
  `crates/threepm-daemon/tests/fixtures/hostmetrics-extended/`, used as
  collection roots; `hostmetrics::extended::` runs on the Mac (eBPF
  with fake maps). The real load is `#[ignore]`d
  (`ebpf::tests::real_load_counts_fork_exec_or_says_not_root`): run the
  test binary by hand on Linux, `--ignored`, once with `sudo` and once
  without.

### 3.8 Top processes (T6)

`HostMetrics.top_processes`: which processes are eating the box, managed
by 3pm or not. One table, not three: the 10 largest by CPU, by memory
(the physical footprint on macOS, the RSS on Linux) and by network rate
(in plus out), merged — at most 30 rows,
sorted by pid so that the node state diffs it row by row. A dimension
with fewer than 10 non-zero figures is shorter, not padded.

- **Sources.** Linux: a walk of its own (`procwalk`: one
  `/proc/<pid>/stat` read per process — `comm`, `utime + stime`, `rss`,
  start time — and `status` once per process for its uid; CPU is a
  delta, so it cannot share T2's walk; until 2026-10 a `sysinfo` walk
  that read `statm` too and cost three times the syscalls) and the
  per-socket byte counters summed per pid: one `sock_diag` dump of both
  families' connected TCP sockets with their `tcp_info` (the
  `TCPDIAG_GETSOCK` request, one walk of the kernel's established table
  where `ss` made one per family), holders from the readable processes'
  fd tables (`top/sockets.rs`) — the figures `ss -tinpH` gave, by the
  same rules (`top/linux.rs`, `parse_ss`); `ss -tinpH` itself where the
  host refuses `sock_diag`. Until 2026-10 it was `ss` every tick: a fork,
  an exec and two walks every 5 s. macOS: `/bin/ps -A -o
  pid=,uid=,rss=,time=,ucomm=` (setuid root: unprivileged, no
  in-process API sees another user's process) and `/usr/bin/nettop -P -n
  -L 1 -x -J bytes_in,bytes_out` (`-n`: without it nettop waits 5 s on
  reverse DNS), plus the footprint from a `/usr/bin/top -l 0 -s <S>
  -stats pid,mem` kept running (below). Elsewhere: no reader, no table,
  no error. Each per-tick fork has a 2 s deadline.
- **The footprint (macOS).** What Activity Monitor and Stats call Memory
  (`ri_phys_footprint`), where the RSS misses the biggest consumers — a
  VM at 8.2 GB of footprint shows 803 MB of RSS, and WindowServer, at
  4 GB, is out of the RSS top ten. `proc_pid_rusage` answers only half the pids
  unprivileged; `top` sees them all, but a `top -l 1` costs 0.78 s of
  CPU per run. So one `top` runs for the tier's life, sampling every
  `S` = the tier period rounded up to whole seconds (at least 1: never
  more often than the tier reads), under `LC_ALL=C`;
  after its first sample it costs ~0.1 s of CPU per sample — measured
  2026-10-01 on the dev Mac: `top` 1.84 % of a core at 5 s (2.2 s over
  2 minutes), the agent 0.77 % and 33.7 MB against 0.74 % and 33.4 MB
  before. A thread reads its stdout block by block
  (`top/footprint.rs`: a block is complete at the row count its
  `Processes: N total` line announced, or at the next `Processes:`
  line; one the stream ends inside is dropped; units `B K M G T` in
  1024 steps, a trailing `+`/`-` ignored, a line that does not read
  skipped) and keeps the latest `pid → bytes` with its instant. A tick
  takes it when at most two intervals old. `top` is started at the
  tier's first read (the option off forks nothing), killed and reaped
  when the tier is dropped (a panic's rebuild leaves no orphan), and
  started again at most once per 30 s when it ended, with one log line.
  One alive but silent for three intervals (5 s at least: stopped,
  stuck) is killed and reaped with a log line, and restarted under the
  same 30 s limit. Its signal mask is cleared in the child
  (`sigmask::clear_in_child`: `Command` keeps the forking thread's) and
  `SIGPIPE` is back to its default, so a `top` left behind by a killed
  agent dies at its next write.
- **Figures.** `cpu_percent` is CPU time over the tick's window, 100 =
  one core, computed here on both platforms (never `ps %cpu`).
  `mem_bytes` is RSS on both platforms: shared pages count in every
  process holding them. `footprint_bytes` is macOS only, absent for a
  pid `top` did not list. When the reading has a footprint map, memory
  ranks on it and a pid without one has no memory rank — never ranked
  on its RSS, the two do not compare; without a map (Linux, or `top`
  down), memory ranks on the RSS.
  `net_*_bytes_per_s` are deltas of cumulative per-pid counters; absent,
  never zero, when the platform has no source, the source cannot see the
  process (Linux: TCP only, and other users' sockets only as root), or a
  counter went down over the window (a socket closed).
- **The first tick primes.** A collector's first reading publishes
  nothing; the table lands on the second tick, 5 s in. A process seen
  for the first time, or whose pid was reused, has no delta either: a
  process younger than one window has no row.
- **`None` keeps what is served.** The collector answers
  `Option<TopPart>`: `None` (the priming tick, two readings under 100 ms
  apart, a platform with no reader) lands nothing and the previous part
  stays, stamp included. So a collector rebuilt after a panic primes
  again without blanking the table.
- **A failed read blanks it.** When the process table cannot be read
  (macOS: `ps` failed or timed out; Linux: the walk found no process), an empty table replaces the one served —
  a stale one would pass for a sample — on any tick, a rebuilt
  collector's first included. The baseline stays, so the next reading is
  measured over both periods. A failed network source costs only the
  network columns of two windows; a wedged `nettop` or `ss` stretches
  its tick to the 2 s deadline, which the ticker absorbs (missed ticks
  are skipped, not caught up).
- **Names.** The executable's name, never a path or the arguments,
  cleaned and capped like the process-tree names. Linux: the kernel's
  `comm`. macOS: `ucomm`, the kernel's own, cut at 16 bytes; a name
  within four bytes of that cap (the padding hides a cut space, a cut
  character drops up to three bytes) is completed from the file name of
  the executable (`proc_pidpath`, unprivileged, no fork), kept only when
  it extends `ucomm`; when the kernel has no path (a deleted
  executable) the cut name stays. `ps`'s `comm` is never read: it is
  what the process wrote as its `argv[0]`, arguments included. `user` is the account name, or the uid as a number
  when it has no entry.
- **Tests.** Parsers on fixtures of real output
  (`crates/threepm-daemon/tests/fixtures/hostmetrics-top/`), on every
  platform; the core without an OS (`hostmetrics::top::`); the tier in
  `hostagent::sampler::` with fake collectors.

</document>

<document url="https://3pm.edgecr4ft.net/docs/internals/extensibility.md" title="Extensibility and pm2 feature map (design)">

# 3pm — Extensibility & pm2 Feature Map

Written 2026-07-10, after the extensibility milestone. Records why the
architecture looks the way it does, which seams exist for new features, and
how the remaining pm2 surface maps onto them. Companion to
OV `3pm-docs-archived/architecture` (mechanisms) and `CLAUDE.md` (playbooks).

## 1. The audit (2026-07)

An architecture audit graded the codebase against pm2's full feature surface.
Verdict: the foundations (pure policy machine, single-actor daemon,
race-safety kernel, lenient wire) were sound, but four structural gaps would
make every future feature more expensive than the last:

1. **Closed response/output layer** — `Response` couldn't grow safely,
   `Err` handling was hand-copied per command, no machine output existed.
2. **Spec construction triplicated** — every per-app field was hand-mapped
   in the struct, the CLI flags, the TOML mirror, and a stringly
   `KNOWN_FIELDS` list, with silent drift.
3. **App ≡ one process** — one pid per state row, name-keyed files, no way
   to express pm2's flagship cluster mode or zero-downtime reload.
4. **No channel into the child** — only stdout/stderr and exit; nothing
   like pm2's `process.send` for readiness, graceful shutdown, metrics,
   or actions.

The milestone closed them in dependency order (2 mechanical, then 2
structural): output layer → spec single-sourcing → instances identity →
control channel.

## 2. What each gap became

### Output layer (gap 3)

`Response` gained a `#[serde(other)] Unknown` fallback (decoding can never
fail on growth); one `unwrap_response` core with tiny `expect_*` wrappers
replaced the per-command matches; the `Out` sink (`threepm-cli/src/output.rs`)
is the single impure output boundary — global `--json` emits the wire types
on stdout (NDJSON for `logs`), chrome and errors go to stderr, exit codes are
stable. **Adding a command's output support is one `Serialize` payload + one
`out.json()` call** (playbook in CLAUDE.md).

### Spec single-sourcing (gap 4)

`SpecPatch` (`threepm-core/src/spec.rs`) is the one description of what
users can configure: CLI flags and ecosystem files both compile down to it;
`apply()` folds it over defaults with validation (typo'd `kill_signal`
fails fast with a did-you-mean). Drift is compiler/test-enforced: a new
`StartSpec` field breaks `apply()`; the new patch field trips the
`known_fields_match_patch` test. **A new per-app field is one struct field +
one mapping line, and TOML support is automatic.** `SpecPatch` is also
serde-able by design — it is the future partial-update primitive
(`restart --update-env`, `3pm set`).

### Instances identity (gap 1)

An app is a name owning N instances; each instance is its own ProcId row
with its own policy state (supervision — backoff, crash-loop, memory guard —
is per-instance by construction; **the state machine was not modified**).
`supervisor::reconcile_app` is the one engine: `start` refreshes the spec
and reconciles the count, `3pm scale` reconciles the count without touching
survivors. Instance 0 keeps the historical unsuffixed log/pid names;
instances get `THREEPM_INSTANCE`/`NODE_APP_INSTANCE`. Doomed rows keep
their numbers occupied so paths never collide with a dying row's cleanup.

### Control channel (gap 2)

`ctl.sock` — same hardening as the other sockets — carries NDJSON both ways.
Children are told `THREEPM_CTL_SOCK` + `THREEPM_GEN` at spawn; `hello{id,gen}`
pins a connection to one incarnation (stale reconnects are hung up). Built on
it: `wait_ready`/`listen_timeout` (readiness gates `launching → online`;
policy Invariant W keeps stop latency bounded by `kill_timeout`) and
`shutdown_with_message` (graceful stop as a message, SIGKILL escalation
unchanged). Unknown message kinds are skipped on both sides — the channel
grows without breaking older peers. Client contract: four lines in any
language (`examples/hello-ready.py`; OV 3pm-docs-archived/architecture §4.3b).

## 3. The seams (where new things plug in)

| You want to add… | Where it goes | Cost |
|---|---|---|
| A CLI command / RPC | clap variant → handler → `Request` variant (+`expect_*` if a new shape) → `Command`/`Outcome` variant in `threepm-daemon/src/domain/` → `handle_rpc` arm (it matches on `Command`) | small; playbook in CLAUDE.md |
| A per-app config knob | `StartSpec` + `SpecPatch` (+ optional flag) | one field + one line; TOML free |
| A lifecycle behavior | new `PolicyInput`/`Action` in the pure machine + daemon executor arm + table/model tests | contained; keep decisions out of I/O code |
| A restart trigger (cron; watch is done this way, `watcher.rs`) | a daemon-side task producing existing `RestartRequested` inputs, reconciled from the rows by `derived::DerivedSet<K>` (`cron.rs` and `watcher.rs` share it) | additive; no policy change |
| A child↔daemon message | `ControlIn`/`ControlOut` variant (lenient decode on both sides) | additive |
| A bus event kind | `Event` variant (`decode_lenient` skips it on old clients) | additive |
| An embedded client (GUI, agent) | `threepm-protocol::client` with an injected launcher | already supported |

## 4. Remaining pm2 surface, graded

**Cheap now (additive on existing seams):** `cron_restart` (copy the
watch-mode shape: a task → `RestartRequested`), `env_<name>` blocks +
`--env` + `--update-env`, `stop_exit_codes`, `reset` (counters),
`sendSignal`, `--sort`/`--mini-list`,
monit/dashboard TUI (renders `ProcessRecord`s + the event bus), `jlist`
(already: `ls --json`), `report` (already: `internals`).

**Send-to-stdin: done by `3pm attach` (2026-09).** An app started with
`tty = true` runs on a pseudo-terminal the daemon holds;
`printf 'cmd\n' | 3pm attach <app>` (pipe mode) writes stdin to it and
prints what the app answers, and on a terminal `3pm attach` is the
interactive console (pm2's `attach`/`send`). An app on pipes keeps a
null stdin: a terminal is opt-in because it changes the app (one
output stream, `isatty` true, killed with a crashing daemon). The seam
is a request that takes its connection over (`Request::takes_connection`,
`RpcHandler::takeover`) and then speaks the live framing both ways —
the shape a future attach through the fleet link reuses.

**Done as modules (2026-09):** `serve` is the first bundled module
(`modules/serve`, its own workspace and lockfile — `docs/modules.md`).
The module system itself (`install`/`uninstall`/`module list`,
`3pm <module> …` dispatch, `spec.module` provenance shielding module
apps from `all`) lives in `threepm-cli/src/modules/` plus one daemon
seam (`resolve_mutation_target`); a `3pm set/get` config store for
modules is the natural next step on the `SpecPatch` primitive.

**Next structural feature — zero-downtime `reload`:** both prerequisites now
exist (N instances per app + ready signal). Shape: per instance, spawn the
replacement, wait for its `ready`, then retire the old one — a policy
vocabulary extension (an overlap state), not a remodel.

**On the control channel:** application metrics and events are in
(`metrics`/`event` lines, `Request::AppMetrics`, `Event::App` — one
connection per process of the app, keyed by `source`). Still open on the
same socket: remote actions (`3pm trigger` → `action` messages; the
per-source connections and peer pids are the groundwork), a Node shim
exposing pm2's `process.send('ready')` API.

**Out-of-process helpers:** the host metrics agent (`hostagent/`) is the
template — same binary, hidden role, `ReadyPipe` handshake, flock
singleton, daemon-side respawn loop, `RpcHandler` on its own socket. A
future collector that must not share the supervisor's fate goes the
same way.

**Plugin territory (keep out of core):** deploy/vizion git tooling,
PM2+/cloud link agent — both candidates for modules now. (Log rotation
used to be listed here; it moved into the core once the shared-sink
design made it a byte counter and a rename under an existing lock — see
OV `3pm-docs-archived/architecture` §6.4. The module system was listed here too until it
collapsed to a manifest, a copy and an external-subcommand dispatch.)

**Node/V8-specific (not applicable to a language-agnostic manager):**
`inspect`, V8/heap profiling, `--trace`, Node cluster fd-sharing (covered
instead by per-instance env + SO_REUSEPORT-style patterns).

## 5. Ground rules that keep this true

- Lifecycle decisions stay in the pure machine; the daemon only translates
  I/O into inputs and executes actions.
- Wire vocabularies (requests, responses, events, control messages, error
  codes) only grow — every decode path tolerates unknowns.
- The spec has one source of truth; validation happens at construction.
- Never assume name ↔ one process; never trust a message without its gen.
- Every feature lands with tests at the layer that owns it (table/property
  for policy, paused-time component for the actor, e2e for the user surface).

</document>