# Features

User-facing features, one section each: what you see, how to get it, and
where it comes from. Internals live in
`viking://resources/3pm-docs-archived/architecture/` (OpenViking).

## Start a command from your PATH: `3pm start npm -- run dev`

`3pm start <target>` takes a file (relative to the cwd, or `--cwd`) or,
when there is no such file and the target has no `/`, a command found on
your PATH, as in pm2: `3pm start npm -- run dev`, `script = "npm"` in an
ecosystem file, `pm.start({script: 'npm', args: 'run dev'})` in
`@3pm/api`. The app is named after the command (`npm`). A file in the cwd
wins over PATH, and a miss says where it looked:
`script 'npm' not found (looked at <cwd>/npm, then PATH)`.

**A managed app first.** Before any of that, a bare word — no `/`, no
blank, not a config file, no `-- args` after it — that names an app the
daemon manages, one of its ids, or `all`, restarts it as pm2 does:
`3pm stop api` then `3pm start api` brings `api` back with the
configuration it has (its ecosystem file's `interpreter`, env, …), not a
fresh one read off the cwd. `start` flags typed alongside are folded in
like `3pm restart api [flags]`, except that `--watch` alone sets watch
mode instead of toggling it. The managed name wins even over a file of
that name in the cwd; `./api` always means the file, and
`3pm start api -- …` always (re)defines the app from a script. Code:
`app_word`/`managed_target` in `crates/threepm-cli/src/commands/start.rs`.

**How it works.** The lookup happens in the CLI, against the shell you
typed the command in: the daemon's environment froze when it was
spawned. The app runs the PATH entry itself
(`~/.nvm/versions/node/v25.2.1/bin/npm`), not the file it links to, so an
upgrade that moves that file does not break it. The interpreter is
inferred from the linked file's extension (`npm-cli.js` → `node`,
resolved to an absolute path), so `#!/usr/bin/env node` is never looked
up in the daemon's PATH. A command whose target has no known extension
runs by its shebang, in the daemon's environment; `--interpreter`
overrides it. An interpreter given as a path (`--interpreter /bin/sh`,
`.venv/bin/python`) is kept as given, links included: BusyBox and venvs
depend on the name they are called by. Code: `crates/threepm-cli/src/commands/lookup.rs`,
`crates/threepm-cli/src/commands/target.rs`.

## Start a command line: `3pm start "python app.py"`

A target with a blank in it that is not a file runs under a shell, as in
pm2: `3pm start "npm run dev"` is `bash -c "npm run dev"` (`sh -c` when
your PATH has no bash). The same in a file (`script = "npm run dev"`, or
pm2's `cmd`/`exec`/`command`) and in `@3pm/api` (`{script: 'npm run
dev'}`). The app is named after the command's words: `npm run dev` →
`npm-run-dev`, `python ./src/app.py --port 3000` →
`python-src-app.py-port-3000` (a character a name refuses becomes `-`,
the `-`/`.` at a word's edges go, 128 bytes at most, `app` when nothing
usable is left); `--name` overrides it. The command carries its
arguments inside the quotes: `-- args`, `args`, `--interpreter` or
`--interpreter-args` beside it are errors, where pm2 drops them
silently. A file whose name has a blank still wins, and a command that
merely ends like a config name (`"node x.js --config app.json"`) stays a
command. `stop <file>` and `--only` find such an app by the name its
command implies. Code: `crates/threepm-cli/src/commands/target.rs`,
`threepm_core::process::app_name_from_command` (mirrored in
`packages/node/lib/spec.js`).

## Interpreter: `none`, its own flags, Python's output

- **`--interpreter none`** (`interpreter = "none"`, pm2) runs the script
  itself, by its shebang, whatever its extension says. On `3pm restart
  app --interpreter none` it clears the interpreter the app had, and in
  an app it beats one set in `[defaults]`. A script run without an
  interpreter must be executable: the CLI says so before anything starts
  (`… is not executable and has no known interpreter — chmod +x it or
  pass --interpreter`), for a file's apps as for the command line — a
  file's app used to become an `errored` spawn instead.
- **`--interpreter-args "--inspect --max-old-space-size=64"`** (pm2's
  `--node-args`; `interpreter_args` in a file, or pm2's `node_args` /
  `interpreterArgs`: a string split on blanks, or a list) go between the
  interpreter and the script: `node --inspect app.js`. `""` clears them
  on a restart; without an interpreter they are an error. `3pm show` has
  an `interpreter args` row. They never leave the host over the fleet
  link (they may carry `--inspect=host:port`).
- **Python's output is not buffered.** Python block-buffers stdout when
  it is a pipe, so a `print()` without `flush()` reached the log 8 KiB
  late. The daemon sets `PYTHONUNBUFFERED=1` (pm2 does too) for an app
  whose interpreter's file name starts with `python`, that runs a `.py`
  script by its shebang, or that is a quoted command line whose first
  word is such a python (`3pm start "python3 app.py"`, run by `bash
  -c`) — at spawn time, under the app's `env`
  (`PYTHONUNBUFFERED = ""` there turns it off), never stored in the spec,
  so it follows the interpreter the app runs now.
- pm2's `exec_interpreter` is accepted for `interpreter`.

Code: `threepm_core::spec::interpreter`,
`crates/threepm-daemon/src/spawner.rs` (`argv`, `implicit_env`).

## Config formats: YAML, JavaScript, stdin, the current directory's file

- **Formats.** Every command that takes a config file (`start`,
  `validate`, `apply`, `runtime`, `stop|restart|delete <file>`) takes
  `ecosystem.toml` (native), `.json`, `.yaml`/`.yml` and pm2's
  `*.config.js`, `*.config.cjs`, `*.config.mjs`. JSON, YAML and
  JavaScript also take pm2's other shapes: a bare list of apps, or one
  app alone (an object with a `script` and no `apps`). pm2's `deploy`
  section is ignored, like any unknown top-level key.
- **JavaScript configs are code**: the CLI evaluates them with the
  `node` of your PATH, as pm2 does — in your cwd and environment
  (`env: { PORT: process.env.PORT }` reads your shell's), stdin closed,
  10 s and 16 MiB of output at most. `module.exports`, `export default`
  and named exports (`export const apps`) all work; functions vanish (the
  export crosses as JSON). `restart --update-env` evaluates the file
  again. Without node: `ecosystem.config.js is JavaScript: evaluating it
  needs node in PATH`; a config that throws shows node's error.
- **stdin.** `3pm start -` (and `3pm validate -`) read the config piped
  in: JSON when it opens like pm2's (`{`, or `[` then `{`/`]`), TOML
  otherwise. Relative paths resolve against your cwd. Refused when stdin
  is a terminal, over 8 MiB, and by `apply` (`--prune` needs the file an
  app came from). Such an app has no config file to read again:
  `restart --update-env` says so.
- **No target.** `3pm start`, `3pm validate` and `3pm apply` alone use
  the ecosystem file of the current directory, the first found of
  `ecosystem.toml`, `.json`, `.yaml`, `.yml`, `.config.js`, `.config.cjs`,
  `.config.mjs` (`Using ecosystem.toml`; the others present get a
  warning). None: `no target given, and no ecosystem file in <cwd>
  (looked for …)`.

Code: `crates/threepm-cli/src/config/{formats,js,input}.rs`; YAML via
`serde_norway` (the maintained fork of `serde_yaml`).

## Restart policies

**What you see.** By default every unrequested exit respawns the app
(`always`, pm2 behavior). With `on_failure`, a clean `exit 0` parks the app
in `stopped` — `3pm ls` shows `stopped`, `exit code 0`, no restart counted —
while a non-zero exit, a signal death, or a failed spawn respawns it with
the usual restart pacing and crash-loop cutoff. `never` parks the app on
any exit.
`3pm show` prints the active policy in its `restart policy` row.

**How to get it.**

```console
$ 3pm start job.sh --restart-policy on-failure
```

```toml
[[apps]]
name = "job"
script = "./job.sh"
restart_policy = "on_failure"   # always | on_failure | never
```

pm2's `autorestart = false` / `--no-autorestart` still work and mean
`never`; giving both a policy and a contradicting `autorestart` is a config
error. A `dump.json` written before this field existed decodes its
`autorestart` bool into the matching policy.

**Where it comes from.** `RestartPolicy` in `threepm-core::spec`; the
decision is `restart_policy.restarts_after(exit_code)` inside the pure
policy's `after_death`, so user `stop`/`restart`/`delete`, watch restarts
and the memory guard (which all park an intent before the exit) are never
gated by it.

## Restart pacing: instant by default, backoff on demand

**What you see.** A crashed app respawns at once (pm2 behavior), so a
crash loop runs through its 16 unstable exits in a moment and `3pm ls`
shows `errored` in red, with the cause below the table. `↺` counts the
respawns. Two knobs pace the respawns, both pm2's:

```console
$ 3pm start api.js --restart-delay 500                 # fixed 500 ms pause before each restart
$ 3pm start api.js --exp-backoff-restart-delay 100     # 100 ms, ×1.5 per quick crash, capped at 15 s
```

```toml
[[apps]]
name = "api"
script = "./api.js"
restart_delay = 0                 # ms; fixed pause before every crash restart
exp_backoff_restart_delay = 100   # ms; > 0 switches exponential backoff on
backoff_factor = 1.5              # ┐ tuning, only with exp_backoff_restart_delay:
backoff_max = 15000               # │ growth per quick crash, cap, and the uptime
backoff_reset_after = 30000       # ┘ after which the delay resets to the first one
```

With backoff on, the app sits in `waiting restart` (yellow) between two
attempts; the delay resets to the first one once an incarnation stays up
`backoff_reset_after`. The 0.1 spelling `backoff_initial` /
`--backoff-initial` is still accepted. Neither knob changes the crash-loop
cutoff: `errored` still comes after `max_restarts` consecutive exits under
`min_uptime`, however they are paced.

**pm2 parity.** Same option names (`restart_delay`,
`exp_backoff_restart_delay`, `--exp-backoff-restart-delay`), same growth
(×1.5, capped at 15 s), same `waiting restart` status between attempts,
same default (instant). Two deliberate differences: pm2 only forgets the
grown delay on a manual restart/reload, so an app that crashed once at
boot keeps a long delay for its next crash hours later — 3pm resets it
after `backoff_reset_after` (30 s) of stable uptime; and pm2 hard-codes
factor and cap where 3pm exposes `backoff_factor` / `backoff_max`. When
both knobs are set, pm2 lets `exp_backoff_restart_delay` win; 3pm starts
the backoff at the larger of the two.

**Where it comes from.** `threepm-core::backoff::next_delay`: the fixed
`restart_delay_ms` unless `StartSpec::exp_backoff_enabled()`
(`backoff_initial_ms > 0`, the unchanged wire name), then the pm2 growth
rule. The policy's `after_death` turns it into one `SetRestartTimer`.

## Crash diagnostics: killing signal and stderr tail

When an app dies, 3pm remembers *why* — not just the exit code:

- **`exit_signal`** — the POSIX name of the signal that killed it
  (`SIGSEGV`, `SIGKILL`, `SIGABRT`…). A segfault, an OOM kill and an
  `exit(0)` all have no exit code; the signal is what tells them apart.
- **`stderr_tail`** — the last 30 lines the app wrote to stderr before
  dying (the panic, the stack trace), so you don't have to open the error
  log and find the right spot.

Both sit next to `exit_code` and are cleared by the next successful spawn,
so a running app never shows a stale death.

```console
$ 3pm show worker
┌───────────────────┬────────────────────────────────────┐
│ status            ┆ stopped                            │
│ exit signal       ┆ SIGSEGV                            │
│ name              ┆ worker                             │
│ instance          ┆ 0                                  │
│ pid               ┆ N/A                                │
│ cpu               ┆ 0.0%                               │
│ mem               ┆ 0b                                 │
│ …                                                      │
└───────────────────┴────────────────────────────────────┘
last stderr before exit (2 lines):
  thread 'main' panicked at src/main.rs:42
  called `Option::unwrap()` on a `None` value
```

`3pm show` prints the signal as a row (or `exit code` when the app exited on
its own) and the stderr tail under the table. In machine-readable form:

```console
$ 3pm ls --json | jq '.[] | {name, exit_code, exit_signal, stderr_tail}'
{"name":"worker","exit_code":null,"exit_signal":"SIGSEGV","stderr_tail":["…"]}
```

The same context rides the `exit` lifecycle event on the bus, which
`3pm logs --json` now forwards — an agent or GUI following the stream gets
the cause of death in the same line that announces it:

```json
{"event":"process","id":2,"name":"worker","kind":"exit","exit_signal":"SIGSEGV","stderr_tail":["…"]}
```

Bounds: 30 lines, 1 KiB per line, 4 KiB per process — enough for a stack
trace, small enough that the whole process table stays cheap to ship. The
tail is captured before the exit is reported (3pm waits, briefly, for the
stderr pipe to drain), so the last line — usually the one that matters — is
there. `exit_signal` is distinct from `last error`: the latter is 3pm's own
diagnostic when the app could not even be spawned (missing file, bad cwd).

## Orphans: `3pm ls` names what a dead daemon left running

**What you see.** Kill the daemon uncleanly (`kill -9`, OOM) and its apps
live on, unmanaged: the next daemon starts empty, `3pm ls` shows an empty
table, and the port you are staring at is still taken. Now the table is
followed by a warning that names the survivors:

```
[3PM] 1 process from a previous daemon still running unmanaged: api (pid 4321)
[3PM] They keep their ports. Run `3pm resurrect` (or `3pm start <name>`) to reconcile them.
```

The label is the pid file's stem under `pids/` (`api`, `worker-2` for
instance 2); past six survivors the list is elided with `+N more`.

**How it works.** Every spawn records `pid + process start time` in
`pids/<app>.pid`. `ls` asks the daemon (`orphans` request) for a read-only
scan of that directory: a file no managed row owns whose identity is still
alive is an orphan; a dead or reused pid is unlinked on the way. The scan
runs off the actor thread and the result is re-filtered against the rows
managed at reply time, so a `start` racing the scan is never reported. It
only reports: `resurrect` and `start` are what SIGKILL the orphan's group
and start a fresh copy (never two copies on one port). `ls --json` is
unchanged (a bare array); an older daemon answers `unknown_request` and
`ls` prints nothing extra.

## Watch mode: restart on file change

`--watch` turns 3pm into a dev loop: edit, save, the app restarts.

```console
$ 3pm start server.js --watch --ignore-watch '*.log' --watch-delay 300
$ 3pm ls
┌────┬────────┬──────┬────────┬───┬────────┬──────────┬──────┬────────┬───────┐
│ id ┆ name   ┆ pid  ┆ uptime ┆ ↺ ┆ status ┆ watching ┆ cpu  ┆ mem    ┆ ports │
╞════╪════════╪══════╪════════╪═══╪════════╪══════════╪══════╪════════╪═══════╡
│ 0  ┆ server ┆ 4021 ┆ 2s     ┆ 0 ┆ online ┆ enabled  ┆ 0.0% ┆ 42.1mb ┆ 3000  │
└────┴────────┴──────┴────────┴───┴────────┴──────────┴──────┴────────┴───────┘
```

- **What is watched**: `--watch` alone is the working directory,
  recursively; `--watch=src --watch=config` names specific dirs or files
  (relative to the cwd, must exist). Ecosystem files take `watch = true`,
  `watch = "src"` or `watch = ["src", "config"]`.
- **What is ignored**: dotfiles/dot-dirs (`.git`, `.env`) and
  `node_modules`, always. `--ignore-watch GLOB` / `ignore_watch = [...]`
  *adds* patterns: a bare name or glob (`*.log`, `tmp`) matches any path
  segment; a path (`logs/`, `src/generated`) is anchored at the watched
  root. An ignored directory hides its whole subtree.
- **When it restarts**: after a quiet period of `--watch-delay` ms
  (floor 100ms), measured from the *last* change — a save through a temp
  file or a `git checkout` is one restart. All instances of the app
  restart together (graceful stop, then respawn). Changes that land while
  the app is mid-restart are not queued.
- **How**: OS events (inotify on Linux, FSEvents on macOS, kqueue on
  BSD). If the native watcher can't be set up — typically inotify's
  `max_user_watches` on a big tree — 3pm polls every 2s instead and says
  so under the `ls` table and in `show`:

  ```console
  ↳ server watch: native watcher unavailable (No space left on device (os error 28)); polling every 2000ms
  ```

  `--watch-poll MS` / `watch_poll` forces polling from the start (NFS,
  container bind mounts, anywhere events don't arrive).
- **Toggling**: `3pm restart server --watch` flips watch mode (off → the
  cwd, on → off) and restarts, like pm2. `3pm start server.js --watch` on a
  running app *sets* it (the spec is replaced). `stop` turns watch off,
  as pm2 does: a later `start <name>` or `restart` runs unwatched until
  `--watch` (or a file that sets it) asks again; `save`/`resurrect`
  keep the current setting.

`3pm show` lists the roots, the effective ignore list and the delay:

```console
│ watch & reload    ┆ enabled (/srv/app)                                 │
│ ignore watch      ┆ .*, node_modules, *.log                            │
│ watch delay       ┆ 300ms                                              │
```

Differences from pm2, on purpose: one watcher per app rather than one per
cluster instance; `ignore_watch` extends the defaults instead of replacing
them (pm2's `ignore_watch: ["logs"]` silently re-watches `node_modules`);
`watch_delay` is milliseconds on the CLI too (pm2: seconds on the CLI,
ms in files); and a watcher that could not start is visible, not a line
in the daemon log.


## Logs: destinations, timestamps, hot rotation

Every instance gets its own files (`logs/<app>-out.log`, `-error.log`;
`<app>-<i>-…` from instance 1) holding the app's output byte for byte.
The knobs are pm2's, with the sharp edges filed off:

```console
$ 3pm start server.js --time                 # "2026-08-25T16:43:54: line" in the file
$ 3pm start server.js -l /var/log/app.log    # one merged file for both streams
$ 3pm start worker.js -i 4 -o '/var/log/w/{name}-{instance}.out' -e NULL
```

- `{name}`, `{instance}`, `{namespace}` expand in custom paths. A custom
  path *without* `{instance}` on a multi-instance app is infixed
  `-<instance>` (instance 0 keeps the path as written) and `3pm start`
  says so — sharing one file between instances is never what you meant.
- `NULL` drops the file but not the stream: `3pm logs` still follows it.
  `--disable-logs` drops file, stream and crash tail (the pipe is still
  drained so the app never blocks).
- `--attach` on `start`, `restart` and `stop` (pm2): after the action
  and the table, stream the affected apps' logs live — no history, Ctrl-C
  to leave, clean exit when the daemon shuts down. Conflicts with
  `--json` (`3pm logs --json` is the machine stream).
- `log_date_format` is validated at `3pm start` (a `%Q` fails there, not
  in every log line); there is no CLI-only default, so an ecosystem app
  and a flag-started app behave the same.

**Rotation happens while the app runs.** The daemon counts what it
writes and rotates the file the moment it crosses the threshold, under
the same lock every writer shares — so the line that crossed is in `.1`,
the next one is in the fresh live file, and a follower on the bus sees a
`log_rotated` event between the two. Nothing is ever lost to a restart
that never came.

```console
$ 3pm logs api --size
0|api | out       3.1 KB  /home/u/.3pm/logs/api-out.log  (+2 rotated, 38.2 KB)
0|api | err         0 B   /home/u/.3pm/logs/api-error.log
$ 3pm logs api --lines 500 --rotated     # history continues into .1, .2.gz…
$ 3pm rotate api                         # now, regardless of size
$ 3pm flush                              # empty every file in place
$ 3pm reloadLogs                         # after an external mv (logrotate create mode)
```

Policy layers, most specific first: per-app `log_rotate_*`, then the
daemon's `THREEPM_LOG_MAX_SIZE|_FILES|_DAYS|_COMPRESS`, then `[logs]` in
`~/.3pm/config.toml`, then 20 MiB / 5 files / 14 days / no gzip. `"0"`
switches rotation off. With `compress`, `.2` and older are gzipped and
`.1` stays plain. An hourly sweep applies the count/age limits to files
that rotate rarely. `3pm internals` reports the effective policy, open
files, bytes written and rotations.

Reading has its own timestamps: the bus carries `ts` (arrival, epoch
ms) on every line, so `3pm logs --timestamp[=FMT]` stamps a stream
without touching the files, `--format` gives logfmt (`ts= app= id=
instance= type= message=`), and `--json` gives the wire events — tail
and stream in one NDJSON schema. `3pm logrotate --print` renders a
logrotate(8) drop-in for hosts that centralize rotation there.


## Log shipping (`3pm agent`, `[logs]` in link.toml)

**What you see.** A linked node sends its logs to the fleet backend by
default: no setup, no per-app setting. The backend keeps the counts of
every kind of line and the lines that carry news, and answers them with
`logsQuery` (see the backend's `FEATURES.md`). `3pm agent status` (or
`3pm link status`) has a `logs` row:

```
logs         24 sources (8 app, 15 file, 1 journal), 451 lines read, 33 shipped, 33 templates; batches 16 ok, 0 retried, 0 dropped
```

It adds `; skipped …` (sources not read, by reason), `; lines dropped …`
(by reason: `budget`, `lag`, `ring`, `stale`) and `; last error: …` when
there are any. The running agent writes it to `<home>/agent/logs/status.json`.

**What is shipped.**

- **The 3pm apps' output**, read from the daemon's bus (a second
  subscription, its own cursor: a flood lags only it). ANSI stripped,
  lines cut at 8 KiB. `--disable-logs` apps publish nothing, and the
  agent's own app is never shipped. Every OS.
- **The system journal** (Linux), entries at `info` or worse
  (`host_level`), kernel lines and Docker containers using the `journald`
  driver included. Read by one `journalctl --follow --output=json` child;
  first start from now, then from the cursor of the last batch the
  backend accepted. Entries older than 24 h are dropped on read.
- **Other programs' log files** (Linux), found by their writer: once a
  minute the host agent lists the regular files held open for writing
  with `O_APPEND` (nginx, mysqld, a Python bot writing in its home,
  dockerd's `<id>-json.log`), with the writer's pid, name, systemd unit
  and container, plus `dpkg.log`, `apt/history.log` and
  `unattended-upgrades/*.log` (written and closed at once). When the
  root-installed `automonit` module runs, its `logs` action (asked every
  5 min) adds what a service says about itself: nginx's and Apache's
  configured logs, MySQL's `log_error`, PostgreSQL's log directory,
  MongoDB's `systemLog.path`, Redis's `logfile`, with their format
  (`nginx-combined`, `postgres-prefix`, `mongo-json`…).
  The host agent lists 2 000 files at most, ranked by the writer's real
  uid: root's and the fixed list first, then system accounts' (uid
  below 1 000) outside the daemon's cgroup, then the room left shared
  evenly between users, subordinate uids (100 000 and up) counting as
  one (one user holding thousands of files cannot push `/var/log` or
  another user's files out). A pass reads at most 4 096 descriptors per
  process, root's processes first, and stops after 20 s with what it
  found; one held up by a hung filesystem is served when it ends.
  A file is tailed from its end at first sight (no history), follows
  rotation and `copytruncate`, and is parked after 10 min of silence;
  512 files at most.

**What is not.** The macOS unified log (on a Mac only the apps' output
leaves). Files the daemon's user cannot read: the agent runs as the
daemon's user, an `EACCES` marks the source skipped, and the next pass
(60 s) tries again — a user added to `adm` starts shipping without a
restart. And what the host agent's verdict keeps back:

| verdict | files |
|---|---|
| `duplicate_journal` | written by `rsyslogd`, `syslog-ng`, `systemd-journald` (`/var/log/syslog`, `auth.log`, `kern.log`: the journal, line for line) |
| `duplicate_bus` / `own` | the 3pm apps' log files (already on the bus), `3pm.log`, `hostagent.log` (by path, whoever holds them, a `3pm update`'s new daemon included: the home's `3pm.log`/`hostagent.log` are `own`, anything under its `logs/` or the `logs/` of a home it hosts (`[users] allow`, as the daemon accepts it) is `duplicate_bus`) |
| `binary` | a NUL byte in the first 4 KiB |
| `private` | shell and REPL histories (`.histfile`, and a `history` anywhere under a dot-directory), `.lesshst`, `lesshst`, `.viminfo`, `*.shada`, anything under `.ssh/` or `.gnupg/` |
| `unknown` | a verdict this build does not know |

Only `ship` is read.

**How it is reduced on the node.** Every line goes through, in order:
redaction (secrets replaced by `<REDACTED>`), template mining (Drain: a
line becomes a template with its variables masked, ids per source, kept
across restarts), then the policy. Always sent: each template's text when
it appears or changes, and its count per 5-minute window. A raw line
only when it is news (`why`): a template never seen before (`new`, at
most 30 a minute per source), warning or worse once per template per hour
(`error`), an important template back after 10 min of silence (`back`),
a rate spike (`spike`) or a return after a long gap (`return`), and the
30 s before a new error (`context`). Then a budget per source, 4 KiB/s
with a 256 KiB burst; what is over it is dropped and counted. A batch
leaves every 10 s (gzip NDJSON, `POST /api/buckets/{cluster}/logs`,
never on the WebSocket); while the backend is away, lines wait in a 4 MiB
memory ring. There is no disk spool: the files and the journal are the
spool, and their positions advance only when the backend answers `200`,
so a line read but not accepted before a stop is read again (and stored
once). On the PoC's corpora, 0.4 to 8 % of the raw bytes left a busy
host, 14–20 % a quiet journal.

**Who decides.** Both ends. The backend says `welcome.logs {apps, host,
files}` when it has an events store; without it the agent ships nothing.
The host has the last word in `<home>/agent/link.toml`, every key
optional (a file written before logs reads as the defaults):

```toml
[logs]
apps    = true      # the 3pm apps' output
host    = true      # the journal
files   = true      # other programs' log files
host_level = "info" # journal ceiling: debug, info, notice, warning, err, crit
exclude = []        # globs (`*`, `?`) on app names and file paths
```

A source flows only when both say yes. A misspelt `host_level` closes to
`crit`, a malformed `[logs]` ships nothing (the agent still runs) — an
unknown key counts as malformed, so a misspelt `file = false` closes
every switch instead of leaving `files` on —, and
`3pm link --force` keeps the section. The agent reads it at start:
`3pm restart agent` after an edit.

**Security notes.**

- Redaction is a pattern list: URL passwords, `Authorization` and cookie
  headers, `password=`/`token:`/`"secret":` pairs, command-line
  passwords (`--password`, `mysql -p`, `sshpass -p`, `curl -u`), bearer
  and Basic credentials, JWTs, PEM private keys, known token shapes (3pm
  `3pmu_`/`3pma_`/`3pmk_`, GitHub, Slack, Stripe, npm, AWS, Google), crypt
  hashes, card numbers. Public ids (`3pmn_`, `3pmc_`), numeric token
  counts and plain words after `password` (`password for root`) are kept.
  A secret in a shape the list does not know is shipped, and so is one
  split over lines: redaction reads a line at a time (a YAML env entry
  with `- name: DB_PASSWORD` on one line and `value:` on the next ships
  its value).
- A file is opened `O_NOFOLLOW`, then checked by `fstat` against the
  `(dev, inode)` the host agent declared: a symlink swapped in, or a
  writer in a container naming a host path, gets nothing read. A
  rotation successor must be a regular file of the same owner.
- Only the root-installed `automonit` module is asked (an app a hosted
  user names `automonit` is not), and a file automonit adds alone must
  look like a log (`/var/log`, a `log`/`logs` directory, `*.log`), sit
  outside `/etc`, `/root` and the like, have one link and the owner it
  declared.
- `private` files are never read, whoever writes them.

**Stop.** At a stop the agent closes its open windows and tries a last
batch within 5 s; its row's stop grace is 10 s (`kill_timeout_ms` in the
agent's `module.toml`, raised on existing rows by `3pm update`).

**Where it comes from.** Wire: `threepm-link::logs`,
`docs/backend-agent/link-protocol.md` §6b. Files by writer:
`crates/threepm-daemon/src/hostmetrics/logfiles.rs` (`Request::LogSources`).
The pipeline: `modules/agent/src/logs/` (`docs/backend-agent/agent.md`,
"Logs"). automonit's side: `3pm-automonit/docs/logs.md`.


## Interactive apps: `tty = true` and `3pm attach`

Some programs want a real terminal: `claude`, `codex`, a REPL, `vim`, a
shell. Started with `--tty`, an app runs on a pseudo-terminal the daemon
holds, and `3pm attach` connects any terminal to it, as tmux would —
with everything 3pm already does (restart policy, logs, crash tail,
hosted users, `3pm update` without a restart) unchanged.

```console
$ 3pm start claude --tty                       # or tty = true in ecosystem.toml
$ 3pm attach claude                            # its screen appears, your keys go to it
[3PM] attached to claude (pid 4242, 120x40) — Ctrl-Z detaches
…
[3PM] detached
$ 3pm attach claude --peek                     # the screen as text, then out (--json: a document)
$ printf 'ls\n' | 3pm attach sh                # no terminal: stdin in, the app's bytes out
$ 3pm attach claude --read-only                # watch: no keys, no resize
```

**The app.** `isatty` is true on 0, 1 and 2; the app leads a session of
its own with the terminal as its controlling tty (`setsid` +
`TIOCSCTTY`, `login_tty(3)`'s recipe), so `3pm stop`/`kill` signal its
group as before. The terminal starts at 120×40 with `stty sane`
settings plus `IUTF8` and `VERASE` = DEL; `TERM=xterm-256color` and
`COLORTERM=truecolor` are set unless `env` names them. It follows the
size of the last client that attaches or resizes and may type (bounded
to 2..=1000 each way, resizes coalesced over 50 ms).

**What the daemon keeps.** One screen per `tty` app, with 1 000 rows of
history, fed by everything the app writes, whether a client is attached
or not. It answers the questions an app asks its terminal (cursor
position, `CSI c`/`CSI > c`, XTVERSION, window size, DECRQM, the
default colours `OSC 10/11/12`), so codex and claude start at full speed
with nobody watching. The screen outlives an exit: attaching to an app
that stopped shows its last screen, and the client waits for the next
incarnation (`restarted`, then a fresh screen).

**What a client gets (`rendered`, the default).** Screens the daemon
draws, never the app's bytes: each client has a mirror of its own
terminal and receives the difference, immediately when it is idle, else
every 16 ms, and not in the middle of an app's synchronized update
(`?2026`: held until it ends, 100 ms at most). Rows that scroll off the top of the app's main screen land in
**your terminal's own scrollback** — scroll up through a Claude Code
conversation with the mouse wheel — and on attach, the history held so
far comes first. The alternate screen is entered and left when the app
does (vim leaves your shell's screen as it was); title and bell are
relayed, the cursor shape too. `--raw` relays the app's bytes as written
instead, for what the rendered view does not model (at your terminal's
risk: its answers to the app's queries go back too). `3pm attach
--peek --json` reports `unmodelled`, the count of sequences the screen
did not understand: above 0, `--raw` shows more.

**Leaving.** `Ctrl-Z` detaches, always, and is never sent to the app
(in `--raw` also its kitty-keyboard and modifyOtherKeys spellings, which
an app may turn on). `SIGTERM`, `SIGHUP`, `SIGINT` and `SIGQUIT` sent to
the client detach too. Whatever the way out, the terminal gets back its
termios, scroll region, colours, cursor keys and keypad, mouse modes,
bracketed paste, focus events, keyboard protocols, a visible cursor of
the default shape, and its main screen. Exit codes: 0 detached, 2 no
such app or not a `tty` app, 3 the app was deleted (or re-specified
without `tty`) or the daemon went away, 1 anything else (a client cut
off as too slow, `input_backlog`).

**Without a terminal (pipe mode).** When stdin or stdout is not a
terminal, stdin goes to the app until its end, the app's bytes come out
unrendered, and the command ends once the app had its input and fell
silent for 150 ms (500 ms after it had it at most: a TUI animates
forever), or when it exits. A large stdin goes at the app's pace.
`--paste` sends stdin as one bracketed paste when the app turned
pasting on (a prompt takes several lines as one message).

**Logs.** A terminal has one output stream: the app logs to one file
(`<app>.log`, or the `log_file`/`out_file` you name; `error_file` is
refused), stripped of escape sequences, cut into lines, a move to the
right after text read as a space — `3pm logs claude` reads as text. The
crash tail is the last output (`3pm show`: "last output before exit").
`disable_logs` drops the log of a TUI that redraws too much; the screen
is kept either way. Keystrokes are never logged; what the app displays
is, like any app's output.

**Across `3pm update`.** The terminal crosses the daemon's `execve` like
the pipes do: the app keeps its pid, its terminal and its screen (main
and alternate, ≤ 48 KiB; beyond that the app is asked to repaint by a
resize), and an attached client re-attaches by itself (`[3PM] the
daemon was updated: re-attached`).

`3pm show` has a `tty` row (`yes (1 viewer)`, or how to attach), and
`3pm ls --json` a `tty_viewers` count. The Node SDK has no `attach`
(a terminal session is not a script's call).

**Limits, deliberately.**

- **A `tty` app dies with a daemon that crashes**: the kernel hangs up
  the terminal (`SIGHUP`) when its last master closes. A clean
  `3pm update` keeps it; a future keeper process holding the master is
  the remedy.
- `Ctrl-Z` never reaches the app from a client's keyboard (in pipe mode
  a `0x1A` in stdin is sent as is).
- In `rendered` mode these do not pass: the kitty keyboard protocol and
  modifyOtherKeys (Shift+Enter arrives as Enter in Claude Code), focus
  events (`?1004`), `?2031` colour-scheme reports, OSC 8 hyperlinks,
  OSC 52 clipboard, kitty/sixel graphics. `--raw` passes them.
- The pty has one size: a client smaller than it sees a cut screen
  (the size follows the last client that attached or resized, not the
  smallest; `--read-only` clients never resize).
- A client that stops reading for more than 10 s (a suspended
  terminal) is cut off; so is one that lets 32 lifecycle notes pile up
  (`too_slow`), or types faster than the app reads (256 KiB,
  `input_backlog`).
- `3pm update` does not carry the history (the next rows scroll into
  the client's scrollback as before) nor origin mode (`DECOM`).
- Version skew: a new CLI asking an 0.37 daemon for `--tty` gets an app
  on pipes (the field is unknown there, and ignored); `3pm attach`
  then says to run `3pm update`. Handing a daemon with a `tty` app over
  to an older image is refused at the pre-check (handoff manifest
  version 2) and `3pm update` falls back to a cold restart; any row with
  `tty`, stopped ones included, asks for version 2 (the conservative
  choice).
- First-run dialogs (claude's workspace trust, codex's update prompt)
  wait for a human to attach and answer them.
- Memory: about 3.7 MiB per `tty` app at 120 columns (the history),
  plus about 1 MiB per attached client (its mirror).

**Where it comes from.** Plan `3pm-rd/3pm-tty-attach-plan.md` (v2);
`threepm-core::{spec (tty), ansi, paths::log_targets}`; the wire in
`threepm-protocol::{message (Attach, Attached, NotTty), shared::attach,
live_io}` and `docs/protocol.md` (§ *Attach*); the daemon in
`crates/threepm-daemon/src/tty/` (`pty`, `screen`, `screen_saved`,
`render`, `viewer`, `writer`, `history`, `attach`),
`child_task_tty.rs`, `supervisor/{tty,pumps,adopt}.rs` and
`rpc_server.rs` (the connection takeover); `vt100` 0.16.2 patched in
`vendor/vt100/` (`PATCHES.md`); the CLI in
`commands/attach{.rs,/session.rs,/term.rs}` and `render/attach.rs`.
Tests: `crates/threepm-cli/tests/e2e_tty.rs` (real ptys: a test
terminal, `Ctrl-Z`, signals, pipes, peek, vim and a shell across
`3pm update`), `spawner_tty_tests.rs`, `tty/*_tests.rs` (the responder
table, exact history, a differential proptest of the rendered view
against a large terminal model), `supervisor/tests/tty{,_attach,_handoff}.rs`.

## Start order: `depends_on`, `start_order`, `start_delay`

```toml
[[apps]]
name = "db"
script = "./db.sh"
wait_ready = true            # db says when it is really up

[[apps]]
name = "api"
script = "./api.sh"
depends_on = ["db"]          # spawned only once every db instance is online
depends_timeout = 30000      # ms; 0 = wait forever (default 60000)

[[apps]]
name = "web"
script = "./web.sh"
depends_on = ["api"]
start_delay = 2000           # ms before the first spawn of a start/restart
start_order = 10             # tie-break among apps started together
```

`3pm start ecosystem.toml` orders the file into waves (a dependency
cycle or an unknown name is an error *before* anything starts) and
starts wave n+1 once wave n is online — `online` means past `wait_ready`,
so "started" means "ready", not "forked a millisecond ago" (OxMgr only
sorts the spawn order). `resurrect` does the same at boot.

The daemon enforces it for single starts and restarts too: an app whose
dependency is not online is **parked** in `launching` (no pid, `show`
says `waiting for: db`, `ls --json` carries `waiting_for`) and polled
every 500 ms. Stop and delete act on a parked row immediately; restart
re-parks it. After `depends_timeout` it starts anyway with a notice in
`last error` — a dead database must not keep the dashboard down
forever. `restart db` does not cascade to `api`.

`stop_exit_codes = [0, 3]` parks the app in `stopped` on those codes
whatever `restart_policy` says (a signal death has no code and never
matches). `3pm reset <app|all>` zeroes the ↺ counters without touching
the process.

`stop`, `restart`, `delete`, `reset` and `logs` take **several targets**
at once (pm2: `3pm delete 0 1 api`), mixing names, ids, `all` and
ecosystem files. `all` absorbs the rest of the list and duplicates fold,
so a row is never acted on twice. A target that resolves to nothing does
not cost the others: the batch acts on every target that does resolve,
notices each one that did not, and exits with the failure code at the
end. `logs` is the exception — an unmatched name is a notice and the
rest still stream; only an empty selection is an error.


## Config workflow: profiles, `validate`, `apply`, `--update-env`

```toml
[defaults]                     # every app starts from here
restart_policy = "on_failure"
log_rotate_size = "50M"

[[apps]]
name = "api"
script = "./server.js"
[apps.env]
NODE_ENV = "development"
[apps.env_production]          # pm2 sugar: env only, with --env production
NODE_ENV = "production"
[apps.profiles.staging]        # any field, with --env staging
instances = 2
[apps.profiles.staging.env]
NODE_ENV = "staging"
```

Resolution order: `[defaults]` → `[[apps]]` → `[apps.profiles.<name>]`
→ `[apps.env_<name>]`. Scalars from the later layer win, `env` merges by
key, lists (`args`, `watch`, `depends_on`, …) are replaced. `--env
<name>` selects the profile on `start`, `apply` and `validate`; a name
the file never defines is an error (pm2 silently ignores a missing
`env_production` — a typo would ship dev settings). `--only a,b` limits
any of the three to a subset (a dependency outside it must already be
managed). The chosen profile and the file path are recorded on the spec
(`show`: `config file`, `profile`; `ls --json`: `spec.source`) and
survive `save`/`resurrect`.

`3pm validate eco.toml` parses, resolves the profile and checks the
semantics without a daemon: unknown fields are **errors** here (a
did-you-mean each), plus per-app spec errors, duplicate names,
dependency cycles, and `depends_on` names outside the file (a warning;
`--check-managed` asks the daemon and makes it an error). Exit 1 on any
error; `--json` is the finding list.

`3pm apply eco.toml` reconciles the daemon with the file and is
idempotent: apps not managed are **created**, apps whose configuration
differs are **updated** (restarted with the new spec — the reason names
the changed fields), apps where only `instances` differs are **scaled**
(rows added or removed, survivors keep their pid; `instances = 0` is
compared as the daemon resolves it, one per CPU, so it never reads as a
change), stopped apps with the same configuration are **restarted**,
everything else is **unchanged** and keeps its pid. `--prune` deletes
managed apps that came from this file but are gone from it — an app
that is still in the file but outside `--only` is *not* gone, and an app
started by hand is never pruned (`--prune-all` does that too, and
refuses `--only`); `--dry-run` prints the plan and exits. `3pm start
eco.toml` keeps pm2's behavior — it always restarts — so habits and
scripts don't change; `apply` is the idempotent verb. Re-applying with
no `--env` an app that was applied with a profile drops the profile and
says so.

Loader rules shared by all three: `--only` is decided on the declared
names before anything resolves (a broken app outside the subset never
blocks; a broken selected app reports its own error); a name declared
twice is an error; env values may be bare numbers or booleans (`PORT =
3000`, stringified); a malformed `env_<name>` table is that app's error
only; `[defaults]` cannot carry `name`, `script`, `profiles` or
`env_<name>` (each is a notice, never silent). `validate` needs neither
a daemon nor a socket-capable `THREEPM_HOME` (only `--check-managed`
connects). Values expand `~` (leading) and `$VAR` / `${VAR}` in `cwd`,
`script`, `interpreter`, `args`, `watch`, the log paths and `env` values
(`$$` is a literal `$`); a variable that is not set is that app's error
— never a silent empty string. `stop`, `restart` and `delete` also
accept a file: every managed app it declares is acted on, an unmanaged
name is a notice.

`3pm restart api --update-env` re-reads the file the app came from
(same profile) and restarts it on the fresh spec, always. An app started
from the command line has no file: the error says so.

## Reconfigure on restart: `3pm restart <app> [start flags]`

pm2's attribute update. `restart` accepts every `start` flag; what you
type is folded into the app's stored configuration, and the app restarts
on the result:

```console
$ 3pm restart api --name web            # renamed in place, then restarted
$ 3pm restart 25 -i 4 --max-memory-restart 300M
$ 3pm restart web --env LOG=debug       # added to the environment, not replacing it
```

What the flags do not name is left exactly as it was — script, args,
interpreter, cwd, log destinations, the ecosystem file the spec came
from. `--env K=V` **merges** into the app's environment (pm2), every
other list-valued flag replaces. The new configuration is the app's from
then on: it is what `show`/`ls --json` report and what `save` persists.

The fold is **per app**, never per row: a spec belongs to a name that
owns N instances, so `3pm restart 25 --name web` (id 25 being one
instance of `api`) renames and restarts the whole app — same ids, same
instance numbers, restart counters kept, pid files and log files moved
along with the name (as `3pm rename` does, below). `-i N` reconciles
the count exactly like `start` does: missing instances are created,
surplus ones deleted, survivors restart on the new spec.

Refusals leave everything running as it was: a `--name` whose target
matched several apps (`restart all --name x` would collapse the fleet
into one name), a name another app already carries, a `--name` on an app
a module owns (`install`/`uninstall` match the manifest on that name —
every other flag applies to a module app as usual), and any patch that
cannot become a valid spec (`--kill-signal SIGNOPE`) — checked for every
matched app before the first one restarts.

`--watch` alone keeps pm2's meaning (toggle watch mode: off → the cwd,
on → off); `--watch=PATH` sets the watched paths like `start` does.
`--update-env` restarts on what the ecosystem file says and cannot be
combined with flags — the two ask for different specs.

## Rename in place: `3pm rename <app> <new-name>`

Renames an app **without restarting it**: same ids, same pids, same
restart counters and instance numbers — the process never sees a
signal.

```console
$ 3pm rename api web        # by name
$ 3pm rename 25 web         # by the id of any of its instances: the whole app
$ 3pm rename api web --json # {"affected": [...]}, under the new name
```

What carries the name follows it:

- **every instance** of the app (a spec belongs to a name, never to one
  row);
- **the pid files** (`pids/web.pid`, `web-1.pid`, …);
- **the log files named after the app** — the defaults
  (`web-out.log`, `web-error.log`, `web.log` with `merge_logs`, per
  instance) and a custom path that uses `{name}` — with their rotated
  generations (`.1`, `.2.gz`, …). The file is renamed under the open
  descriptor, so the app writes on into the moved file without a gap
  and `3pm logs web` shows its whole history. A custom path without
  `{name}` is not the name's and stays where it is;
- the name on the event stream (`3pm events`, the `Log` lines of the
  running process included) and in `ls`/`show`;
- watchers and cron restarts (re-armed under the new name; a cron's
  next run is recomputed).

What does **not** follow until the app's next spawn: the `name`
variable the running process was started with, an action call already
in flight, a terminal viewer already attached (its title). Other apps'
`depends_on`, and the ecosystem file the app came from, still say the
old name: `restart --update-env` then looks the new name up in the
file.

Refusals change nothing: a target matching several apps (`all`, a
pattern — refused by the CLI before it asks), a name another app
carries, a module's app (a module names its own apps), an invalid name,
and a log file already at the new name (a deleted app's history left
without `--flush`): nothing restarts to open a fresh file, so moving
onto it would clobber it — remove it first. A hosted user renames its
own apps and they stay its own. `3pm restart <app> --name <new>` takes
the same rename step, then restarts. Node SDK: `client.rename(target,
name)`.

## Event stream: `3pm events`, server-side filters, shutdown announce

**What you see.** One line per daemon event — lifecycle transitions
(with exit code / signal / stderr-tail size), log lines, rotations, lag
markers and the shutdown announce — filtered *by the daemon* before it
serialises anything, so a chatty app costs nothing to a follower that
did not ask for its output:

```console
$ 3pm events -f 'process:*' -p api          # lifecycle of one app only
12:34:56 process         3|api     online
12:35:10 process         3|api     exit code=1 signal=SIGSEGV stderr_tail=2 lines
$ 3pm events -f log:err -f log_rotated       # globs are repeatable
12:36:02 log:err         3|api     warn: slow query
$ 3pm events --json | jq .                   # raw wire events (NDJSON)
$ 3pm kill                                   # in another shell → the follower ends:
12:35:40 daemon_shutdown           reason=kill
[3PM] daemon shutting down (kill)
```

Event names: `process` (also `process:<kind>` — `start`, `online`,
`exit`, `restart`, `stop`, `delete`, `errored`), `log` (also
`log:<stream>` — `out`, `err`; the v1 spellings `log_out`, `log_err`
and `log_*` are accepted as patterns for one release), `log_rotated`,
`lagged`,
`daemon_shutdown`; `*`/`?` globs. `-p` scopes
app events to one name; daemon-level events always pass. `--raw` drops
the time column and chrome; `--timestamp FMT` changes it.

A follower (`events` or `logs`) exits **0** with the reason when the
daemon is stopped on purpose (`3pm kill`, SIGTERM, home removed) and
non-zero with "daemon connection closed unexpectedly" when the daemon
died without announcing — the difference a supervisor script wants.
`3pm logs` uses the same filter under the hood (selected ids + the kinds
its view shows).

**How to get it.** Nothing to configure: any client of `pub.sock` can
send one `EventFilter` JSON line right after connecting (`docs/protocol.md`,
*Subscription filter*); clients that send nothing get everything, as
before.

**Where it comes from.** `EventFilter` in `threepm-protocol::filter`
(pure, `*`/`?` matcher shared with watch ignores), `bus::stream_events`
(handshake + drain-on-cancel) and `Supervisor::begin_shutdown` (the
single announce) in `threepm-daemon`, `commands/events.rs` +
`render/events.rs` in the CLI.

## Application metrics and events (`3pm show`, `3pm events -f app`)

**What you see.** pm2's custom metrics and events, for any language and
any number of processes per app. A process of the app connects to the
control socket, names itself and reports; `3pm show` grows a section:

```console
$ 3pm show saw
 … the usual describe table …
 Custom metrics
┌───────────┬──────────────────┬────────────┬───────────┬──────────────────────┬──────┬─────┐
│ source    │ name             │ labels     │ kind      │ value                │ unit │ age │
├───────────┼──────────────────┼────────────┼───────────┼──────────────────────┼──────┼─────┤
│ cv-engine │ cv.xruns         │            │ counter   │ 12                   │      │ 0s  │
│ cv-engine │ cv.callback      │            │ histogram │ n=48210 p50≈1.2 p99≈4│ ms   │ 0s  │
│ renderer  │ http.requests    │ method=GET │ counter   │ 5321                 │      │ 1s  │
└───────────┴──────────────────┴────────────┴───────────┴──────────────────────┴──────┴─────┘
$ 3pm show saw --json | jq '.[0].app_metrics.metrics[] | .def.name'
$ 3pm events -f app
12:34:56 app             3|saw     [cv-engine] midi.port_lost warn ×3 MidiError: connection lost {"port":"ES-9"}
```

Three kinds — `gauge`, `counter` (cumulative, rates at read time) and
`histogram` (base-2 exponential buckets, quantiles at read time so a
fleet can add them up) — with up to three labels each, a unit and a
description. Events carry a name, a level, flat attributes, an optional
exception with its stack trace (machine paths stripped) and a
fingerprint; a crash loop is rate-limited per fingerprint and shows up
as `×N` on the next event that passes, never as a flood.

A process that makes series from what it finds (one per service a
monitoring module discovers) declares **families** — a name with no
series until a labelled child exists — **removes** a series when its
subject is gone (the daemon drops it after its two-minute TTL), and
reads its own **usage** to stay within the caps (Rust SDK 0.37;
`packages/rust/ROADMAP.md` for the other languages). The daemon keeps a
series' unit and description: it announces `THREEPM_CTL_CAPS=sticky_meta`
to its apps, and an SDK that reads it sends them once per connection.

**How to get it.** Nothing is injected into the child: the app embeds
an SDK — the same surface in four languages (`docs/app-metrics.md`):

```js
const { io } = require('@3pm/api');                 // Node / Electron (packages/node)
io.init({ source: 'worker', builtins: true });
io.counter('http.requests', { labels: { method: 'GET' } }).inc();
io.histogram('http.latency', { unit: 'ms' }).observe(12.5);
io.notifyError(err);
```
```python
import threepm_io as io                             # Python, stdlib only (packages/python)
io.init(source="cv-engine")
io.histogram("cv.callback.jitter", unit="ms").observe(0.42)
```
```cpp
#include <threepm_io.hpp>                           // C++17 header-only (packages/cpp)
auto late = threepm::io::instance().counter("host.late_callbacks");
late.inc();                                         // wait-free, fine on the audio thread
```
```rust
use threepm_io::{io, Init, Metric};                 // Rust, serde only (packages/rust)
io().init(Init::new().source("worker"));
io().counter("http.requests", Metric::new()).inc(1.0);
```

**Install.** The four SDKs are published on `https://3pm.edgecr4ft.net`
only, not on npm, PyPI or crates.io; `sdk/sdk.json` lists every version,
its files and their sha256:

```console
$ npm install https://3pm.edgecr4ft.net/sdk/node/3pm-api-0.64.0.tgz
$ pip install --index-url https://3pm.edgecr4ft.net/simple/ threepm-io
```
```cmake
FetchContent_Declare(threepm_io URL https://3pm.edgecr4ft.net/sdk/cpp/threepm-io-cpp-0.64.0.tar.gz URL_HASH SHA256=…)
FetchContent_MakeAvailable(threepm_io)
```
```toml
[registries.threepm]                                # .cargo/config.toml
index = "sparse+https://3pm.edgecr4ft.net/sdk/cargo/"
[dependencies]                                      # Cargo.toml
threepm-io = { version = "0.64.0", registry = "threepm" }
```

`just publish-sdk` (`distribution/scripts/publish-sdk.sh`) ships each SDK
whose own version is not online yet: it runs the SDK's tests, installs the
fresh archive into an empty project (npm, a venv, a CMake `FetchContent`
build, a cargo build against the sparse registry served locally), uploads
the archives before the indexes and never removes an older version. The archives are reproducible, so a `--force` republish keeps the
sha256 a `URL_HASH` pins. `--index-url`, not `--extra-index-url`: with the
latter pip would also take a same-named package from PyPI.

Outside 3pm every call is a no-op; under 3pm the SDK connects in the
background, sends its whole registry every second, never blocks, never
throws, reconnects with backoff and gives up after five refused `hello`s
(a process that outlived its incarnation). Labels are bounded in the SDK
(3 keys, 50 sets then `overflow="true"`), histograms reproduce the
daemon's bucket grid bit for bit (`packages/exphist-vectors.json`), and
Electron renderers reach the daemon through `io.ipcBridge(ipcMain)` +
`@3pm/api/renderer`. Or speak the NDJSON yourself on
`$THREEPM_CTL_SOCK` — `{"msg":"hello","id":$THREEPM_ID,
"gen":$THREEPM_GEN,"source":"worker"}`, then `{"msg":"metrics","m":[…]}`
every second and `{"msg":"event",…}` when something happens
(`examples/metrics.py`, `docs/protocol.md` "Control channel"). Every
process of a multi-process app (Electron main, a Python engine, a C++
host) opens its own connection with its own `source`; a process that
restarts reconnects under the same source and replaces its predecessor.

**Built-in metrics (Node, Electron).** `io.init({ builtins: true })`
reports the event loop (lag histogram, native p99/max, utilization),
every GC pause by kind, the V8 heap with its limit and the memory
outside it (external, ArrayBuffers), handles, CPU, major page faults and
involuntary context switches (`{ heap_spaces: true }`: the heap per V8
space). In an Electron app, `io.electron(require('electron'))` in the
main process adds every Chromium process by type (count, CPU, working
set), the power state (battery, thermal state, CPU speed limit) and
events for crashed processes, hung windows and sleep/wake;
`createRendererIo(ipcRenderer, { builtins: true })` in a preload adds the
renderer's main-thread lag and long tasks, JS heap, Blink memory, CPU
and DOM size. The full list: `docs/app-metrics.md`, "Node".

**Rust SDK (`packages/rust`, `threepm-io`).** The same metrics, events
and **custom actions** with two dependencies (`serde`, `serde_json`) and
std threads — no async runtime, so it drops into a sync program, a tokio
one or a plugin host alike. `io()` is the process's instance,
`Series` is `Clone + Send + Sync`, and no telemetry call returns a
`Result` or panics: what cannot exist comes back as an inert handle plus
a line under `THREEPM_IO_DEBUG=1`. Actions carry Node's contract —
`ctx.commit_file(ext)` for a result too big for one line, `remote`,
`timeout_ms`, `concurrent`, `output_max_bytes`, a local `ctx.input` — with
the handler's error type being anything that prints. Two things a thread
cannot do that an event loop can: a `metric` callback runs on the flush
thread, and nothing holds the process up, so a short-lived program
flushes before returning and a worker that only answers actions ends its
`main` with `io().serve()`. **Stream sessions** since 0.50.0:
`io().stream(name, Action::new().idle_ms(…), |session, params| …)` serves
one, `io().open(node, app, action, params, Open::new())` opens one, and
`Session` is `send` / `send_frame` / blocking `recv` / `end` / `close` /
`stats` / `result`, over the same `data.sock` legs and abandon sink as
Node's (one chunk per frame, no batching; on a blocked socket a frame
past 100 ms goes on its own age, never the newest, and past 8 MiB the
oldest non-`KEY` frame goes first — only the newest `INIT` is kept);
`io().node()` reads `link.toml`. Not implemented yet: `io.call`, blob
inputs/outputs (a file that crosses the fleet), the built-in
`3pm:profile`, and Node's `builtins`. Surface, bounds and every
difference: `packages/rust/README.md`.

Bounds (1 000 live series per instance, 200 names, 50 label sets then
`overflow="true"`, 10 events/s, 120 s TTL swept every 10 s…) and what happens past them:
`docs/metrics.md`, "Application metrics"; refusals are counted on the
record and shown by `3pm show`. Twenty-three label keys are reserved,
because the fleet backend sets them itself on every series: `cluster`,
`node`, `app`, `instance`, `ns`, `src`, `overflow`, `vmrange`, `le`,
`hostname`, `os`, `arch`, `agent_version`, `mount`, `window`, the host
dimensions `core`, `iface`, `blockdev`, `sensor`, `gpu`, `array`, and
the process-tree ones `proc`, `pid`.

**Where it comes from.** `threepm-protocol`: `app_metrics` (wire types),
`exphist` (the frozen bucket grid), `Request::AppMetrics` (always
answered with the instance's whole state: the daemon cuts no diff),
`Event::App`. `threepm-daemon`:
`app_metrics/{limits,validate,state,answer,events}.rs` (pure: grammars,
caps, the last value of every series under one revision that moves only
when a reader would see a change, token buckets), the control
server (one connection per source, peer pid, validation off the actor),
`supervisor/appmetrics.rs`. CLI: `render/app_metrics.rs` (the section,
approximate quantiles), `render/events.rs` (the `app` line). SDKs:
`packages/node/lib/io/` (`@3pm/api`), `packages/python/threepm_io/`,
`packages/cpp/include/threepm_io.hpp`, `packages/rust/src/` (`threepm-io`,
a standalone workspace), each tested against a fake control socket
(`just sdks`) and under a real daemon
(`crates/threepm-cli/tests/e2e_io.rs`, and `e2e_actions.rs` for the Rust
client's actions).

## The fleet backend's API from Node (`@3pm/api/fleet`)

**What you see.** A script that wants to read a whole fleet — not this
host's daemon, but the backend every node reports to:

```js
const { Client } = require('@3pm/api/fleet');

const api = new Client({ baseUrl: 'https://fleet.example.com', token: '3pmu_…' });
for (const b of await api.listBuckets()) {
  console.log(b.name, (await api.listNodes(b.cluster_id)).length, 'nodes');
}
for await (const [kind, payload] of api.follow(id)) …   // the live feed
```

A method per endpoint, `api.url(name, …)` for the routes that are bytes
rather than JSON, `ApiError {status, code, detail}` with the backend's
own `ERR_*`, and `api.discover()` to follow whichever backend answers.
A token authenticates; there is no browser session here.

**Where it comes from.** The backend declares its routes once
(`3pm-backend/src/api/surface/table.rs`) and `just api` generates the
client from them — twice, from one template: the ES module the backend
serves at `/api/v1/client.js`, and this CommonJS copy at
`packages/node/lib/fleet/api.js`. The template marks its browser-only
blocks (the token prompt, the `EventSource` reader) and the Node build
drops them, so the two can never drift. The generator skips this output
when there is no `3pm` checkout beside the backend.

`@3pm/api` now declares an `exports` map: `.`, `./renderer`, `./fleet`
and `./package.json` — the three entry points that worked before are
unchanged, and deep imports into `lib/` are closed.

## Custom actions and profiling (`3pm trigger`, `3pm profile`)

**What you see.** pm2's custom actions (`io.action`), for 3pm: an app
exposes functions on its control connection, `3pm trigger` runs one on
every instance of a name or on one, and the fleet backend can run the
same through the agent. The SDK's built-in `3pm:profile` action takes a
CPU or heap sampling profile of the process; `3pm profile` wraps it and
prints the hot frames.

```console
$ 3pm trigger api                      # what each source declares
┌──────────┬────────┬─────────────┬────────┬─────────┬────────────────┐
│ instance │ source │ action      │ remote │ timeout │ description    │
├──────────┼────────┼─────────────┼────────┼─────────┼────────────────┤
│ api:0    │ main   │ 3pm:profile │ yes    │ 75s     │ CPU or heap …  │
│ api:0    │ main   │ cache.flush │ yes    │ 5s      │ drop the cache │
│ api:1    │ main   │ 3pm:profile │ yes    │ 75s     │ CPU or heap …  │
│ api:1    │ main   │ cache.flush │ yes    │ 5s      │ drop the cache │
└──────────┴────────┴─────────────┴────────┴─────────┴────────────────┘
$ 3pm trigger api cache.flush '{"all": true}'
api:0 [main] cache.flush ok (12 ms)
  {
    "flushed": 4
  }
api:1 [main] cache.flush ok (9 ms)
  {
    "flushed": 3
  }
$ 3pm profile api --instance 0 --duration 10s
sampling the CPU of api for 10s…
api:0 [main] 3pm:profile ok (10024 ms)
  {
    "kind": "cpu",
    "duration_ms": 10000,
    "sampling": 1000,
    "bytes": 184322
  }
  file: /home/u/.3pm/actions/api/1726300000000-1f.cpuprofile (180.0kb)
 CPU profile: 9500 samples over 10.0 s, 7800 ms busy (79.6% of the time) → /home/u/.3pm/actions/api/1726300000000-1f.cpuprofile
┌──────┬───────────┬─────────────────────┬──────────────────────┐
│ %    │ self (ms) │ function            │ location             │
├──────┼───────────┼─────────────────────┼──────────────────────┤
│ 62.4 │ 6120.0    │ hot                 │ /srv/app/worker.js:6 │
│ 9.2  │ 900.0     │ (garbage collector) │                      │
└──────┴───────────┴─────────────────────┴──────────────────────┘
$ 3pm profile api --heap --duration 30s --src renderer   # an Electron renderer, over its debugger
$ 3pm events -f proc_action                              # the audit line of every call
12:34:56 proc_action     3|api     [main] action cache.flush (local) ok 12 ms
```

The file opens as is in Chrome DevTools (Performance → Load profile)
and in speedscope. Both kinds are *sampling* profiles: `cpu` samples the
stack, `--heap` samples allocations (which functions allocate, not what
the heap holds). There is no heap snapshot: it would freeze the app while
taken, weigh hundreds of MB and carry its data, secrets included.
Locally, `3pm trigger` needs nothing more than the daemon's socket, like
every other command: whoever can talk to the daemon can run any declared
action, `remote: false` included (that flag only closes the fleet path). `--json` prints the outcomes (and, for `profile`,
the summary) for scripts; `3pm show` lists the actions under *Custom
metrics*.

**How to get it.** The Node SDK, the Rust SDK and — since
`[Unreleased]` — the Python SDK (`io.action`, `io.call` with a file in
and a file out, `io.serve`; no stream yet: `packages/python/README.md`);
C++ ignores the messages (it declares nothing yet):

```js
const { io } = require('@3pm/api');
io.init();                                             // source `main`: `3pm:profile` declared by default (`profile: false` to opt out; other sources opt in with `profile: true`)
io.action('cache.flush', { description: 'drop the cache', timeout_ms: 5000 }, async (params) => {
  const n = await cache.flush(params.all);
  return { flushed: n };                               // JSON, ≤ 512 KiB
});
io.action('dump', async (params, ctx) => {             // a bigger result: a file
  await fs.promises.writeFile(ctx.file + '.tmp', JSON.stringify(state));
  return { bytes: await ctx.commitFile('json') };
});
io.action('reset', { remote: false }, () => reset());  // `3pm trigger` only, never the fleet backend
```

A process that declares an action of its own is **held up to serve
it**: a worker whose script is only `io.init()` and `io.action(…)` stays
online instead of exiting once the script has run (the SDK's control
socket is unref'd, so nothing else would hold it). The SDK's own
`3pm:profile` does not count, so a process that only reports metrics
still exits when its work is done; `io.init({ keep_alive: false })` opts
out, and the daemon's `shutdown` lets go. On the calling side, `io.call`
and `io.open` placed right after `io.init()` wait for the control
connection (`connect_wait_ms`, 5 s) instead of failing `node_offline`,
`await io.ready()` says when it is up, and a script awaiting a call is
held until the answer arrives.

An Electron app declares actions in its renderers too (`createRendererIo`
with an `ipcRenderer` that has `on`): each window keeps its own, dropped
when it commits a navigation (a reload too), crashes or closes — a
navigation that never commits (a cancelled `will-navigate`, a download)
keeps them, and the new page is asked to declare its own. A declaration
is acknowledged by the bridge and resent until it is, so one made before
`io.ipcBridge()` still arrives; a renderer's `shutdown()` withdraws it;
renderers cannot take `main`, `pid-*` or `worker-*`; and only the window
a call went to can answer it. The main process's `io.ipcBridge` profiles
a renderer through `webContents.debugger` — DevTools already attached is
an immediate, explicit error, and a source shared by several windows is
refused (give each window its own `source`).

Without a `source`, `io.init()` names the connection `main` in the
process 3pm spawned, `worker-<pid>-<threadId>` in a worker thread and
`pid-<pid>` in a forked child or cluster worker, so they never evict
each other. Two processes that both claim `main` (`sh -c 'node a & node
b'`) settle it: the refused one falls back to `pid-<pid>` (and drops its
default `3pm:profile`). A child started by a non-Node parent otherwise
cannot be told apart — name it.

**The rules.** A source declares its whole list at once (32 at most,
the built-in `3pm:profile` included; names `^[A-Za-z0-9_.:-]{1,64}$`, `3pm:` reserved for the SDK); the
daemon keeps it per connection and sends it again after a reconnect.
A call is bounded: the caller's `--timeout` never exceeds the
declaration's (30 s default, 10 min ceiling); one call at a time per
(source, action) unless it declares `concurrent`, 16 per source, 32
plain calls per daemon (streams counted apart, 16), else `busy`; a name
fans out to 16 instances at most; `params` ≤ 64 KiB, a result ≤ 512 KiB
inline (nested 64 levels at most) or ≤ 64 MiB as a file. Triggered from
the fleet backend the two bounds are the link's: a result travels gzip
inside the reply — 256 KiB of gzip for the whole reply, and a result
over that fails `too_large` saying to return a file instead — while the
file itself is **uploaded to the backend before the reply**, up to the
action's declared `output.max_bytes` (32 MiB when it declares none), and
the reply names the artifact it became. Files live under `~/.3pm/actions/<app>[-<n>]/`
as `<epoch_ms>-<call>.<ext>`, private to the user (`0700`); the daemon
keeps 20 per instance and 200 MB in all, oldest out — never a file whose
call is still gathering, and a file younger than 10 minutes is spared
by the count — and collects what a timed-out call left behind (after
every file, and hourly). A result file must be a regular file: a
symlink is refused. An action never takes the last slot of an app's
control queue, so `shutdown` always gets through. Every call —
succeeded, failed, refused — is one `proc_action` event on the bus with
its origin (`local` / `remote`), never its params or result; a trigger
refused as a whole (ambiguous source, nothing declares the action, more
than 16 instances) leaves one per instance it named. Per-instance
errors are typed (`failed`, `timeout`, `busy`, `gone`, `refused`,
`too_large`) and `3pm trigger` exits 1 when any instance failed, 2 when
nothing declares the action (the message lists what is declared).

Wire contract: `docs/protocol.md` (`dispatch` / `called`, the control
channel's `actions` / `action` / `action_reply`, `proc_action`).

## Zero-downtime update: `3pm update` hands the daemon off in place

```
$ 3pm update
[3PM] Updating daemon v0.9.2 → v0.10.0 (handoff: apps keep running)
[3PM] Saved 12 process(es)
Daemon updated to v0.10.0 in 180 ms — 12 process(es) kept running, 0 restarted
```

The daemon re-executes the new binary **in the same process**: the apps
keep their pids and their pipes, every log line survives (including the
bytes read but not yet written at the switch), an app that exits during
the switch keeps its real exit code, restart timers resume with the time
they had left, `3pm logs -f` stays attached, and clients that talk to the
daemon during the few milliseconds of the switch simply wait. A same-
version `update` is a live reload of `config.toml` (log rotation policy
and the like). `3pm internals` shows the image's `boot id`, how many
handoffs it went through and the last one's outcome.

**Every failure has a floor no worse than before.** The new binary is
run first as a pre-check on a draft of the handoff; a binary that cannot
take over (too old, incompatible, unreadable config, disk full) is
refused and `update` falls back to the cold path (save → kill → respawn
→ resurrect, the apps restart once) — `--no-fallback` exits 3 instead
and leaves the apps alone, `--cold` asks for the cold path outright. A
failed `execve` resumes the old image with nothing lost; a new image
that cannot adopt the apps stops them with their own signals and
restarts them from the saved list; a new image that dies outright is
caught by a small guardian process that stops the apps and restarts the
daemon on the same sockets. `--json` reports `mode` (`handoff`, `cold`,
`emergency`, `recovered`), `kept`, `restarted`, `duration_ms` and, on a
fallback, `fallback_reason`.

Not carried across (documented, bounded): an SDK `ready`/metric/event
sent during the switch (the SDK resends its registry on reconnect; a
`ready` lost is covered by `listen_timeout`), a watched file changed in
the window, a cron tick landing exactly in it. Two hundred apps cross the switch
in about 100 ms on a laptop; an app in backoff, one on its kill timeout
and one waiting to say `ready` each finish their cycle under the new
image at the moment they would have anyway, and a signal aimed at the
daemon mid-switch is held until the new image can act on it
(`tests/e2e_handoff_states.rs`).

**3pm's own processes are restarted at every update.** The apps are
what a handoff keeps running; what runs 3pm's own code is not. The
daemon is re-executed, the host agent is stopped by the old image and
started again by the new one, and every live module app — the fleet
agent, the `3pm serve` sites, a module's `[[apps]]` — is restarted once
the new daemon serves, **whether the release changed it or not**. So a
module never runs code, nor carries process state, older than the daemon
it talks to, and `just bundle && 3pm update` is the whole dev loop. The
price, at every `update` (a same-version one included, which is also the
live reload of `config.toml`): the fleet link drops for the time of an
agent restart — it comes back on its own, no new enrollment — and a
served site blips. `--skip-modules` leaves the module apps alone.

**Modules follow, after the daemon.** The apps of a module keep running
the old code through the switch (same pid), so `update` then refreshes
them, in place — same row, same name, a new `spec.module.version` stamp
— never deleted and recreated. Two populations: an installed copy
(`~/.3pm/modules/<name>/`) whose bundled version changed is replaced by
a staged copy and its rows restarted (the manifest apps on their new
spec, the sites it serves and the apps started by hand from its files
as they are; a `[[apps]]` entry the release added is started, one it
removed is deleted); an app running straight from the bundle without an
install — the agent after `3pm link`, a site served with `3pm serve`
before any install — is restarted on the new binary and re-stamped when
the `module.toml` next to its script moved on. A row already at the
bundle's version is restarted as it is — same spec, same stamp, nothing
reconciled: an app of the manifest that you deleted stays deleted, and
a module app whose every instance is **stopped** stays stopped (stopped
on purpose; an update is no reason to bring it back). The decision is
per row, never per module: an installed copy whose files are current but
whose rows are not (a refresh that failed after the copy) is re-stamped
by the next `update`. `--skip-modules` leaves them all alone. The cold
path restarts every app anyway: only the module apps behind their stamp
are restarted once more to carry it, never the current ones. A module
app whose directory vanished is left as is, with a notice.

```
Daemon updated to v0.10.0 in 180 ms — 12 process(es) kept running, 0 restarted
Module [serve] 0.9.2 → 0.10.0 reinstalled — 2 app(s) restarted
Module [agent] 0.9.2 → 0.10.0 refreshed from the bundle — 1 app(s) restarted
```

and, when the release did not touch a module (or on a same-version
`update`):

```
Module [agent] 0.10.0 — 1 app(s) restarted
```

`--json` adds `modules: [{name, from, to, source: "installed"|"bundle",
restarted, started, deleted}]` — `from == to` for a module restarted as
it is; absent under `--skip-modules`, when no module app is live, and on
a cold path with nothing to re-stamp. The module restarts are counted
there, not in `restarted`.

**Where it comes from.** OV `3pm-docs-archived/architecture` §4.7 and the plan in
`3pm-handoff-plan.md` in `~/3pm-rd-doc-2026-09-09.tgz`; daemon
side `crates/threepm-daemon/src/handoff/`
(`orchestrate`, `exec`, `guardian`, `inherit`, `adopt_main`, `emergency`,
`manifest`, `check`) and `bootstrap/daemon_main.rs`; actor side
`supervisor/{quiesce,handoff,manifest,adopt}.rs`; client side
`crates/threepm-protocol/src/client/handoff.rs`; CLI
`commands/{daemonctl,update_handoff,module_refresh}.rs`,
`render/update.rs`. Tests: `crates/threepm-cli/tests/e2e_handoff.rs`
(real daemons, a second copy of the binary standing in for the new
version, test knobs `THREEPM_HANDOFF_{HOLD_MS,CRASH_AT,EXEC_FAIL}` read
only under the test marker), the guardian as a real forked process in
`handoff/guardian_tests.rs`, the actor's extraction/adoption under paused
time in `supervisor/tests/{handoff,adopt}.rs`, the modules step against
a bundle bumped in place in `tests/e2e_modules.rs` (both populations,
`--skip-modules`, the cold path, a vanished directory) and against the
real agent in `modules/agent/tests/e2e.rs`, the transient states and the
200-app switch, the exit and the held signal inside the window in
`tests/e2e_handoff_states.rs`.

## Boot in your own session: `3pm startup --level user`

**What you see.** `3pm startup` installs a system service (root, the
daemon running as you). A desktop app that must live in your GUI
session — macOS AudioUnits, the keychain, anything that checks the Aqua
audit session — needs the daemon there too: `3pm startup --level user`
installs a LaunchAgent under `~/Library/LaunchAgents` (`systemctl
--user` on Linux) without root, and `3pm unstartup --level user` removes
it. `3pm save` records what comes back at login.

**Where it comes from.** `commands/startup.rs` (`ServiceLevel::User`
through `service-manager`), `cli.rs` (`--level`).

## Hosted users: one root daemon, per-user sockets

**What you see.** A root daemon can **host** the accounts you declare, so
a non-root user drives its own apps — start, stop, logs, metrics — without
a daemon of its own, and root still sees the whole machine in one
`3pm ls`. Two lines in root's `config.toml`, then a restart of the daemon:

```console
# /root/.3pm/config.toml
[users]
allow = ["srt"]              # optional: dir = "/var/lib/3pm/users", max_apps = 32

$ sudo 3pm update            # 3pm.log: "hosting user" user=srt home=/var/lib/3pm/users/srt
```

`srt` now has its own `rpc.sock`, `pub.sock` and `ctl.sock` in
`/var/lib/3pm/users/srt/`, and its CLI finds that home on its own — no
`THREEPM_HOME` to set, no daemon to start:

```console
srt$ 3pm start cam.py --name tabloid-cam --persist false
srt$ 3pm ls                            # its own rows, nothing else
srt$ 3pm logs tabloid-cam              # /var/lib/3pm/users/srt/logs/…, then the stream
srt$ 3pm start x --user root
[3PM][ERROR] cannot run as root: a hosted user runs its apps as itself
srt$ 3pm start kiosk                   # a name root already holds
[3PM][ERROR] `kiosk` belongs to root
srt$ 3pm save
[3PM][ERROR] forbidden: `save` is a daemon-level command (hosted user)
srt$ 3pm ls                            # with the root daemon stopped
[3PM][ERROR] this home (/var/lib/3pm/users/srt) is hosted by the root 3pm
daemon, which is not running — a hosted user never starts a daemon of its own
```

Root administers everything, and can run an app **as** a hosted account
without that account being involved at all:

```console
$ sudo 3pm start api.js --user srt     # or `user = "srt"` in an ecosystem file
$ sudo 3pm ls                          # every row, srt's included
$ sudo 3pm show api                    # the account it runs as, its owner, whether it persists
$ ps -o user,pid,cmd -p $(sudo 3pm show api --json | jq .[0].pid)   # srt
```

**Two axes, on purpose.** `user` is *who the app runs as*: the uid, gid and
supplementary groups of the child, and the home its files go to. `owner` is
*whose connection started it*: what `3pm ls` shows that caller, and what it
may stop, restart or delete. Root starting an app as `srt` produces
`user = srt`, no owner: root's app, running as `srt`, its logs in `srt`'s
home. `srt` starting an app produces both — the daemon stamps `owner` from
the connection and forces `user` to the same account. `owner` is never a
field you set: a client that sends one is overwritten, silently and on
every path (`start`, `apply`, a patched `restart`, a rename, `scale`).

**What the daemon holds to.**

| | |
|---|---|
| Identity per connection | each user's three sockets are chowned to it (0600) and peer-uid checked against *that* uid; the daemon knows who is calling before it reads the request |
| Root's commands stay root's | `save`, `resurrect`, `kill`, `internals`, `orphans` and the `3pm update` handoff answer `forbidden` to a hosted caller (V1: `save` would let one account decide what root relaunches at boot) |
| The host is everyone's | `3pm hostmetrics` (and the SDK's `hostMetrics()`) answers a hosted caller: CPU, frequencies, memory, network, sensors are the machine's. What names the other accounts is dropped from its reply — `top_processes` (their processes, root's included) and `sessions` (who is logged in) |
| Nothing of root's environment | a hosted child's environment is built from scratch: `PATH`/`LANG`/`TZ` carried over, the account's `HOME`/`USER`/`LOGNAME`/`SHELL`, the app's own `env`. Root's `3pm link` credentials and `SUDO_*` never cross |
| Nothing of root's descriptors | the daemon's sockets are never inherited by a hosted child, including one spawned by the image that just took over an in-place handoff |
| Files root writes, the account reads | the hosted home and its `logs/`, `pids/`, `actions/` are `root:<gid> 0750`; logs, rotated generations and pid files are `root:<gid> 0640`, set explicitly rather than left to the umask. An app of a hosted user may not choose its log paths (`out_file`, `error_file`, `log_file` are refused): root would open — and `delete --flush` unlink — whatever it named |
| No way back to root | the child drops groups, gid then uid, proves `setuid(0)` fails, and on Linux sets `PR_SET_NO_NEW_PRIVS` |
| Bounded | at most `[users] max_apps` rows per account (default 32), 64 live connections per hosted socket, and `watch` roots that stay inside the app's `cwd` |
| The fleet is not a privilege | a hosted app calls other nodes (`io.call({node})`, `3pm call --node`), sends and receives files, and opens streams exactly as a root app does — no setting, nothing to grant. Its blobs travel on `agent/data-<uid>.sock`, which the agent binds for its uid alone (0600) |
| Names stay global | a name another owner holds is refused on both sides; root is told how to free it (`3pm delete <name>`) |

**`persist = false`.** A new per-app field (`--persist false`, or in an
ecosystem file), for apps that must not come back by themselves: the app is
supervised, restarted and carried across `3pm update` exactly like any
other, but it is left out of `dump.json` — so out of `3pm save`, out of the
implicit save `3pm update` does, and absent after a reboot. It is what an
ephemeral per-frame backend wants: at boot, the app that owns the camera
starts it, not the resurrect list. And it is not part of the configuration
`3pm apply` compares: changing `persist` alone updates the running rows in
place, so the backend holding the camera is never cycled for it.

**Edge cases worth knowing.**

- **Name squatting.** App names are one global namespace. A hosted account
  can take `kiosk` before root does; root, who sees every row, frees it
  with `3pm delete kiosk` (the refusal says so). V1 has no `--force`:
  taking a running app away from its owner is a decision, not a flag.
- **`forbidden` exits 1**, like any other failure — only `not_found` has
  its own code (2). Scripts read `code` from `--json`.
- **Adding or removing an account takes a daemon restart** (`3pm update`):
  `[users]` is read once, at start. There is no `3pm users add` in V1.
- **A daemon that is not root ignores `[users]`** entirely, with a warning
  in its log: it could not become another user anyway. `--user` on such a
  daemon is refused at start rather than quietly running as the daemon.
- **The account needs a private primary group** (`useradd -m -U srt`).
  A shared one (`users`) is refused: every member of the group would read
  the app's logs. So is uid 0, and a second name for a uid already allowed.
- **`PR_SET_NO_NEW_PRIVS` breaks setuid binaries** — chromium's setuid
  sandbox among them. An app that needs one stays a root app dropping its
  own privileges (`setpriv`), not a `user =` app.
- **A hosted client is cut for a moment by `3pm update`.** Handoff V1
  rebinds the hosted sockets in the new image instead of carrying them
  across the exec; a hosted client waits up to 5 s for the socket to answer
  again and reconnects (`3pm logs --follow` included). Apps keep running.
- **`3pm orphans` scans root's `pids/` only**, so a hosted app left behind
  by a dead daemon is not listed there.

**Where it comes from.** `crates/threepm-daemon/src/users/` (`config` reads
`[users]`, `home` prepares and binds, `caller` is the per-connection
identity, `env` the clean environment, `file_owner` the 0640 files, `gate`
the connection cap), `supervisor/admit.rs` (stamping and every refusal),
`spawner_user.rs` (the privilege drop), `bootstrap/services.rs` (one accept
loop set per user) and `threepm-core/src/paths/hosted.rs` +
`threepm-protocol/src/client/home.rs` (the marked home a client finds and
never spawns into). Wire contract: `docs/protocol.md` → *Hosted users*;
files and `[users]`: `docs/logs.md`; the fields: `docs/ECOSYSTEM.md`.
Tests: `crates/threepm-cli/tests/e2e_users.rs` and
`e2e_users_multi.rs` (root-gated, `just e2e-users`: the lane creates the
accounts it hosts; the second suite is the boundary between three of
them), `supervisor/tests/users.rs` under paused time, and the unit suites
of each `users/` module.

## Containers: `3pm runtime` (PID 1, auto-exit)

**What you see.** One foreground process that owns the apps, streams
their output to the container's stdout/stderr, drains on SIGTERM, and
**exits when there is nothing left to supervise** — so the
orchestrator's restart policy is the outer loop:

```console
$ 3pm runtime ecosystem.toml --env production     # every `start` flag works
[3PM] 3pm runtime: daemon pid 7, home /root/.3pm
[3PM] App [api] launched
0|api | listening on :3000
[3PM] App [api] exit code=1 signal=SIGSEGV
[3PM] App [api] errored
[3PM] nothing left to supervise: stopping the daemon (exit 2)
```

| Exit | When |
|---|---|
| `0` | SIGTERM/SIGINT → every app drained through its policy |
| `2` | nothing left to supervise: zero apps selected, or every app `stopped` / `errored` (a crash loop counts — the container restart policy takes over). `--no-auto-exit` disables |
| `1` | the daemon died without announcing, or a start failed |
| `130` | a second signal forced the daemon down |

`--raw` gives bare lines (pm2-runtime's default; 3pm prefixes `id|name |`
because containers usually run several apps), `--format` logfmt,
`--timestamp [FMT]`, `--json` the wire events. Log *files* are off
(`NULL`) unless `--log-files` or an explicit `-o`/`-e`/`-l`. `--delay
SECS` waits before starting. Unlike `oxmgr runtime`, the apps live in a
real daemon: `docker exec c 3pm ls|logs|events|restart api` all work.

**How to get it.**

```dockerfile
FROM node:22-slim
COPY --from=build /src/target/release/3pm /usr/local/bin/3pm   # or a release tarball
WORKDIR /app
COPY . .
CMD ["3pm", "runtime", "ecosystem.toml", "--env", "production"]
```

Full example: `examples/Dockerfile`; cheatsheet: `docs/RUNTIME.md`.

`3pm runtime` refuses to start if a daemon already owns `THREEPM_HOME`
(it must own the daemon to own the exit code).

**Where it comes from.** `commands/runtime.rs` (one `select!` over the
bus, signals and a `waitpid(-1)` reaper; `nothing_to_supervise()` is the
pure auto-exit predicate), `threepm-protocol::spawn_foreground`
(`SpawnMode::Foreground`: same session, inherited stdio, caller-reaped)
and `3pm __daemon --foreground` in the daemon.

## Resource limits: CPU ceiling, cgroup enforcement, GPU denial

**What you see.** `max_memory_restart` has two siblings. `max_cpu_percent`
recycles the app the moment one CPU sample crosses the limit — `3pm show`
counts a restart and the daemon log says `max_cpu_percent exceeded`. Both
are soft guards: the app is *observed* and gracefully restarted, and both
judge its **whole process tree** — the RSS sum and the CPU sum of the
spawned pid and every descendant, what `3pm ls` shows — unless
`limit_scope = "leader"` narrows them to the spawned pid (children that
are meant to grow: a build, a browser). On
Linux, `cgroup_enforce = true` additionally makes them **hard**: each
incarnation runs in its own cgroup v2 leaf under `/sys/fs/cgroup/3pm/`
with `cpu.max` (throttled by the kernel) and `memory.max` (OOM-killed by
the kernel) set from the same two fields. `deny_gpu = true` hides GPUs
from the app through the CUDA/ROCm visibility variables.

A guard restart is paced and counted like a crash: after the graceful
stop the row emits `restart` and sits in `waiting restart` for
`restart_delay` / the exponential backoff, and a trip within
`min_uptime` + 10 s of the incarnation going `online` (or 3 sampler
periods, when `THREEPM_METRICS_INTERVAL_MS` makes that longer; under
`wait_ready`, `online` is the ready signal or `listen_timeout`) counts
toward `max_restarts`. An app
over its limit from the start therefore ends `errored` (16 trips with
the defaults) instead of cycling forever, while one that runs a while
between two trips never spends its budget. A user `restart` / `stop` /
`delete` during the guard's stop starts a new series. `max_restarts = 0`
ends the app `errored` on its first trip, as on its first crash.
`restart_policy = "never"` and `stop_exit_codes` do not prevent a guard
restart — they judge the exits an app makes on its own, and a guard
restart is the daemon's recycle, as in pm2 — so leave the guard off on
an app that must never be restarted.

**How to get it.**

```console
$ 3pm start train.py --max-cpu-percent 150 --max-memory-restart 4G --cgroup-enforce --deny-gpu
```

```toml
[[apps]]
name = "train"
script = "./train.py"
max_cpu_percent = 150       # percent of one core; >100 on multi-core hosts
max_memory_restart = "4G"
limit_scope = "tree"        # "tree" (default): the sums over the process tree; "leader": the spawned pid alone
cgroup_enforce = true       # Linux only: hard cpu.max / memory.max
deny_gpu = true             # CUDA_VISIBLE_DEVICES="" NVIDIA_VISIBLE_DEVICES=none …
```

`max_cpu_percent` must be a positive number; a spec that says otherwise
is rejected at the CLI edge and again at the daemon boundary.
`cgroup_enforce` on macOS, or on a Linux host without cgroup v2 at
`/sys/fs/cgroup`, fails the spawn *before* the app starts — the row goes
`errored` and `3pm show` names the reason — so an enforce spec never runs
unenforced. A permission error tells you to run the daemon as root or
delegate `/sys/fs/cgroup/3pm` to its user. `deny_gpu` is best effort: an
app that unsets those variables can still open `/dev/nvidia*`. Full
reference and cheatsheet: `docs/resources.md`.

**Where it comes from.** The policy's `CpuSampled{gen, tree_cpu,
leader_cpu, sample_period_ms}` and `MemorySampled{gen, tree_rss,
leader_rss, sample_period_ms}` inputs carry both figures and the
sampler's period (the supervisor's `metrics_interval`); in
`threepm-core/src/policy/guards.rs`, `scoped` picks the one
`limit_scope` names inside `step`, `on_resource_sampled` is shared by
the two guards (gen-guarded, online rows only, never during daemon
shutdown), and `after_guard_stop` respawns through the crash path
(`respawn_or_errored`), the trip's uptime (since
`ProcState::online_at`, carried by the handoff manifest as
`online_at_age_ms`) judged against `min_uptime` +
`settle_margin_ms(period)`: `GUARD_SETTLE_MS` (10 s) or
`GUARD_SETTLE_PERIODS` (3) periods, whichever is longer.
`threepm-daemon::cgroup` owns
the cgroup file protocol against an injected root (unit-tested on every
platform, `cgroup_tests.rs`): `CgroupFs::prepare` makes the
incarnation's leaf (`<name>-<id>-<instance>-<lineage>-g<gen>`) and
writes its limits before the spawn, the child writes itself into the
leaf's `cgroup.procs` between fork and `exec` (`spawner_cgroup.rs`) — a
failed join fails the spawn and `discard` removes the leaf, as `prepare`
does itself when a limit cannot be written —, and
`CgroupFs::cleanup`, off the actor on `ChildExited` / delete, kills what
is left in the leaf (`cgroup.kill`, else SIGKILL member by member with
the leaf frozen, so no pid is reused between the read and the kill),
removes the cgroups the app made below it and the leaf, then the
instance's older empty leaves. The log files are opened before all of
it: one that cannot be opened fails the spawn before any process or
leaf exists.
`deny_gpu` is `spawner::GPU_DENY_ENV`, applied after the user's `env` so
it wins.

## Limits and scheduling: `ulimit_*`, `nice`, `ioprio`, `oom_score_adj`, `cpu_affinity`

**What you see.** An app that runs its threads in `SCHED_FIFO` (real-time
audio: a PipeWire client, a plugin host, a DSP) needs a real-time
priority limit and room to lock its buffers in RAM. A 3pm daemon started
by systemd has neither (`Max realtime priority 0`, `Max locked memory`
8 MB: a system unit never goes through PAM's `limits.conf`), and every
app inherits them. `ulimit_rtprio` and `ulimit_memlock` set the app's
`RLIMIT_RTPRIO` / `RLIMIT_MEMLOCK` (soft = hard) right before `exec`;
`/proc/<pid>/limits` shows them, the app's whole tree inherits them, and
`3pm show` lists them as `rtprio limit` / `memlock limit`. 3pm does not
change any scheduling itself: the app asks for `SCHED_FIFO` for the
threads that need it.

**How to get it.**

```console
$ 3pm start ./onyx --user srt --ulimit-rtprio 95 --ulimit-memlock unlimited
```

```toml
[[apps]]
name = "onyx"
script = "./onyx"
user = "srt"
ulimit_rtprio = 95          # Linux only, 0–99
ulimit_memlock = "4G"       # a size, or "unlimited"
```

From the Node SDK: `start({ script, ulimit_rtprio: 95, ulimit_memlock: '4G' })`.

Who may ask for what is judged at `start`, never by a restart loop:

- a **root** daemon grants any value — it sets the limit before the drop
  to `user`, so past its own — provided it holds `CAP_SYS_RESOURCE`
  (root in a stock container does not: it is then judged like the next);
- a daemon that is **not root** grants up to its own hard limit, and
  refuses more (`ulimit_rtprio 95: above this daemon's hard limit (0);
  only a root daemon (with CAP_SYS_RESOURCE) can raise it`);
- an app a **hosted user** owns (started over its own sockets, like the
  frame backends tabloid starts as `srt`) gets at most what the root
  daemon's `config.toml` grants; when nothing is granted it may still
  keep or lower what the daemon runs with (`ulimit_memlock = "0"`), never
  raise it:

```toml
[users]
allow = ["srt"]
max_ulimit_rtprio = 95
max_ulimit_memlock = "4G"     # or "unlimited"
```

`ulimit_rtprio` on macOS is refused (no `RLIMIT_RTPRIO`); `ulimit_memlock`
works there too. `no_new_privs`, set on a hosted user's apps, does not get
in the way: an unprivileged process may take `SCHED_FIFO` up to its
`RLIMIT_RTPRIO` whatever it says (PipeWire runs that way). A new value
takes effect at the next spawn (`apply` restarts a changed app); a
handoff (`3pm update`) keeps the running apps' limits. The rules above
judge `start`, `apply`, a patched `restart` and a rename; a resurrect
(`dump.json`, boot), a crash respawn and a plain `3pm restart` re-run
the admitted spec as it is, like `max_apps`: a lowered `[users]` grant
reaches an app already admitted only once it is `3pm delete`d and
started again. A
daemon older than the field ignores it: update 3pm before relying on it.
With `cgroup_enforce` on a kernel built with `CONFIG_RT_GROUP_SCHED`
(rare: Ubuntu and the Surface kernels leave it off), the app's own
cgroup has no real-time budget and `SCHED_FIFO` fails whatever the
limit.

**The rest of a desktop session's scheduling.** A desktop session also
hands its apps a nice limit (PAM, rtkit), and the frame's processes use
it: Chromium lowers its audio threads to nice -8, the plugin host and the
MIDI scheduler fall back to -11 when `SCHED_FIFO` is refused. Under a
systemd-started daemon `Max nice priority` is 0 and those calls fail
silently. Six more fields, same model — set by the child on itself
before `exec`, judged at `start` — fill that in and add the guards a
kiosk wants:

| Field | Flag | Sets | Range |
|---|---|---|---|
| `ulimit_nice` | `--ulimit-nice` | `RLIMIT_NICE`: a nice down to `20 - v` | 0–40, Linux |
| `ulimit_rttime` | `--ulimit-rttime` | `RLIMIT_RTTIME`, µs a real-time thread may run without blocking before the kernel kills it (SIGKILL, no SIGXCPU warning: soft = hard) | > 0, Linux |
| `nice` | `--nice` | the app's nice value (`setpriority`) | -20–19 |
| `ioprio` | `--ioprio` | the I/O class: `idle`, `best-effort[:0-7]`, `realtime[:0-7]` | Linux, BFQ / mq-deadline |
| `oom_score_adj` | `--oom-score-adj` | the OOM killer's bias, -1000 = never | -1000–1000, Linux |
| `cpu_affinity` | `--cpu-affinity` | the CPUs it may run on (`sched_setaffinity`) | `"0-3,6"`, Linux |

```toml
[[apps]]
name = "frame"
script = "./frame"
user = "srt"
ulimit_nice = 31            # its threads may go to -11
nice = -5
oom_score_adj = -900        # the last app the OOM killer picks
ulimit_rttime = 200000      # a FIFO thread hogging 200 ms is killed

[[apps]]
name = "indexer"
script = "./indexer"
nice = 10
ioprio = "idle"             # disk only when nobody else wants it
cpu_affinity = "2-3"
```

`3pm show` lists them (`nice limit`, `rttime limit`, `nice`, `io
priority`, `oom score adj`, `cpu affinity`). The `pre_exec` order is the
limits first, then nice, I/O class, OOM score and CPUs, so a `nice` below
the daemon's can rest on the app's own `ulimit_nice`. A refused step
names its field: `nice -11: cannot set it: Permission denied`.

Who may ask for what, judged at `start` like the limits:

- **anything that only gives up** — a higher nice or OOM score, the idle
  or best-effort I/O class, a lower limit, a subset of CPUs — needs no
  privilege and is never refused;
- a daemon **without the capability** refuses: a `nice` below its own
  without `CAP_SYS_NICE`, unless `20 - nice` fits the app's `ulimit_nice`
  (or the daemon's soft `RLIMIT_NICE`); an `oom_score_adj` below its own
  or a limit above its hard one without `CAP_SYS_RESOURCE`; the realtime
  I/O class without `CAP_SYS_NICE` / `CAP_SYS_ADMIN`;
- a **CPU** the daemon may not run on is refused (`cpu_affinity 2-5: CPU
  4-5 not available to this daemon (it runs on 0-3)`), not left to an
  `EINVAL` restart loop;
- an app a **hosted user** owns gets a nice below the daemon's only
  down to `20 - max_ulimit_nice`, an `oom_score_adj` below the daemon's only down
  to `min_oom_score_adj`, never the realtime I/O class (it starves every
  other app's disk), and a `ulimit_rttime` no higher than the daemon's:

```toml
[users]
allow = ["srt"]
max_ulimit_nice = 31          # nice down to -11
min_oom_score_adj = -900
```

All but `nice` are refused off Linux. `ioprio` is honoured by the BFQ
and mq-deadline disk schedulers; `none`, the NVMe default, ignores it.
`nice` ranks an app against the processes of its own CPU group only: a
`tty` app (its own session, so its own autogroup under
`CONFIG_SCHED_AUTOGROUP`, the Ubuntu default) or a `cgroup_enforce` app
(weighed by `cpu.weight`) is not ranked against the others by it. A
value this 3pm cannot read in a `dump.json` or from a newer peer
(`ioprio = "deadline:9"`) fails that app's spec only, not the dump.

**Where it comes from.** `StartSpec.ulimit_*`, `nice`, `ioprio`,
`oom_score_adj`, `cpu_affinity` (`threepm-core/src/spec.rs`; the value
types in `spec/ulimit.rs` — a number of bytes or `"unlimited"` on the
wire, never `u64::MAX` — and `spec/sched.rs` — `IoPrio` and `CpuList`
cross the wire as their text); the `pre_exec` step `spawner_rlimit.rs`
(after the cgroup join and the terminal login, before `spawner_user`'s
drop, each step's errno offset by `RLIMIT_BASE + step × 0x1000`);
`check_ulimits` in `supervisor/admit_ulimits.rs`; the caps in `[users]`
(`config_file.rs`, `users::UlimitCaps`).

## Cron restarts

```toml
[[apps]]
name = "nightly"
script = "./job.sh"
cron_restart = "0 2 * * *"     # pm2's 5 fields; "30 0 2 * * *" for seconds
```

`--cron-restart "0 2 * * *"` (alias `--cron`) on `start`. Five fields
(minute first, pm2) or six (seconds first, OxMgr); lists, ranges, steps,
month/day names, `7` = Sunday, and when both day-of-month and
day-of-week are restricted either matches (Vixie cron). The expression
is validated at `start`/`validate`, stored in its 6-field spelling, and
`show` prints it with the **next fire** in local time (`ls --json`:
`next_cron_epoch_ms`). Each tick restarts the app's **online** rows,
exactly like a file change: a `stopped` app is never revived by its
schedule, a row mid-restart skips the tick, and the scheduler is
dropped while every row is stopped. Local time zone, DST-aware
(re-evaluated at every fire).

## Target patterns: `3pm logs 'victoria*'`

**What you see.** Anywhere a command takes a process, a target holding
`*` or `?` is a **pattern** rather than a name — no flag, no mode:

```
$ 3pm logs 'victoria*'          # the three victoria apps at once
$ 3pm restart 'worker-?'        # worker-1 … worker-9
$ 3pm delete 'bench*'
$ 3pm show 'api*'
$ 3pm events --process 'web-*'
```

`*` matches any run of characters (possibly empty), `?` exactly one —
the same vocabulary as `ignore_watch` and `3pm events --filter`. The
match is **whole-string** and case-sensitive: `victoria*` selects
`victoria-logs`, and the substring form is spelled `*victoria*`. A
pattern is matched against the same two fields a literal name is — the
app name **and** the namespace — so a namespace pattern sweeps
everything filed under it. Ids are never matched: a numeric target is an
id, and `1*` is a name pattern.

Quote it. An unquoted `victoria*` is the shell's to expand, and it
silently becomes a filename (or nothing) before 3pm ever sees it — the
"no process matches" notice says so.

**Where it applies.** `logs`, `stop`, `restart` (including
`--update-env`), `delete`, `reset`, `show`, `sendSignal`, `flush`,
`rotate`, and `events --process`. `scale` refuses one: an instance count
belongs to a single app.

A pattern that selects nothing is a **notice** when other targets
matched (`3pm logs api 'ghost*'` still follows `api`) and the run's
error when nothing is left to act on — for a mutation the error comes
*before* the first request, so nothing is half-done.

**How it works.** Patterns are resolved in the CLI, against the process
list the daemon just returned, and only the names they selected go on
the wire. The protocol carries no glob: no `Target` variant to grow, and
no daemon that could resolve a pattern differently from the one that
printed the list you were looking at. `events --process` is the
exception — it is a server-side filter, so the glob travels and the
daemon matches it.

Module rows (`3pm install …`) are shielded from a pattern on a mutation
exactly as they are from `all`: `3pm stop 'a*'` never stops the agent,
only its spelled-out name does. Reads hide nothing — a module's logs are
logs like any other.

## `sendSignal`

`3pm sendSignal SIGUSR2 api` (aliases `send-signal`, `signal`; bare
names like `usr2` are accepted) delivers a signal to the running process
of each matched row — the process itself, as pm2 does, or its whole
process group with `--group`. It is not a lifecycle operation: nothing
about restart accounting changes, and if the signal kills the process
that is an ordinary exit for the restart policy. A stopped app is an
error (`exit 2`), an unknown signal name gets a did-you-mean.

## pm2's process dumps: `3pm jlist`, `3pm prettylist`

For scripts written against pm2, `3pm jlist` prints the process list as
`pm2 jlist` does (one line of JSON) and `3pm prettylist` as `pm2
prettylist` does (Node's `util.inspect`): `{pid, name, pm2_env, pm_id,
monit}` per process, with pm2's keys in `pm2_env` (`status`,
`restart_time`, `NODE_APP_INSTANCE`, `exec_mode`, `watch`,
`pm_out_log_path`, …) and the app's variables at the top of `pm2_env`.
A field appears as often as in pm2's output, so `grep -c` counts match.
Module apps are left out. `exec_mode` is derived: `cluster_mode` for an
app of several instances, `fork_mode` otherwise. Both are hidden from
`3pm --help`; `3pm ls --json` stays the native format.

## `3pm cd`: go to an app's working directory

A command cannot move the shell that ran it, so `3pm cd` answers in two
ways, chosen by where its stdout goes:

```
$ 3pm cd api                  # a terminal: a shell in the app's cwd
[3PM] /bin/zsh in /srv/api (exit to return)
$ cd "$(3pm cd api)"          # piped: the path, for this shell
$ 3pm cd api --json
{"name":"api","cwd":"/srv/api"}
```

The shell is `$SHELL` (`/bin/sh` when unset), exec'd in place: no 3pm
process waits behind it. A folder that no longer exists is an error
rather than a shell somewhere else. The target is one name or id (the
instances of an app share its cwd); `all` and patterns are refused. For
a real `cd` without the subshell, a shell function does it:

```sh
3cd() { cd "$(3pm cd "$1" | cat)"; }
```

## Host metrics agent (`3pm hostmetrics` served from a separate process)

**What you see.** `3pm hostmetrics` answers from a cache in ~10ms and
its overview starts with a `sampled` row (`2026-08-28T15:02:48Z (3s
ago)`; `collected_at_epoch_ms` in `--json`). `3pm ls` — and every
command that reprints the table — ends with pm2's one-line host summary,
`host metrics | cpu: 12% 45º | load: 1.23 | ram: 60% | gpu: 22% 1199/11538mb
| en0: ⇓ 1.2mb/s ⇑ 300kb/s | disk: ⇓ 512kb/s ⇑ 2.0mb/s /data 85%`
(thresholds colored like pm2; virtual/loopback NICs and idle segments
hidden; the throughput segments appear from the agent's second fast
sample on — about 2 s — since they are rates over its 1 s window).
`3pm internals` shows a
`host agent` row — the agent's pid and how many times the daemon had to
respawn it — and an `internal processes` table listing every process 3pm
runs for itself (the daemon, the agent) with pid, status, uptime, cpu,
RSS and respawns (`internal_processes` in `--json`; the agent's gauges
are sampled on the daemon's metrics tick and never carried over from a
previous incarnation). If the agent is down or wedged, the command fails fast with
`err{unavailable}` and a message naming `hostagent.log` — the daemon never
collects inline.

**How to get it.** Nothing to enable: the daemon spawns
`3pm __host-agent` at boot, restarts it with exponential backoff (1s →
30s) when it dies, and terminates it at `3pm kill`. The agent samples in
cost tiers, each on its own ticker: cheap gauges (cpu, memory, load,
PSI, NIC/disk counters) every second, walks and forks (process table,
socket table, sensors, `nvidia-smi`/`ioreg`, `statvfs`) every 10s, the
NTP probe and identity every 5 minutes — a hung GPU tool or a stalled
NFS mount never delays the cheap ones. `THREEPM_HOSTMETRICS_INTERVAL_MS`
(min 200), `THREEPM_HOSTMETRICS_MEDIUM_MS` and
`THREEPM_HOSTMETRICS_SLOW_MS` override the periods (each floored by the
previous). Its log is `~/.3pm/hostagent.log`. An agent that is alive
but no longer answers (stopped, deadlocked) is pinged every 5s
(`THREEPM_HOSTAGENT_HEALTH_MS`) and replaced after 6 misses in a row
(~30s); one answering with another protocol is left alone and named
once in `3pm.log` (run `3pm update`). On Linux, once a new binary was
installed over the old one, a respawn re-runs the daemon's own image
(`/proc/self/exe`), so it still works and the two stay the same
version until `3pm update`.

```console
$ 3pm ls | tail -1
host metrics | cpu: 38.8% 62.8º | load: 7.42 | ram: 63.5% | gpu: 22% 1199/11538mb | en0: ⇓ 1.2mb/s ⇑ 300kb/s | disk: ⇓ 512kb/s ⇑ 2.0mb/s
$ 3pm hostmetrics | head -3
 box — Linux 6.8 (x86_64)
┌─────────┬───────────────────────────────────┐
│ sampled ┆ 2026-08-28T15:02:48Z (3s ago)     │
$ 3pm internals | grep 'host agent'
│ host agent  ┆ pid 50632, 0 restarts │
$ 3pm internals | tail -6
 internal processes
┌────────────┬──────────────────┬───────┬────────┬────────┬──────┬────────┬───┐
│ role       ┆ command          ┆ pid   ┆ status ┆ uptime ┆ cpu  ┆ mem    ┆ ↺ │
│ daemon     ┆ 3pm __daemon     ┆ 50631 ┆ online ┆ 2D     ┆ 0.4% ┆ 26.7mb ┆ 0 │
│ host-agent ┆ 3pm __host-agent ┆ 50632 ┆ online ┆ 2D     ┆ 0.2% ┆ 35.9mb ┆ 0 │
└────────────┴──────────────────┴───────┴────────┴────────┴──────┴────────┴───┘
```

**Where it comes from.** Host-wide collection (`/proc` parsing,
`sysinfo`, an NTP probe, `nvidia-smi`/`ioreg` subprocesses) is the code
most exposed to platform edge cases and third-party leaks; running it in
its own process means a panic, a leak or a hang there can never take the
supervisor — and your apps — down. Rates (`rx_bytes_per_s`, `tx_bytes_per_s`,
`errors_per_s` per NIC; `read_bytes_per_s`, `write_bytes_per_s` per
mount; `rate_window_ms`) are derived agent-side from two consecutive
fast samples, so `--json` and a future fleet poller get them too; the
fast collector keeps its previous CPU ticks so only its very first
sample pays the 200ms warmup. `sampled_at_epoch_ms` carries each tier's
stamp (the overview adds `sampled (medium)` / `sampled (slow)` rows
only when they lag). The `ls`
line is best effort under a 500ms budget: no agent, no sample yet or a
wedged agent means no line, never an error. The agent speaks the daemon's own
NDJSON RPC on `host.sock` (`ping` + `host_metrics`), the daemon's
`Request::HostMetrics` handler is a pure proxy with a 2s connect window
(rides out a respawn) and a 3s request bound. Lifecycle mirrors the
daemon's: `flock` singleton (`hostagent.lock`), readiness byte, pid file,
self-termination when its parent or the home directory disappears.
Details: OV `3pm-docs-archived/architecture` §8.2.

## Batteries in `3pm hostmetrics` (`batteries[]`, `on_ac`)

**What you see.** On a laptop or a tablet, `3pm hostmetrics` gains a
`batteries` section — one row per pack, titled with whether the host is
on mains — and `3pm ls` gains a `bat:` segment on its host line, red
under 10% and yellow under 25% (a charge is the one gauge on that line
where *low* is the bad end), with a `⚡` while charging. Nothing appears
on a machine without a battery.

```console
$ 3pm hostmetrics | tail -6
 batteries — on mains
┌───────────────────────┬──────────┬────────┬────────┬─────────────┬───────────────┬────────┬──────┐
│ battery               ┆ state    ┆ charge ┆ health ┆ energy      ┆ power         ┆ cycles ┆ temp │
│ BAT1 (Li-ion M1009169)┆ Charging ┆ 84%    ┆ 99%    ┆ 37.3/44.5Wh ┆ 15.9W @ 8.71V ┆ 65     ┆ N/A  │
└───────────────────────┴──────────┴────────┴────────┴─────────────┴───────────────┴────────┴──────┘
$ 3pm ls | tail -1
host metrics | cpu: 8.1% 44º | load: 0.42 | ram: 51% | bat: 84% ⚡ | wlp1s0: ⇓ 12kb/s ⇑ 4kb/s
$ 3pm hostmetrics --json | jq '.batteries[0].power_w, .on_ac'
15.918
true
```

**How to get it.** Nothing to enable, and nothing to run as root: the
host agent reads `/sys/class/power_supply` on its medium tier, next to
sensors and the GPU tools. Linux only — macOS would need IOKit and the
section stays absent rather than faked.

**Where it comes from.** The attributes are world-readable files, so the
collector is a directory walk. What it has to get right is that hardware
disagrees about what it exposes: the kind of a supply comes from its
`type`, never its name (`BAT0`, `ACAD`, `axp20x-battery`,
`macsmc-battery`); a `scope=Device` supply is a mouse or a headset and is
skipped, while an **absent** `scope` means System (inverting that drops
the real pack on most laptops); a gauge that reports amp-hours has no
`energy_now` at all and is converted through its voltage; the draw is
carried unsigned because gauges disagree on the sign, with `status`
giving the direction; and `temp` is in tenths of a degree and usually
missing.

Above all, **an absent reading is absent, never zero**. A full pack on
the dock draws a real `0 W` and a pack without a sensor has no
temperature: collapsing the two is exactly how a charged battery comes
out reading empty. Every field is an `Option` on the wire and renders
`N/A`, and health (`full_wh / design_wh`) is left unclamped — a fresh
pack can legitimately hold more than its design capacity. Field list and
platform blanks: `docs/metrics.md`, *Power*.

## 3pm's own metrics in `3pm hostmetrics` (`3pmDaemon`)

**What you see.** `3pm hostmetrics` ends with two tables about 3pm
itself: `3pm processes` — the daemon and the host agent with pid,
uptime, cpu, RSS, open descriptors and threads — and `3pm runtime` — the
daemon's counters: version, uptime, `boot_id`, handoffs; apps by status;
restarts (unstable ones in yellow); armed timers and bus subscribers;
log sinks, rotations and bytes written; `RLIMIT_NOFILE`; agent respawns;
and when the part was sampled. Errored apps turn the `apps` row red.
`--json` carries it as `3pmDaemon: {at_epoch_ms, processes[], runtime?}`,
so anything that already reads the host snapshot — the fleet link
included — gets 3pm's own health for free.

```console
$ 3pm hostmetrics | tail -22
 3pm processes
┌────────────┬───────┬────────┬──────┬────────┬─────┬─────────┐
│ role       ┆ pid   ┆ uptime ┆ cpu  ┆ mem    ┆ fds ┆ threads │
│ daemon     ┆ 50631 ┆ 2D     ┆ 0.4% ┆ 26.7mb ┆ 41  ┆ 9       │
│ host-agent ┆ 50632 ┆ 2D     ┆ 0.2% ┆ 35.9mb ┆ 18  ┆ 7       │
└────────────┴───────┴────────┴──────┴────────┴─────┴─────────┘

 3pm runtime
┌────────────┬─────────────────────────────────────────────────────┐
│ daemon     ┆ v0.9.3, up 2D, boot 0x1f3a, 1 handoff               │
│ apps       ┆ 12 (11 online, 1 errored)                           │
│ restarts   ┆ 37 (2 unstable)                                     │
│ timers/bus ┆ 4 armed, 1 subscriber                               │
│ logs       ┆ 12 sinks, 3 rotations, 1.1gb written; 3pm.log 4.0kb │
│ nofile     ┆ 1024 / 1048576                                      │
│ host agent ┆ 0 restarts                                          │
│ sampled    ┆ 2026-08-28T15:02:40Z (11s ago)                      │
└────────────┴─────────────────────────────────────────────────────┘
$ 3pm hostmetrics --json | jq '."3pmDaemon".processes[0].mem_bytes'
26700000
```

**How to get it.** Nothing to enable. The section rides the medium
period (`THREEPM_HOSTMETRICS_MEDIUM_MS`, 10s); it is absent until the
tier has landed once and from a daemon predating it.

**Where it comes from.** The host agent measures 3pm the way it measures
the box — from the outside. Two sources, kept apart so either can fail
alone: the OS for the process rows (the daemon is the agent's parent
pid, the agent is itself; descriptors and threads of a foreign pid are
Linux-only, on macOS the daemon row's `open_fds` comes from the daemon's
own count and `threads` is absent — never faked), and the daemon's
`Request::Internals` over `rpc.sock` for the counters nothing else can
see, reduced to a health subset (no paths, no configuration — it leaves
the box). One connection per tick, 2s bound: a daemon that is down,
wedged, mid-`3pm update` or version-skewed blanks `runtime` for that
tick (`counters unavailable` in the table) while the rows keep coming;
the previous part is served meanwhile. The part never delays the first
snapshot, and `3pm internals` keeps its own 1s self-sample — it does not
depend on the agent. Details: `docs/hostagent.md` → *3pm's own metrics*,
OV `3pm-docs-archived/architecture` §8.2.

## Extended Monitoring (`3pm hostmetrics --extended`, PSS in `3pm show`)

**What you see.** `3pm hostmetrics --extended` swaps the overview for the
kernel's own counters: one table per family (`cpu`, `meminfo`, `vmstat`,
`netstat`, `cpuidle`, …, ~1 200 values on a 32-core Linux box), one line
per row and field, the family's kind in its title, and last the sources
that could not be read and why (`permission`, `error`, eBPF's
`not_root`…). A family only partly readable says `(partial: <reason>)`
in its title. `3pm show` puts the PSS sum beside the RSS sum, and its
` Processes` table gains `pss` (Linux) or `footprint` (macOS), `threads`
and `fds` — `-` for a member that could not be read. The other per-process
counters (faults, I/O, context switches) are in `--json`.

```console
$ 3pm hostmetrics --extended | head -12
 labomatic — Ubuntu Linux (Ubuntu 20.04) (5.4.0-153-generic, x86_64)
 extended monitoring — 23 families, 1228 values

 cpu — counters since boot, 32 rows
┌─────┬────────────┬────────────┐
│ row │ field      │      value │
├─────┼────────────┼────────────┤
│ 0   │ guest      │          0 │
│ 0   │ guest_nice │          0 │
│ 0   │ idle       │ 4031282977 │
│ 0   │ iowait     │     143096 │
$ 3pm show tree | grep -E ' mem |pss|sleep'
│ mem               │ 4.4mb RSS · 949.0kb PSS (3 processes, sums) │
│ pid     │ ppid    │ name  │ uptime │ cpu  │ mem     │ pss     │ threads │ fds │ ports │
│ 1316237 │ 1316236 │ sleep │ 4s     │ 0.0% │ 580.0kb │ 149.0kb │ 1       │ 5   │       │
```

**How to get it.** On by default (`[hostmetrics] extended` in
`config.toml`, `THREEPM_HOSTMETRICS_EXTENDED` above it). With the mode
off, `--extended` says so in one line and `show` is exactly what it was.
On macOS the host map is present and empty (`--extended` says the
platform has no extended source); per process, macOS has `footprint`
instead of PSS. What it costs on the link, measured on edgecr4ft.net
(2026-09-29, two nodes side by side, same 4 apps, 14 min): 777 B/s up
with the mode on against 529 B/s off, **+248 B/s** — only the fields
that moved leave, compressed; `extended = false` is per node.

**Short-lived processes (eBPF).** A sampler never sees a process that
lives less than its period — on a busy Linux server, most of them. With
the agent running as root, two kernel tracepoints count every exec and
exit exactly: `shortlived` (execs, exits, exits under 1 s, total
lifetimes, exits whose exec was not seen since the agent started — a
fork without exec such as a Postgres backend or a prefork worker, or a
process older than the agent —, and what a full kernel map refused,
`by_comm_full` / `starts_full`), `shortlived_by_comm` (exits and lifetime
per process name — the kernel's 16-byte name, never a path — for the 10
busiest names of each 10 s window, the rest under `other`; only processes
whose exec was seen, so fork-only workers show in the totals alone) and
`shortlived_by_comm_max` (the longest life per name in the window). No
clang, BTF or headers on the node: one prebuilt object ships inside
`3pm`. Not root (the agent's `CapEff` holds neither `CAP_SYS_ADMIN` nor
`CAP_BPF` + `CAP_PERFMON`, checked before anything is loaded; those are
necessary, not sufficient — tracefs permissions, Ubuntu's
`perf_event_paranoid = 3` and, before Linux 5.11, `RLIMIT_MEMLOCK` can
still refuse at the try, with the same answer): `unavailable` says
`not_root`, and nothing is retried before the agent restarts; `load_failed` (with the reason in the daemon
log) or `bpf_disabled` otherwise. Cost: not measurable over 5 000
fork+exec, one 50 ms load.

```console
$ sudo 3pm hostmetrics --extended | grep -A11 '^ shortlived — '   # a root agent
 shortlived — counters since the agent started
┌──────────────────────┬──────────────┐
│ field                │        value │
├──────────────────────┼──────────────┤
│ by_comm_full         │            0 │
│ execs                │         2402 │
│ exits                │         3446 │
│ exits_under_1s       │         2200 │
│ exits_without_exec   │         1153 │
│ lifetime_ns          │ 315147170152 │
│ lifetime_under_1s_ns │  36298126357 │
│ starts_full          │            0 │
$ 3pm hostmetrics --extended | tail -3                              # the same host, agent not root
├────────────┼──────────┤
│ shortlived │ not_root │
└────────────┴──────────┘
```

**Where it comes from.** Families print in the collection table's order
(`threepm_daemon::host_extended_family_order`), each `<family>_gauge`
right after its family, a family the table does not name last; rows sort
naturally (`2` before `10`). Values are shown as read: bytes for
`meminfo` (its `HugePages_*` lines are page counts), whole numbers
plainly, fractions to three decimals, sysfs units untouched (µs, kHz,
m°C). Nothing is summarized or hidden — `--json` carries the same map.
Details: `docs/hostagent.md` → *Extended Monitoring*, `docs/metrics.md`.

## Top processes (`3pm hostmetrics`)

**What you see.** Three sections at the end of the host part of `3pm
hostmetrics`: `top processes — cpu`, `— memory`, `— network`, ten rows
each (`pid`, `name`, `user` when known, then the section's two figures), for every
process of the host, whoever owns it. `--json` carries the one table
they are cut from, `top_processes`.

**How to read it.**
- `cpu` is CPU time over the last window: 100 % is one core busy.
- Memory is headed `footprint` on macOS: what Activity Monitor calls
  Memory, compressed pages and graphics memory included, from one `top`
  the host agent keeps running (about 1.8 % of a core at 5 s). A
  process `top` has not listed yet (started since its last sample)
  shows `N/A` there and has no place in the memory section. If `top`
  cannot run, the column falls back to `rss`.
- It is headed `rss` on Linux: the resident set, where shared pages
  count in every process holding them. An app 3pm manages has its PSS
  in `3pm show`; this table does not.
- `in/s` and `out/s` are bytes per second (`kb` = 1 024 bytes). `N/A` is
  a direction the platform could not measure for that process, not
  silence. On Linux they are TCP only, and cover other users' processes
  only when the daemon runs as root; without `ss` there is no network
  section.
- A section lists only processes with a figure above zero; one with
  none is not drawn.
- A process appears once it has been seen twice, so the first five
  seconds after the daemon starts show no table, and a process that
  lives less than a period never shows.
- Names are the executable's, never a path or arguments.

**Turning it off.** `[hostmetrics] top_processes = false` in
`config.toml` then `3pm update`, or `THREEPM_HOSTMETRICS_TOP_PROCESSES=0`
in the daemon's environment then a daemon restart (`3pm kill`, then any
command: a handoff keeps the old environment).
`THREEPM_HOSTMETRICS_TOP_MS` moves the period (5 000 ms).
A linked node sends the table to its backend: the names and owners of
the heaviest processes of the box.

**Limits.** No per-process disk I/O, no history, no UDP on Linux. On
macOS a long name whose executable was deleted since it started (a
rebuilt binary still running) shows cut at 16 bytes.
Details: `docs/hostagent-tiers.md` §3.8, `docs/metrics.md`.

## Network traffic per app (`net ↓/↑` in `3pm ls`)

**What you see.** `3pm ls` has a `net ↓/↑` column: what each app
received and sent per second over the last five seconds
(`1.2mb/s 310kb/s`). `3pm ls --tree` gives each process its own, a
`×N` group the sum of its members. `3pm show` adds a `net` line with
the totals since the app started (`↓ 1.2mb/s ↑ 310kb/s (total 4.1gb /
880mb)`), and a `net ↓/↑` column to its ` Processes` table (empty for a
process that moved nothing; the first column to go on a narrow
terminal). `--json` carries `net` on every record and tree member
(`rx_bytes_per_s`, `tx_bytes_per_s`, `rx_bytes`, `tx_bytes`,
`partial`), and `3pm internals` says how it is measured. A linked
node sends it to its backend, which keeps the history
(`process.net.received` / `.sent`, per process group
`subprocess.net.*`).

```console
$ 3pm ls
┌────┬────────┬──────┬────────┬───┬────────┬──────────┬──────┬────────┬─────────────────┬───────┐
│ id │ name   │ pid  │ uptime │ ↺ │ status │ watching │ cpu  │ mem    │ net ↓/↑         │ ports │
├────┼────────┼──────┼────────┼───┼────────┼──────────┼──────┼────────┼─────────────────┼───────┤
│ 0  │ api    │ 4000 │ 2h     │ 0 │ online │ disabled │ 3.1% │ 84.2mb │ 1.2mb/s 310kb/s │ 8000  │
│ 1  │ worker │ 4012 │ 2h     │ 0 │ online │ disabled │ 0.3% │ 18.0mb │ 0b/s 0b/s       │       │
└────┴────────┴──────┴────────┴───┴────────┴──────────┴──────┴────────┴─────────────────┴───────┘
```

**How to read it.**
- An app's figure is what it exchanged **with the outside of the app**:
  its whole process tree, each socket counted once however many members
  hold it, and a connection between two of its own processes (Electron
  and its `node`, `npm` and its server) left out — each of its bytes
  would count twice, once per end. Traffic with another app or a local
  database counts. A member's own figure counts every socket it holds,
  those connections included, so the members may add up to more than
  their app (an inherited socket, a conversation inside the app) or less
  (below).
- Short connections count. On Linux the daemon reads the host's TCP
  sockets every 5 s (`sock_diag`, no fork, no root) and hears every
  socket's close from the kernel with its final counters; on macOS it
  keeps one `nettop` running, which reports the closed flows too.
- **Linux, one blind spot**: an *outbound* connection opened and closed
  between two walks of the app's processes (10 s; 30 s for an app
  holding thousands of sockets) is never seen alive, and a closed socket
  no longer says whose it was: it is not counted. A longer one is found
  at the next walk and counted whole. An inbound one (a server
  answering) is always the app's: its local port is one the app listens
  on. When several apps share that port (`SO_REUSEPORT`), it is nobody's
  in particular — `3pm internals` counts it as `unattributed`.
- **What it costs is your apps'**, not the host's: the kernel filters
  both what the daemon reads and the close notifications it hears on
  your apps' ports, so a busy nginx next door costs nothing; an app
  holding 10 000 connections costs about 1 % of a core.
- Linux counts TCP, "sent" being what the peer acknowledged; macOS
  counts what `nettop` does, TCP and UDP, loopback included.
- Empty cell: no figure yet (the first ten seconds of an app, or of the
  daemon) or no source on this platform — never a guessed `0`. `≥`
  before a figure: the kernel dropped some notifications that window
  (`3pm internals` counts them), the figure is a lower bound.
- The totals restart from zero with the app, and with a daemon update.
- The column is drawn once some app has a figure (not on a platform with
  no source, nor from an older daemon), and goes right after `ports`
  when the terminal is narrow.

**Limits.** No UDP on Linux; the outbound short connections above (and
a connection inside an app that opens and closes between two windows
counts once, on its server side); UDP inside an app not paired; nothing
on other platforms. eBPF would close the
Linux gaps, as root only: not done.
Details: `docs/metrics.md` ("Network traffic per app").

## Responsive `3pm ls` (table and host line fit the terminal)

**What you see.** The process table and the `host metrics` line under it
adapt to the terminal width instead of wrapping cells (`statu`/`s`,
`watch`/`ing`). Columns are dropped whole, lowest priority first —
`ports`, `watching`, `pid`, `uptime`, `↺`, `mem`, `cpu` — until the
natural widths fit; `id`, `name` and `status` are never dropped, and only
a very long `name` may still wrap (never below 8 columns). Widths are
terminal columns, so a full-width name (CJK, emoji) gets the column it
needs on screen. The host line sheds segments the same way: `disk`, then
NICs, `gpu`, `load`, `ram`; `cpu` always stays. Piped output (`3pm ls |
grep`) is not a tty and is never cut: every column, every segment, whole
names, no color — the same for `3pm fleet`'s listings. A
reader that stops early (`3pm ls | head`, `3pm logs | head -20`) ends
the command silently with exit 0 — every 3pm command survives a closed
pipe, streamers included.

```console
$ 3pm ls          # 60-column terminal
┌────┬──────┬──────┬────────┬───┬────────┬──────┬────────┐
│ id ┆ name ┆ pid  ┆ uptime ┆ ↺ ┆ status ┆ cpu  ┆ mem    │
╞════╪══════╪══════╪════════╪═══╪════════╪══════╪════════╡
│ 0  ┆ api  ┆ 4000 ┆ 2h     ┆ 0 ┆ online ┆ 0.3% ┆ 42.1mb │
└────┴──────┴──────┴────────┴───┴────────┴──────┴────────┘
host metrics | cpu: 12.3% 52.4º | load: 1.23 | ram: 60.1%
```

**Where it comes from.** `render/table_layout.rs` measures every
column's widest header/cell (one `Col::text` source shared with the
painter, so measuring and rendering can never disagree) and picks the
columns; `render/table.rs` pins each kept column to that width so
comfy-table never squeezes them. `render/hostline.rs::fit_segments`
drops ranked segments by `visible_width` (ANSI-stripped). Property test:
for any width ≥ 26 and any names, no rendered line exceeds the terminal.

## Live table: `3pm ls --watch`

**What you see.** pm2's `pm2 ls --watch`, without the blink: the table
(link line, orphan notices and `host metrics` line included — exactly
what `3pm ls` prints) is redrawn every 300ms under a `Last refresh:`
stamp, **in place, where the cursor is**. Whatever was in the terminal
above stays there; the counters, statuses and pids change under your
eyes; Ctrl-C leaves the last table on screen with the prompt under it.
`--interval <MS>` sets the period (100ms floor; 1 s by default with
`--tree`, the daemon's tick — a tree reply that overruns the frame's
budget is left out of that frame, with the host line and the orphan
notice). The terminal is
re-read at every tick, so the table follows a resize (columns dropped or
regained). Off a tty (`3pm ls --watch | head`) there is no cursor
control at all: frames follow each other, a blank line between them,
and a reader that leaves ends the command with exit 0. `--json --watch`
prints one JSON array per tick, one line each (NDJSON, like `logs
--json`). The command ends on Ctrl-C, a closed pipe, or a daemon that
goes away.

```console
$ 3pm ls --watch --interval 500
Last refresh: 2026-09-07T11:02:17+02:00
┌────┬──────┬──────┬────────┬───┬────────┬──────┬────────┐
│ id ┆ name ┆ pid  ┆ uptime ┆ ↺ ┆ status ┆ cpu  ┆ mem    │
╞════╪══════╪══════╪════════╪═══╪════════╪══════╪════════╡
│ 0  ┆ api  ┆ 4000 ┆ 2h     ┆ 0 ┆ online ┆ 0.3% ┆ 42.1mb │
└────┴──────┴──────┴────────┴───┴────────┴──────┴────────┘
host metrics | cpu: 12.3% 52.4º | load: 1.23 | ram: 60.1%
```

**Where it comes from.** `render/watch.rs::Screen` is pure: given the
frame text and the terminal size it decides how to reach the previous
frame — cursor up by the previous frame's height (`ESC[nA`), every line
closed by an erase-to-end-of-line, the last by an erase-below — and
remembers the height and width it painted for. Two cases cannot climb:
a frame taller than the screen (the terminal scrolled: repaint from the
top of the visible screen, pm2's shape minus the blink) and a width
change (the old lines rewrapped: print a fresh frame below, resume in
place from it). The climb is counted in terminal rows, not lines: a
failure note under the table carries the raw error text and may wrap,
and every row it occupies is climbed (`rows_of`: visible width over
columns) — counting lines instead leaked one stale `Last refresh:`
above the frame at every tick. `commands/ls.rs` composes one `Listing` per tick — the same
body as a plain `ls`, so the two can never disagree — and, when the
host-line reply timed out (the connection may still carry it),
reconnects instead of reusing the client.

## Process trees: an app is every process it spawned (`3pm ls --tree`)

**What you see.** The `cpu` and `mem` of an app are those of its whole
**process tree** — the spawned pid and every descendant — not of the
spawned pid alone: an `npm run dev` that starts Electron and a dozen
helpers reads `62.2% / 3.0gb`, not `0% / 71mb`. `3pm ls` writes `saw
+13` when an app has other processes; `3pm ls --tree` (`-t`) draws them
under the row, compacted — a chain of single children is one line (`npm
› bash › node › Electron`), twins are grouped (`Electron Helper ×4`) —
and `--all` unfolds everything, one row per process. `⚠` marks a
process that left the app's process group, which `3pm stop` will not
reach. `3pm show` says `cpu 62.2% (14 processes)`, `mem 3.0gb (14
processes, RSS sum)` and lists every process under ` Processes`.
`--json` gains `proc_count` and `tree_rev` on every record and, with
`--tree` or in `show`, a `tree` object per app. The resource guards
judge the tree too (`limit_scope`, see *Resource limits*).

```console
$ 3pm ls --tree
┌────┬──────────────────────────────────┬───────┬────────┬───┬────────┬──────────┬───────┬─────────┬────────────────┐
│ id │ name                             │ pid   │ uptime │ ↺ │ status │ watching │ cpu   │ mem     │ ports          │
├────┼──────────────────────────────────┼───────┼────────┼───┼────────┼──────────┼───────┼─────────┼────────────────┤
│ 1  │ saw +13                          │ 12865 │ 5m     │ 2 │ online │ disabled │ 62.2% │ 3.0gb   │ 5002,6789,9222 │
│    │ └─ npm › bash › node › Electron  │ 12888 │ 5m     │   │        │          │ 1.4%  │ 323.1mb │ 5002,9222      │
│    │    ├─ Electron Helper (Renderer) │ 12963 │ 5m     │   │        │          │ 19.4% │ 777.0mb │                │
│    │    ├─ Electron Helper (GPU)      │ 12964 │ 5m     │   │        │          │ 10.2% │ 131.0mb │                │
│    │    ├─ Electron Helper ×4         │       │ 5m     │   │        │          │ 0.3%  │ 230.0mb │                │
│    │    ├─ ONYX Instrument Host       │ 12975 │ 4m     │   │        │          │ 25.1% │ 1.3gb   │ 6789           │
│    │    ├─ node                       │ 12990 │ 4m     │   │        │          │ 0.8%  │ 105.0mb │                │
│    │    ├─ python                     │ 12991 │ 4m     │   │        │          │ 0.6%  │ 121.0mb │                │
│    │    └─ ONYX Plugin Host ⚠         │ 13002 │ 4m     │   │        │          │ 4.4%  │ 90.2mb  │                │
└────┴──────────────────────────────────┴───────┴────────┴───┴────────┴──────────┴───────┴─────────┴────────────────┘
$ 3pm show saw | sed -n '/^ Processes/,$p'
 Processes (14, RSS summed: 3.0gb, cpu 62.2%)
┌───────┬───────┬────────────────────────────┬────────┬───────┬─────────┬───────┬────────────┐
│ pid   │ ppid  │ name                       │ uptime │ cpu   │ mem     │ ports │            │
├───────┼───────┼────────────────────────────┼────────┼───────┼─────────┼───────┼────────────┤
│ 12865 │ 4242  │ npm                        │ 5m     │ 0.0%  │ 71.3mb  │       │            │
│ 12880 │ 12865 │ bash                       │ 5m     │ 0.0%  │ 2.0mb   │       │            │
│ 12887 │ 12880 │ node                       │ 5m     │ 0.4%  │ 47.8mb  │ 5002  │            │
│ 12888 │ 12887 │ Electron                   │ 5m     │ 1.0%  │ 202.0mb │ 9222  │            │
│ 12963 │ 12888 │ Electron Helper (Renderer) │ 5m     │ 19.4% │ 777.0mb │       │            │
│ 12964 │ 12888 │ Electron Helper (GPU)      │ 5m     │ 10.2% │ 131.0mb │       │            │
│ 12965 │ 12888 │ Electron Helper            │ 5m     │ 0.2%  │ 61.0mb  │       │            │
│ 12966 │ 12888 │ Electron Helper            │ 5m     │ 0.1%  │ 58.4mb  │       │            │
│ 12967 │ 12888 │ Electron Helper            │ 5m     │ 0.0%  │ 56.1mb  │       │            │
│ 12968 │ 12888 │ Electron Helper            │ 5m     │ 0.0%  │ 54.5mb  │       │            │
│ 12975 │ 12888 │ ONYX Instrument Host       │ 4m     │ 25.1% │ 1.3gb   │ 6789  │            │
│ 12990 │ 12888 │ node                       │ 4m     │ 0.8%  │ 105.0mb │       │            │
│ 12991 │ 12888 │ python                     │ 4m     │ 0.6%  │ 121.0mb │       │            │
│ 13002 │ 12888 │ ONYX Plugin Host           │ 4m     │ 4.4%  │ 90.2mb  │       │ ⚠ detached │
└───────┴───────┴────────────────────────────┴────────┴───────┴─────────┴───────┴────────────┘
 ⚠ ONYX Plugin Host (13002) left the process group: `3pm stop` will not reach it
$ 3pm ls --json | jq '.[0].proc_count'
14
```

**How to get it.** Nothing to enable: every app has it once the daemon
runs a version with trees (`3pm update`; an older daemon renders `ls`
and `show` without trees and says `run \`3pm update\``). `-t`/`--tree`
and `--all` on `ls`, with `--watch` too (its period becomes 1 s, the
daemon's tick). `THREEPM_PROCESS_TREE=off` in the daemon's environment
restores the spawned-pid-only meaning for every app; `limit_scope =
"leader"` per app keeps the guards on the spawned pid while the figures
stay the tree's (an app saved by an earlier 3pm, in `dump.json` or
across `3pm update`, reads back as `leader`: an upgrade never widens a
limit). Which processes are an app's, what cannot be seen
(another user's processes, the environment of Apple platform binaries
on macOS, a nested daemon), the caps and every field: `docs/metrics.md`
→ *Process trees*.

**Where it comes from.** The daemon's metrics tick reads the process
table once per second (`sysinfo`, cpu and memory in one pass, off the
runtime; the whole table every thirtieth tick, in between only the pids
born or gone, the leaders, the members and 3pm's own — `metrics/table.rs`)
and attributes every process to an instance by, in order, the cgroup
leaf (`cgroup_enforce`), the parent pid, the leader's real process
group (orphans reparented to a subreaper), the memory of the members
already seen, keyed by `(pid, start time)` under `(id, gen)`, and the
environment the spawner injected (`THREEPM_ID`/`THREEPM_GEN`/
`THREEPM_CTL_SOCK` and the daemon's lineage `THREEPM_BOOT` — the classic
double-fork daemon, never a survivor of an earlier daemon of the same
home); then every descendant of a member, and never a process older
than the leader, nor 3pm's own daemon or host agent — `metrics/tree.rs`,
pure and table-tested, with a bench on a thousand rows. The totals land
on the record through `apply_metrics`; the tree itself stays on the
runtime and travels only on request (`process_trees {ids, gen?}`, 256
lines per tree and 2 000 per reply shared between the trees that list
members, `omitted` counting the rest, the totals always over every
member) — `supervisor/trees.rs`. The policy
receives both figures (`MemorySampled{gen, tree_rss, leader_rss}`) and
picks by `limit_scope` inside `step`, so the decision stays pure. The
CLI compacts through `threepm_protocol::tree::compact` (pure, in
`threepm-protocol` for the fleet dashboard to reuse; proptest: sums
conserved, every pid in exactly one row, `⚠` never hidden) and renders
in `render/tree.rs` (`custom_styling` for the dimmed `+N`,
`unicode-width` for the cut at the column's width, labels widening it
by up to 32 columns); snapshots from 40 to 120 columns, e2e `tests/e2e_tree.rs`
on a real shell with two sleeps. `3pm internals` shows `last metrics
tick` (`metrics_tick_ms`): about 12 ms for a thousand processes.

**A stuck process does not stop the sampler.** Reading the environment
of a process (Linux `/proc/<pid>/environ`) or its `io` /
`smaps_rollup` (Extended Monitoring) waits on the process's own
memory-map lock: on a process stuck in `D` state (a hung NFS mount, say)
the read never returns, and the whole tick — every sample, so
`max_memory_restart` and `max_cpu_percent` for every app — used to wait
with it, silently. These reads now run on a reader thread under a
500 ms deadline, within a 1 s budget per tick (`metrics/risky.rs`,
`metrics/guarded.rs`): a read that misses is retried on that process
with a backoff (1 to 60 ticks), not at all while its reader is still
stuck in the read, and the reads over the budget wait for a later tick
— meanwhile the process is not adopted through its environment and
keeps its last extended counters. The tick goes on, and at most four
readers are ever left stuck, daemon-wide, before both reads are
switched off until one returns (logged once each way). A probe not
made is no answer: the process is asked again on the next tick. The
refresh itself, and `process_start_time` on the supervisor, no longer
ask sysinfo for `io`, `exe` or tasks (on macOS `process_start_time`
skips sysinfo, whose refresh reads a new pid's `KERN_PROCARGS2`
unguarded — in the sweep, the stall report covers it). One sweep runs at a time; a sweep
that still runs past `max(5 × period, 5 s)` is logged as an error and
shown by `3pm internals`:

```text
│ last metrics tick     │ 12ms                                             │
│ metrics sweep         │ stalled for 83s — no samples, no resource guards │
```

(`DaemonInternals.metrics_stalled_ms`, counted from when that sweep
began, absent while healthy; the line goes when the sweep returns). The
loop keeps waiting for that sweep and stays cancellable. Tests:
`metrics/guarded_tests.rs` (deadline, reader replacement, the shared
cap, a panic told apart), `metrics/risky_tests.rs` (budget with reads
just over and under the deadline, backoff, quarantine while stuck),
`metrics/tests/hung.rs` (a real
tree with one member's read hanging: the others keep their figures; a
hung sweep reported stalled then recovered), the supervisor's
`internals_report_how_long_the_metrics_sweep_has_been_stalled`, and the
`internals` snapshots.

## Node.js client: `@3pm/api` (`packages/node`)

**What you see.** A zero-dependency Node library with pm2's programmatic
shape — `connect`, `start`, `list`, `describe`, `stop`, `restart`,
`delete`, `scale`, `save`, `launchBus` — promise-based, talking straight
to `rpc.sock` / `pub.sock`, auto-spawning the daemon when none is up:

```js
const { ThreePM } = require('@3pm/api');
const pm = await ThreePM.connect();
await pm.start({ script: 'app.js', name: 'api', instances: 2, max_memory_restart: '200M' });
const bus = await pm.launchBus({ process: 'api', subscribe: ['process:*', 'log:err'] });
bus.on('process:exit', (e) => console.log(e.exit_code, e.stderr_tail));
bus.on('log:err', (e) => console.error(e.line));
bus.on('daemon:shutdown', (e) => console.log('bye', e.reason));
```

Start options use pm2 names and units (`min_uptime`, `kill_timeout`,
`watch: true`, `interpreter` inferred from the extension and resolved to
an absolute path client-side, unknown keys rejected with a did-you-mean);
targets are ids, names or `all`. Bus events come as pm2-style aliases
(`log:out`, `process:online`, … — the two built from a field of the
event, `log.stream` and `process.kind`) *and* wire names, plus a raw
`event` catch-all so a newer daemon's kinds are never lost; the server-side
`EventFilter` handshake is used so unwanted events never cross the
socket. Daemon errors surface as `ThreePMError{code}`; `connect()`
checks `PROTOCOL_VERSION` and tells you to `3pm update` on skew.

**Where it comes from.** `packages/node/lib/{rpc,bus,spec,client}.js`;
`npm test` runs unit tests, decodes the frozen `fixtures/wire/v2`
fixtures of `threepm-protocol`, and drives a real daemon in an isolated
`THREEPM_HOME` (leak-proof, like the Rust harness).

## Modules

**What you see.** Optional features shipped with the install as
*modules* — directories with a `module.toml` — enabled per machine like
`pm2 install`: the bundled ones without any download, the others from
the download site. `3pm module list` shows what is
installed and what this 3pm bundles; `3pm install <name>` copies a
bundled module (or any local directory holding a manifest) into
`~/.3pm/modules/`, starts its `[[apps]]` and saves the process list —
on a module already installed it swaps the files and restarts its rows
in place (the sites it serves included; nothing is deleted and
recreated), and `3pm update` does the same for every installed module
whose bundled version changed, plus the apps running from the bundle
without an install (see *Zero-downtime update*); `3pm uninstall`
deletes its apps (and any app running from its files) and the
directory. `3pm <module> …` runs a module's command — the
installed copy, else the bundled one, so command-only modules need no
install at all — and unknown subcommands get a did-you-mean over
built-ins and modules. `3pm` passes `SIGINT`, `SIGTERM`, `SIGHUP` and
`SIGQUIT` on to the module and exits with its status; a `3pm` killed
outright (SIGKILL) leaves no orphan behind it — the module sees its
death as the EOF of an inherited pipe (`THREEPM_PARENT_FD`, every unix;
`PR_SET_PDEATHSIG` too on Linux) and stops through its own `SIGTERM`
path (`3pm fleet ssh -R` still closes the node's port). Module apps show in a second **Modules** table in
`3pm ls`, are targetable by name, and are skipped by `stop|restart|delete
all`, `delete --flush all`, `apply --prune-all` and `restart
--update-env`: install/uninstall own them.

**A module the release installs by itself.** `[module] auto = true` in
a **bundled** manifest means the module is part of 3pm the way a
built-in command is — `autoupdate` is the first one, so that any node
can be updated from the dashboard without someone ssh-ing in. 3pm then
does what `3pm install` would have done (copy, start the apps, save) at
the two moments it has a reason to look: on a daemon the CLI just
spawned — the first command on a fresh machine, and every boot without
a startup service — and after a `3pm update`, which is how a host
already running learns that a release added one. It says so once:

```console
[3PM] Module [autoupdate] installed (v0.32.0): it ships with this 3pm — `3pm uninstall autoupdate` if you do not want it
```

**And the host has the last word.** `3pm uninstall <name>` writes the
name into `<home>/modules.off` and nothing puts the module back — not
the next update, not the next boot — until `3pm install <name>` asks
for it again, which forgets the refusal. A module already installed is
left to the ordinary refresh; one that will not install is a warning,
never a failed command (`3pm ls` on a cold start still prints its
table); and a 3pm that predates the key ignores it.

**Modules that are not bundled come from the download site.**
`3pm install automonit` — a name that is neither a path, nor bundled,
nor installed — reads `<site>/modules/automonit/latest/version.json`
(`$THREEPM_CDN_URL`, `https://3pm.edgecr4ft.net` by default, the site
`autoupdate` reads), takes the archive built for this platform, checks
its size and its sha256 before opening it, unpacks it with every entry
checked (no absolute path, no `..`, nothing outside `automonit/`, no
link leading out, a `module.toml` that names `automonit`), and installs
it as `3pm install ./dir` would. `automonit@0.42.0` pins a version,
`automonit@latest` unpins, and `3pm update` takes a newer release of
every unpinned site module, in place; a site out of reach is a warning.
Only the configured site, only https (a `file://` or loopback mirror
aside), and the archive URL is rebuilt from the site, never taken from
the index. Downloads go through `curl`, else `wget`: no HTTP stack in
the binary.

```console
$ 3pm install automonit
[3PM] Downloaded automonit v0.42.0 (linux-x86_64) from https://3pm.edgecr4ft.net, sha256 verified
[3PM] Module [automonit] installed (v0.42.0) at /home/me/.3pm/modules/automonit
$ 3pm install automonitt
[3PM][ERROR] no module named `automonitt` (did you mean `automonit`?) — available: automonit; not on https://3pm.edgecr4ft.net either (cannot read https://3pm.edgecr4ft.net/modules/automonitt/latest/version.json: download failed (exit status: 22): curl: (22) The requested URL returned error: 404 Not Found); or pass a directory holding a module.toml
```

**How to get it.**

```console
$ 3pm module list
$ 3pm install serve               # or: 3pm install ./my-module
$ 3pm install automonit           # from the site; automonit@0.42.0 pins
$ 3pm serve ./dist 8080           # runs the bundled command even before install
$ 3pm uninstall serve
$ 3pm install --json serve        # {"module":{name,version,path,action},"affected":[…]}
```

```toml
# module.toml
[module]
name = "my-module"
version = "1.0.0"
command = "bin/my-module"         # optional: `3pm my-module …`

[[apps]]                          # optional: started by `3pm install`
name = "worker"
script = "bin/worker"
```

**Where it comes from.** `threepm-core/src/module.rs` (provenance,
naming rules, bundle roots), `StartSpec.module` (daemon-stamped, like
`source`), `threepm-cli/src/modules/` (manifest loader reusing the
ecosystem resolver, registry over installed + bundled, atomic install
copy, `3pm <module>` dispatch via clap's external subcommands),
`commands/module.rs`, `commands/module_refresh.rs` (the in-place
refresh shared by reinstall and `update`), `render/modules.rs`; daemon
shielding in
`supervisor/records.rs::resolve_mutation_target` and
`persist.rs::load_dump`. Contract and authoring guide: `docs/modules.md`.
Covered by unit/proptest at every layer, supervisor component tests
(`supervisor/tests/modules.rs`), `tests/e2e_modules.rs` with the fixture
modules under `tests/fixtures/modules/`, and snapshots of the Modules
table and `module list`.

## Static file server (`3pm serve`)

**What you see.** pm2's `serve`, as the first bundled module: a directory
served over HTTP by a supervised app that is *online* once its socket is
bound (`wait_ready`), listed with its port under **Modules** in `3pm ls`
(below the apps started by hand; `stop|restart|delete all` skip it, name
it to act on it), restarted like any app. Serving again on the same name
or port restarts the site with the new options. Without a port, 8080 or
the next free port above it is used. The command ends with the links to
open — `Local: http://localhost:<port>` and, when bound to every
interface, `Network: http://<LAN ip>:<port>` for a phone on the same
Wi-Fi — and a QR code of the network link (`--no-qr`). No process table.
`3pm serve page.html` serves the file's directory with that page as the
homepage, opens it in your browser (from a terminal; links only, no QR
code) and stays in the foreground: Ctrl-C stops the server and deletes the app (`--detach` to
leave it running; `--attach` gives a directory the same session
behavior, and the command also leaves when the app is stopped elsewhere). `--spa` answers unknown routes with the homepage (pm2's two-step
fallback: same basename at the root, then the homepage), `--ftp` lists
directories (index.html first, directories before files, dotfiles hidden,
a trailing-slash redirect so relative links work), basic auth challenges
with a 401, a `404.html` at the root replaces the plain 404, responses
carry pm2's CORS headers, HEAD works, other methods get a 405, bodies are
streamed. Path traversal is a 403 and a malformed percent-escape a 400
(GHSA-9prm-75wm-xpg6) — never a crash. A symlink that resolves outside
the served directory is a 403 too (and hidden from listings; one that
stays inside is served like its target). This is enforced by the open
itself, not by a check before it: the served directory is held open and
every file is resolved from that descriptor one component at a time
(`openat` + `O_NOFOLLOW`, `readlinkat` for the links met on the way,
`..` never past the root, an absolute target only when it spells a
location under the root — and then re-walked from the root, never
trusted), and the bytes streamed are read from the handle that walk
produced. Swapping a link between the router's decision and the read
changes nothing. A port already in use is reported
by the command instead of by a restart loop a minute later. Ecosystem
files written for pm2 keep working: `PM2_SERVE_*` env is honored.

**How to get it.**

```console
$ 3pm serve                              # ./ on 8080 (or the next free port), links + QR code
$ 3pm serve report.html                  # its directory, report.html as homepage, opened in the browser, Ctrl-C stops it
$ 3pm serve ./dist 8080 --spa --name site
$ 3pm serve --port 0 --ftp               # kernel-chosen port, directory listing
$ 3pm serve . --basic-auth-username admin --basic-auth-password s3cret
$ 3pm serve ./dist 8080 --host 127.0.0.1 --homepage app.html --namespace web
$ 3pm --json serve ./dist 0              # {"affected":[…]}
```

**Where it comes from.** `modules/serve` (own workspace and lockfile:
hyper 1, mime_guess, percent-encoding, base64): `options.rs`
(`THREEPM_SERVE_*` > `PM2_SERVE_*` > pm2 defaults), `router.rs` (pure
request → decision over an injected filesystem probe, property-tested to
never leave the root, lexically or through a symlink), `lookup.rs` (the
probe for real: a descriptor-relative walk with rustix — `openat`/
`readlinkat` — whose resulting handle is what gets served),
`listing.rs`, `server.rs` (hyper, streamed files,
clean exit on SIGTERM/SIGINT, readiness and `shutdown` through
`threepm_protocol::module::ctl`),
`start.rs` (the `3pm serve` side: `SpecPatch` → `Request::Start`,
busy-port check, pm2's output). Covered by unit/proptest per file and
`modules/serve/tests/e2e.rs` against a real daemon (real HTTP on a
kernel-chosen port: types, CORS, 400/403/404/405, escaping symlinks, SPA, ftp, auth,
`--json`, busy port, pm2 env).

## Fleet link agent (`3pm agent`)

**What you see.** The second bundled module links this 3pm to a fleet
backend (`3pm-backend`, the server side, its own repository). You paste one line from the
backend's bucket page — `3pm link` and `3pm unlink` are built-in sugar
over `3pm agent link` / `3pm agent unlink`: every argument is handed to
the bundled module untouched (`3pm link --help` is the agent's help),
the module owns the exit code, `--json` on either side of the verb
reaches it, and without an `agent` module the command fails with a
reinstall hint before any daemon is spawned:

```
$ 3pm link 3pmc_<cluster> 3pmk_<token> wss://fleet.example.com
$ 3pm link 3pmc_<cluster> 3pmk_<token>   # no server: the hosted backend, wss://fleet.edgecr4ft.net
$ 3pm link                    # the link: cluster, node, server, app, `link connected for 3m 2s`; exit 0 only when established
$ 3pm link stop               # pause: stop the agent app, keep the enrollment (saved, survives a reboot)
$ 3pm link start              # resume on the existing enrollment (registers the app again if it was deleted)
$ 3pm link status             # = 3pm agent status
$ 3pm link delete             # = 3pm unlink: disconnect, delete the app, forget the enrollment (also `rm`, `remove`)
$ 3pm unlink                  # = 3pm agent unlink
$ 3pm agent link 3pmc_<cluster> 3pmk_<token> wss://fleet.example.com
$ 3pm agent status            # cluster, node, server, dashboard, the app's row
$ 3pm agent unlink            # delete the app, save, forget the enrollment
$ 3pm agent status --json     # {"linked":…,"cluster_id":…,"node_id":…,"process":{…}}
```

`3pm link` alone (no positional, no `THREEPM_LINK_*` in the environment)
is a check: the status view plus a `link` line — `connected for 3m 2s
(session 2, server 0.1.0)`, with `, 3 unreadable frames` when this
backend and this agent have disagreed about a field (`frames_invalid`,
zero on a healthy link and then unsaid) — or `NOT established — <why>` (not linked, app
stopped or not registered, daemon unreachable, no session reported yet,
`link down for 12s (closed 4409 ERR_SESSION_REPLACED), retrying`,
`parked: the node was revoked`) — and exits 0 only when the link is up,
1 with `link not established: <why>` on stderr otherwise (`--json` adds
`session: {connected, for_ms, sessions, retries, parked,
last_disconnect?, server_version?, frames_invalid}`). The running agent writes
`<state dir>/session.json` on every change of its session and removes
it on a clean exit; the reader trusts it only when its `pid` is the
app's current one.

**The link line above the table.** A host that is linked opens `3pm ls`
with one line, pm2's `⇆ PM2+ activated | …` (a 3pm with no backend prints
nothing at all, and `--json` is untouched):

```
⇆ 3pm link | node 3pmn_wdcaepioyd6kflr5 | dash: http://127.0.0.1:9616/b/3pmc_dnhiue6kcwdsi4hn
⇆ 3pm link | node 3pmn_wdcaepioyd6kflr5 | link down for 5s (transport error), retrying
```

Green when the link is established, red with the reason when it is not —
the same verdict as bare `3pm link` (`threepm_link::local::status`, one
definition for the three readers). The address is the backend's page for
this cluster, derived from `link.toml`'s server (`ws://` → `http://`,
`wss://` → `https://`, path `/b/<cluster-id>`); `3pm agent status` prints
it as its `dashboard` line and `3pm link <…>` ends on it, so the bucket
page is one click from the terminal. The CLI reads `link.toml` and
`session.json` itself — no module spawned, no extra request, the app's
row comes from the list the table just printed — and drops the address
first on a narrow terminal, the reason last.

`link stop` and `link start` never run on a host that is not linked
(they point at the pasted line instead); `stop` on an app that was
deleted by hand says so and points at `link start`, which re-creates it
from `link.toml`. `--json`: `{"stopped":true,"app":…,"affected":[…]}` and
`{"started":true,"app":…,"registered":bool,"affected":[…]}`.

A provisioning script sets `THREEPM_LINK_CLUSTER` / `THREEPM_LINK_TOKEN`
/ `THREEPM_LINK_SERVER` (/ `THREEPM_LINK_INSECURE`) instead — the
environment **wins** over the positionals. The server is optional: named
neither by the third positional nor by `THREEPM_LINK_SERVER`, it is the
hosted backend `wss://fleet.edgecr4ft.net`, so `THREEPM_LINK_SERVER`
alone points every link of a machine at another backend. `ws://` to a
host that is not loopback is refused with `ERR_INSECURE_SERVER` unless
`--insecure`; a malformed token is reported without echoing it. `--name`
(default `agent`), `--cap
metrics:read|proc:restart|proc:stop|proc:action` (repeatable, default
all; a `link.toml` written by an earlier build (`version = 1`) holding
exactly the first three reads as all, since `version = 2` the list is
literal; a remote `restart`/`stop` never reaches a module app, the agent
included), `--no-env-keys`, `--force` (enroll again over an existing
link). What a call from the backend may do on this host is that `--cap`
list and nothing else: it is declared in `hello.caps` so the backend can
show it, and enforced here — the backend grants no capability. The
supervised app is configured by `THREEPM_AGENT_STATE_DIR` (default
`<home>/agent`) and `THREEPM_AGENT_NAME`; the pace of the link is the
backend's (`welcome.pace`: the state tick, the application metrics and
the process trees, each mandatory, each floored at a second), so the
`THREEPM_AGENT_INTERVAL_MS` / `THREEPM_AGENT_FULL_EVERY_S` an older
`3pm link` left on the app are not read, whatever they say; identity and server live in `link.toml`,
never in env (`3pm show` prints env).

`link` generates an Ed25519 node key and enrolls it in one request:
`POST /api/v1/enroll` (JSON over the same HTTPS the fleet routes use,
ten seconds for the whole of it) carrying the cluster, the token, the
public key, what this host is, the link versions this build speaks and a
proof signed over the token id and that key. The answer names the node
(`node_id`, `state`, the link version both sides share, the server's
clock) or refuses with one `ERR_*` code and its HTTP status. Nothing is
written before the answer is checked: a version outside this build's
range, or clocks more than five minutes apart (`ERR_CLOCK_SKEW`, the
delta shown), stops there — the operator learns of a skewed clock at
`3pm link` time rather than at the first handshake. Then `node.key`
(owner-only) and `link.toml` under `<home>/agent`, the supervised
`agent` app started (`wait_ready`, this binary as `__run`) and the
process list saved. A refusal (`ERR_TOKEN_EXHAUSTED`,
`ERR_TOKEN_INVALID`, a dead port…) exits 1 with nothing written. The
node's identity is that key and nothing else: no certificate, no cluster
CA — the backend holds the public half on the node's row and checks
every proof against it, and a host that was linked before this keeps its
node and simply drops the two files it no longer needs. An existing link
is kept unless `--force` (a fresh key, hence a new node id; the app is
restarted in place): the same line again — same bucket, same server —
says `Already linked … — kept` and exits 0 (`{"kept": true, …}` in
JSON), a link to another bucket or server exits 1.

**Install and link in one line.** The installer takes the same words
after `link`, and runs `3pm link` with them once 3pm is installed (by
its full path, so the shell's PATH does not matter). Optional: without
`link` it only installs or updates. Re-run as the updater, the same line
keeps the link:

```
$ curl -fsSL https://3pm.edgecr4ft.net/install.sh | sh -s -- link 3pmc_<cluster> 3pmk_<token> wss://fleet.example.com
$ curl -fsSL https://3pm.edgecr4ft.net/install.sh | sh -s -- v0.35.0 link 3pmc_<cluster> 3pmk_<token>
$ curl -fsSL https://3pm.edgecr4ft.net/install.sh | THREEPM_LINK_CLUSTER=3pmc_… THREEPM_LINK_TOKEN=3pmk_… sh
```

A refused link leaves the install in place and exits 1 with the command
to retry; the token is never echoed. `distribution/install.sh`. A reusable or long-lived token yields a *pending*
node, said so on the command line.

**musl hosts (Alpine).** The same line installs there: the installer
detects musl (no `getconf GNU_LIBC_VERSION`, a musl `ldd` or loader)
and downloads the static `3pm-linux-<arch>-musl` tarball, modules
included; `THREEPM_LIBC=glibc|musl` overrides the guess. A musl build
updates to musl through the autoupdate module. Not on musl: the
logged-in-users host metric (no utmp, always 0) and the agent's glibc
malloc tuning.

**Where it comes from.** `modules/agent` (lib + bin, own workspace and
lockfile; deps on `threepm-core`/`threepm-protocol`/`threepm-link`):
`cli.rs`, `inputs.rs` (pure resolution, env > positionals, the insecure
guard), `options.rs` (`from_env`/`to_env` round trip), `state/`
(`StateDir`: `node.key` created 0600 before any byte lands, temp + fsync
+ rename for every file; the *shape* of the two readable files and the
verdict over them live in `threepm-link`'s `local` module — `LinkConfig`
TOML with unknown keys ignored, `SessionFile`, `not_established` — since
the CLI reads them too for its link line, the agent's own half of
`session.json` (the fleet bearer, the count of unreadable frames) being
flattened next to it), `identity.rs` (the key and
the two proofs it signs), `tls.rs` (clear text or `webpki-roots`;
`--insecure` over TLS refused until the hardening step), `transport/`
(bounded `WebSocketConfig`, sub-protocol `3pm.link` — no version in the
name: `link_min..=link_max` is the whole of it, and a peer outside that
range is a 400 at the upgrade — HTTP rejections mapped to `ERR_*`), `codec.rs` (one deflate stream per connection,
`Sync` flush per frame), `enroll.rs` (the POST and what it makes of the
answer, 10 s), `start.rs` (`SpecPatch` → `Request::Start` → `Save`),
`link_cmd.rs`, `status_cmd.rs` (pure `view`/`render` + `Describe`),
`unlink_cmd.rs` (`Delete` tolerant of NotFound, `Save`, remove),
readiness and `shutdown` through `threepm_protocol::module::ctl`,
`run.rs` (`__run`: ready as soon as the identity is loaded — a backend
outage never makes the app `errored` — then the session until a
shutdown), `transport/handshake.rs` (three frames — challenge → hello →
welcome, the signature riding the `hello` it authenticates — 10 s,
version range and ±5 min skew checked before signing),
`transport/writer.rs` (the one writer: every frame — the node's state
included — queues on one bounded channel and is never dropped for being
late; a tracked frame tells its sender whether the bytes went out on the
socket, and one nobody waits for any more is skipped; ping every 20 s,
flush every 5 s, a full write buffer is a dead link),
`transport/reader.rs` (never blocks: a `call` is handed to the session,
which acknowledges it as its first act — so a call nobody could take is
never acknowledged — `error` kept for the close, a tag this build does
not know skipped, a body it cannot read counted (`frames_invalid`) and,
when it was a `call`, answered `err malformed` instead of being left to
die on its deadline; 60 s of silence — pongs count — is a dead link), `session.rs` (connect → handshake → pump → classify →
`next_delay_ms` with full jitter; a stable session resets the attempt;
revoked parks with an hourly probe; shutdown sends `Close(Normal)` and
waits 2 s for the echo; `THREEPM_AGENT_LOG` picks the log level),
`node_state.rs` (the one task that carries the node, see "The node's
whole state as diffs on the link" below: one `state` frame every
`welcome.pace.state_s` seconds — a second at the least, the wire's own
floor — from `List` + `HostMetrics`, then `AppMetrics` and
`ProcessTrees` for what moved, paced by `pace.metrics_s` and
`pace.trees` (`off` / `shape` / `figures {every_s}`), counted in ticks
by `node_state/paces.rs`;
records are redacted before they are shaped; while the daemon is down
the frame still goes, with an empty table and no `daemon`, and nothing
is said gone), `stream.rs` (what that task is made of: the tick clock,
what wakes the loop — the tick, a session status change (a `welcome`,
the link up or down), `state_resync`, shutdown — and the writer's
confirmation of a frame, waited 5 s), `daemon.rs` (the long-lived daemon
client of the node-state task, and of each of the commands' pool, one
connection each: every call bounded to 3 s, reconnection 1 → 30 s),
`events.rs` (the daemon bus filtered to `process`, `daemon_shutdown` and
`app`, decoded leniently, forwarded as `emit` without the stderr tail; a
`daemon_shutdown` is forwarded and then the agent closes politely;
resubscription 1 → 30 s). Covered by unit tests per file (writer and
reader under paused tokio time), `modules/agent/tests/session/` (the
session in process against the mock: handshake, reconnection ≤ 3 s after
a cut, 4403 revoked ⇒ parked, 403 at the upgrade ⇒ probe on the stop
schedule, ack before reply, a `call` this build cannot read refused
`malformed` and counted, enrollment refused for its own reason
(token, clock skew, a backend of another link generation), polite close)
and `modules/agent/tests/e2e.rs`: a real 3pm dispatching to the staged
module against a mock backend (`tests/support/mock_backend.rs`, its own
thread, per-instance cluster, enrollment over HTTP, link route with the
node looked up from the `Authorization` header, every agent frame
recorded — a frame link 4 cannot read fails the test where it happens —
orders: cut / close with a code / push a frame or a raw line) — enroll,
app online and in `dump.json`, state files and modes, `status`, refusal
without state, pending + `--force`, unlink, and the running app: the
signed `hello` seen, reconnect after a cut with the same boot id, clean close on
`stop` (exit path, no SIGKILL), new boot id on `restart`, parked when
revoked, and the node's state with a real sleeper: ≥ 3 `state` frames
under one `gen` with a clock that never goes back, every section whole
and with its CRC the first time it is seen, the host and the table
rebuilt from the frames alone, the daemon's pid and protocol version,
the sleeper with its env key *names* and no env value / home path /
script path anywhere in the captured frames, `emit {process}` on
`restart`, `emit {daemon_shutdown}` then Close 1000 on `3pm kill`. The
shared wire vocabulary is `crates/threepm-link`.

**The backend side (`3pm-backend`, step 11).** `GET /api/v1/link` admits
a node before any frame flows: `Authorization: 3pm-node
<cluster-id>:<node-id>` — both public, a claim and not a credential —
names the row, whose key and state are read from SQLite on every
handshake (401 `ERR_NODE_UNKNOWN`, 403 `ERR_NODE_PENDING` /
`ERR_NODE_REVOKED`), then challenge → hello → welcome (4401 / 4426 /
4408 / 4400 on refusal), the `hello`'s signature verified against that
row's public key. The session feeds an in-memory hub: presence `online` →
`stale` (3 s without a frame) → `offline` (30 s, tunable), the last
snapshot per node, a 300-point `{t_s, cpu, rss_kb}` ring per process, a
second session of the same node closing the first 4409, frames over 1
MiB closed 1009, 60 s of silence closed. `GET /api/buckets/{id}/events`
is the dashboard's SSE feed — `init` (the same shape as `GET
/api/buckets/{id}`: bucket, advertise, nodes with their displayed state,
snapshots, history), then `node` / `telemetry` / `emit`, monotonic ids,
`Last-Event-ID` replay of the last 4 096 events or a `reset`. The
dashboard (step 12) is three embedded files: `/` lists and creates
buckets; `/b/<id>` mints a token into the line to paste, shows every
node with its displayed state and, per node, the process table at 1 Hz
with the raw `telemetry` JSON under a `<details>`; `reset` reloads, the
browser reconnects by itself. The backend's `tests/it/real_agent.rs`
proves the whole vertical with a real `3pm`. Node API (step 13): `GET /nodes`, `POST
/approve`, `POST /revoke` (the live session is closed 4403 and the agent
parks itself); a host that knocks while pending or revoked leaves
`last_auth_error` on its row, so the page shows who is waiting and one
click admits it. See OV `3pm-backend-docs-archived/backend`.

### The node's whole state as diffs on the link (`state`)

**What you see.** Nothing on the host. On the wire, **one mechanism**:
the agent sends one `state` frame a second carrying only what moved in
the node — the host, the process table, each instance's application
metrics, each process tree — and an empty frame when nothing did: the
heartbeat. Measured on two real 10-minute traces (payload bytes on the
link, everything included): **357 B/s** on a dev Mac with 11 apps,
**463 B/s** on a Linux server with 8 — where the host and the table
alone cost 525 and 755 B/s as whole frames. It replaces the three
streams the link carried before (`telemetry`, `app_metrics`, `trees` —
and `host_patch`, a fourth that never left development builds), each
with its own revisions, gap detection,
resync message and safety net: they are gone, and so is link version 1
— an agent and a backend on different sides of that line refuse each
other at the handshake with `ERR_LINK_VERSION` ("update 3pm").

**Under the hood.** The daemon hands over **complete states** and cuts
no diff (its `app_metrics` answer is always whole; `app_metrics_rev`
and `tree_rev` only say whether asking again is worth it); the agent is
the only diff engine; the backend has one reader.
`crates/threepm-link/src/node_state/`: a node's state is a set of
**sections** (`host`, `procs`, `app/<id>`, `tree/<id>`), each an
independent JSON document with its own revision — a gap resends one
section, a section's full state fits one frame (nothing is chunked).
`mod.rs` is the engine, schema-less: `NodeWriter` (cut against the
state last **committed**, commit on the socket writer's confirmation,
`cut_gone`, `resync(secs)`), `NodeReader` (a patch applies only under
its `gen` and onto `rev - 1`; a `StateGap` leaves the section
untouched; a full patch with a wrong CRC is accepted and turns the
check off until the next session). `json.rs`: the canonical form
(`null` members dropped), tables `{"~r", "~o"?}`, rounding through the
decimal text, JSON Merge Patch, CRC-32. `host.rs` and `shape.rs` turn
the typed snapshots into sections and back: the host keyed by a fixed
path table at 3 significant digits; the process table keyed by id
**without `uptime_ms`** — the start is worked out once per
`(id, gen, pid)` (`Starts`; `created_at_epoch_ms` is the row's
creation, not the spawn) and the reader's uptime is the frame's clock
minus it — and without `app_metrics_rev`; metrics keyed by `sid` and
histogram buckets by index, **never rounded**; trees keyed by pid.
Wire: `AgentFrame::State(StateFrame {ts, gen, secs})`,
`ServerFrame::StateResync {secs}`, `welcome.pace`, the `state_frame`
fixture family (fixtures under `tests/fixtures/link/v4/`).
`modules/agent/src/node_state.rs`: one task, one tick, one daemon
connection — `List` + `HostMetrics`, then `AppMetrics` for the
instances whose `(gen, app_metrics_rev)` is not the one the backend's
section was shaped from, and `ProcessTrees` for those with more than
one process whose `(gen, tree_rev)` moved (their figures when
`pace.trees` is `figures {every_s}`, their shape alone when it is
`shape`, nothing at all when it is `off` — and then the sections held
are said gone). Everything is owed in full on a new session, on
`state_resync`, and when the daemon was replaced; a section carries its
CRC on every full patch and once a minute, each section at its own
moment of it (the ones that move every tick are not all checked on the
same one); what is no longer there (an
instance deleted, a tree back to one process, the host agent down) is
said `gone`, but nothing is while the daemon itself is down; frames
stay under the 1 MiB cap, a section never split. Cost at the most a
node manages (20 instances at the 1 000-series cap, all moving): 3.5
MB/s of whole states from the daemon at 0.28 ms an answer, ≈ 60–70 ms
a tick in the agent, 176 KB/s on the link; 20 × 50 series: 3 ms,
8 KB/s. Design and measures: `docs/backend-agent/link-protocol.md` §6
(the R&D plan that produced them is no longer in the tree). Tests: `node_state::`
(engine, JSON primitives, shapers — proptests through text), the
agent's `node_state::` (one frame a tick, a frame not written, resync,
gone, the cap) and `tests/session/state.rs` (the mock backend rebuilds
the node from the frames alone).

### Link traffic by category (`traffic`)

**What you see.** `3pm agent status` has a `traffic` row — `↑ 612 B/s
1.4 msg/s · ↓ 40 B/s 0.1 msg/s · overhead ~42 B/s est. · host 310 est.,
extended 170 est., procs 70 est., logs 12 … (as of 40s ago)` — and
`--json` the whole table (`traffic.counters`, `traffic.rates`), read from
`<state>/traffic.json`, which the running agent rewrites **at most once a
minute** (never `session.json`; removed by an unlink). The backend gets
the same counters as the `traffic` section of the node's state, every 5 s
(`max(1, 5 / state_s)` ticks), only the counters that moved; the `logs`
row gains `B sent`.

**What is counted** (`modules/agent/src/link_traffic.rs`, process-wide
relaxed counters, kinds and directions from `threepm_link::traffic`):
WebSocket up, a frame's compressed payload (what the backend's
`rx_bytes` counts) — the `hello` included; a `state` frame split over
its patches pro rata of their JSON length (`extended` and `top` carved
out of `host`, tree members' `x` out of `trees`), one message per patch,
a heartbeat one `control` message; down, a Text frame's payload by what
it decoded to (`calls` or `control`); pings one `control` message of 0
bytes, pongs not at all. HTTP: the body, one message per request (every
retry) and per answer — `logs`, `artifacts` (profiles, dumps, ssh
recordings), `blobs` (the relay's legs, per chunk), `api` (the rest of
`FleetHttp`). The per-frame **total is exact, the split an estimate**:
measured on a real node (`rd/logs-bytes-counter/lot0-preview.md`), ±5–10 %
on host and procs, ±25–40 % on the carved-out sub-values. Overhead
(28 B a message, WS header + TLS record) is shown, never counted. A
`3pm agent fleet …` command line counts nothing that is shipped.

**Tests.** The agent's `link_traffic::` (a host+tree frame's shares sum to
the bytes written; heartbeat, pings, hello, a decoded server frame; the
file at most once a minute), `http::logs::` (a retried POST counts
twice), `node_state::traffic_cut`, `status_cmd` (the row, `--json`) and
`tests/session/traffic.rs` (the section arrives; Σ up WebSocket bytes ==
what the mock backend received).

### The agent's threads: four workers, sixteen for blocking work

**What you see.** `3pm-agent __run` idles at **five threads** on any
machine, instead of one worker per core — 33 of them on a 32-core host.
`modules/agent/src/runtime.rs` names both figures. **Four tokio
workers**, because a worker can be *in* a transfer (the relay's copy,
the `data.sock` server, the WebSocket writer) and a node answers up to
`CALLS_IN_FLIGHT_MAX` calls at once. And a **blocking pool capped at
sixteen** instead of tokio's 512: the module itself hands that pool one
kind of work, `commands::pack` (an action's file read by 64 KiB steps
and gzipped, one bounded pass per call, cancelled when the call goes
away), but the pool is shared with the libraries underneath, and DNS
lives there — `tokio::net::lookup_host` on a cold dial, and hyper-util's
resolver under every HTTP client. A `spawn_blocking` cannot be
cancelled, so a lookup against a silent nameserver holds its thread for
as long as the resolver takes; sixteen keeps a burst of those from
queueing the packing of every call behind them. Five at rest is not a
ceiling: the pool spawns its threads on demand and retires them ten
seconds later, so the worst case is 1 + 4 + 16. The doc comment on the
constant carries the inventory, and a test walks `src/` and fails if a
new caller appears without being added to it. The host metrics agent
did the same thing first, 38 threads to 8
(`crates/threepm-daemon/src/hostagent/mod.rs`).

**What it is not.** It is not the memory fix. The resident set that
used to follow the bytes relayed was glibc's arenas, and
`MALLOC_ARENA_MAX=2` on the agent's own spec is what settled it (*The
agent runs with `MALLOC_ARENA_MAX=2`*, `src/arenas.rs`); pinning tokio
to four workers was measured there and moved nothing. What this buys is
the threads: a footprint that does not depend on the host's core count,
and 58 MB less virtual memory on a 32-core box — 28 thread stacks of
2 MiB. Throughput is unchanged: the relay is bound by the network long
before it is bound by four cores.

## Update 3pm from the dashboard (`autoupdate`)

**What you see.** A third bundled module, `autoupdate`, whose only
surface is two custom actions: a node updates itself when someone calls
`update` on it — from the fleet backend, from `3pm trigger`, from
`io.call` — instead of someone ssh-ing in to paste the install line.

```console
$ 3pm trigger autoupdate            # what it declares (nothing to install: see below)
$ 3pm trigger autoupdate check      # {"current":"0.32.0","latest":"0.33.0","outdated":true,…}
$ 3pm trigger autoupdate update     # {"started":true,"pid":40881,"from":"0.32.0","to":"0.33.0",…}
$ 3pm trigger autoupdate update '{"version":"0.33.0"}'   # or a named one (newer only)
$ tail -f ~/.3pm/autoupdate.log     # what the installer said
```

`update` installs a **signed release** and nothing else. It downloads
`v<X>/release.txt` — the version, then the sha256 of every archive and of
that release's own `install.sh` — and `v<X>/release.sig`, its ed25519
signature, and checks it against the two public keys built into the
module (`RELEASE_KEYS` in `modules/autoupdate/src/manifest.rs`: the key
`just publish` signs with on the publishing Mac, and an offline rescue
key). The manifest must name the version asked for, which must be
strictly newer than the running one; then the archive for this platform
and `v<X>/install.sh` come down and must match their signed sha256. Only
then does it run that `install.sh` (written to disk, never piped into a
shell) with `THREEPM_ARCHIVE` pointing at the archive already checked, so
nothing is downloaded twice. A bad signature, another version, a wrong
hash, a target the manifest does not list: nothing runs, and
`autoupdate.log` says why. Whoever controls the download site can
withhold a release, not forge one. The installer replaces `~/.3pm/bin/3pm` and its bundled
modules and ends on `3pm update`: the daemon hands off to the new binary
in place, the apps keep running, the modules are refreshed. `check`
reads `latest/VERSION` from the same site, the file `just publish`
writes last, and compares it with the version these module files shipped
with. `$THREEPM_CDN_URL` moves both to another site (the knob
`install.sh` itself honours), and `curl` — or `wget` — does the fetching:
the module carries no HTTP stack, as the installer carries none.

**The reply does not wait for the update.** It cannot: the update's last
step restarts every module row, this app included, so a handler that ran
`install.sh` in the app's own process group would be killed by the
command it launched and the call would answer `gone`. So the action
*starts* an updater and answers at once with its pid and the log path.
The updater calls `setsid` first thing, leaves the app's process group,
and outlives the restart — 3pm's `killpg` deliberately does not follow a
process that left (*`3pm stop` does not chase a `setsid` escapee*), and
this module's app sets no `cgroup_enforce`, the one thing that would
take it along. A `flock` on `~/.3pm/autoupdate.lock` is held by the
updater process itself, so a second `update` while one runs is refused
with the log path rather than doubled, and a crashed updater leaves no
stale marker behind.

A version named by the caller is checked before it reaches anything
(`^v?[0-9][0-9A-Za-z.+-]{0,31}$`): `{"version":"1.2.3; reboot"}` is a
failed call, not a shell. The log is appended to, rolled to
`autoupdate.log.1` past 512 KiB.

**Every hour, by itself.** The app does not wait to be asked: it reads
`latest/VERSION` every hour and, when a newer 3pm is out, starts the same
updater the action would — to that exact version, so a release published
between the check and the download cannot change what gets installed.
The first check comes within five minutes of the app starting (a jitter,
so a fleet restarted together does not reach the site in the same second),
a check is skipped while an update runs, and every `just publish` reaches
the fleet within the hour. `3pm logs autoupdate` says what it decided
(`0.35.0 is out (running 0.34.0): updating (pid 40881)`) and an
`autoupdate.started {from, to}` event goes to the feed.

A published version is **tried once**: the version is written to
`<home>/autoupdate.last` before the updater starts, and never tried again
by the timer. `current` is the version of the module's own files, so an
update that does not change them — it failed, or `install.sh` could not
hand off a daemon run from another binary (a development checkout) —
would otherwise re-run the installer every hour. The next release is a new
version and gets its own attempt; the `update` action ignores the record,
because a caller who asks, forces. `THREEPM_AUTOUPDATE_EVERY` in the
daemon's environment changes the interval (`30m`, `90s`, bare seconds) or
turns the timer off (`off`); the actions stay either way.

**How to get it.** Nothing: `module.toml` says `auto = true`, so the
module installs itself (*A module the release installs by itself*,
above) — on a fresh machine with the first daemon 3pm spawns, on a host
already running at its next `3pm update`. The install is saved, so the
app comes back at the next boot, and `3pm update` refreshes it with the
rest. A host that does not want it says so once, with `3pm uninstall
autoupdate`; `3pm install autoupdate` brings it back.

**How it works.** `modules/autoupdate/`, a standalone workspace whose
whole runtime is `threepm-io` (the Rust SDK — the module is an ordinary
3pm app using `io().action`), `rustix` for `setsid`/`flock` and the
installer's own `curl`: `src/worker.rs` (the supervised app: init,
declare, `io().serve()`), `src/actions.rs` (what the two calls answer),
`src/updater.rs` (the detached role, the lock, the log), `src/schedule.rs`
(the hourly check: `every`, `decide`, `tick`) and
`src/release.rs` (the site, the versions, `curl`/`wget`). Covered by
unit tests per file and `modules/autoupdate/tests/e2e.rs` against a real
daemon, with a release site on disk — `curl` fetches `file://`, so the
e2e drives the real code path and downloads nothing, including the test
that restarts the app under a running updater and still sees the install
finish.

## Fleet RPC (`3pm call`, `io.call`, `3pm fleet`)

**What you see.** A custom action (`io.action`, `3pm trigger`, see
*Custom actions and profiling* above) can now take bytes and answer with
bytes, and can be reached on another node of the bucket a linked host
belongs to — not just on this machine.

**The wire and the actions.**

- An app calls an action on its own behalf (`io.call`), of this host or
  of another node of the bucket. A call of this host sends a file by
  path and nothing is copied, so the path is the contract: when the call
  is placed the daemon opens it and judges what it finds — no link, no
  fifo, no device, no directory, no relative path, no file of another
  account, and a size the action accepts — then hands the handler that
  very path. It is an entry control, not a promise about the file's
  future: a caller that replaces its own file afterwards changes only
  what its own handler reads. A call that carries bytes to another node
  names the instance they go to, and never a path of this host.
- An action declares what it accepts: `input {max_bytes, ext}`, `stream`,
  and `concurrent` (1 by default, 16 at most) — and the most its file may
  hold, `output {max_bytes}` (1 byte to 256 MiB): past it `commitFile`
  fails `too_large` with both sizes before a byte leaves, and a caller
  on the fleet reserves that much of its blob quota instead of 256 MiB. The daemon runs at most
  `concurrent` calls of one action per source, 16 per connection, and —
  counted apart — 16 streams or 32 plain calls per daemon; a refused call
  names the rule it met. A source that declares no `input` is never given
  one, and the refusal says so.
- `3pm call` / `io.call` place a call with bytes attached, on this host
  or on another node of the bucket. The bytes travel as a blob, never on
  the 1 MiB control line; the transfer budget is the size at 128 KiB/s,
  clamped to [60 s, 24 h].
- `3pm link` gives the daemon a call gate: the agent's own app line gets a
  32-byte secret in its environment at every launch (never in the spec,
  never in `describe`, never in a log), and only a control connection that
  sends it back opens the gate. No gate, no route for the bytes: a call
  that names a node is refused at once, and the calls in flight when it
  closes end `gone`.
- A `live` blob is framed: 20-byte big-endian header (`len`, `flags`
  KEY/END/INIT/DISCONT, `hdr`, `seq`, `ts_us`), messages of at most 1 MiB,
  a longer header accepted and skipped. Frozen vectors:
  `crates/threepm-protocol/tests/wire_samples/live_frames.json`.
- **What one stream carries over a distance.** The agent relays a leg
  over one TCP connection to the backend, which makes a second one to
  the node at the other end. Measured under `tc netem delay 30ms` —
  60 ms of round trip per hop — one stream carries **5 MB/s at the
  latency floor**: a hundred 50 KiB images a second arrive with a p50 age
  of **61.0 ms** and a p90 of 61.3 against a floor of 60, ten of 1 485
  lost; at thirty a second, p50 61.2, p90 61.5, nothing lost. Above that,
  what the kernel's send buffer allows is **up to about 43 MB/s where the
  path permits it** — the buffer Linux sizes for that round trip
  (2 626 560 bytes here) divided by it, read off the socket and never
  measured end to end; the highest measured is the 5 MB/s above, which is
  all the camera offered. Until this lot the agent pinned that buffer at
  256 KiB to keep a frame's latency down, and the same stream capped at
  **4 MB/s**: the same hundred images a second delivered 4 of an offered
  5, p50 216 ms, 18 % of them dropped. `TCP_NOTSENT_LOWAT` keeps the
  latency now (`modules/agent/src/data/outbound.rs`) and nothing keeps
  the ceiling. On a platform without that knob (macOS, the BSDs, Windows)
  the buffer and the 4 MB/s both stand. Both ends carry the change: a
  fleet only half updated is still capped by the other one. The cost is
  kernel memory and not resident memory — `sk_wmem_queued` against
  `tcp_mem`, three tuned TCP sockets per node (the link session and two
  warm relay connections), auto-tuning settling near twice the
  bandwidth-delay product rather than at the `tcp_wmem` ceiling.

**The Node SDK.**

- The Node SDK places calls: `io.call(node, app, action, params, {file,
  output, instance, source, timeout_ms})` on another node of the bucket
  or on this machine (`node: null`, where the file is taken by path and
  nothing leaves the host). Bytes travel as a blob over the agent's Unix
  socket, hashed before the call and checked on arrival; failures are
  typed (`e.code`: `node_offline`, `too_large`, `daemon_too_old`, …).
- An action declares what it takes — `io.action(name, {input: {max_bytes,
  ext}}, handler)` — and its handler gets the file ready as `ctx.input`;
  `await ctx.commitFile(ext)` sends its own output back the same way,
  bounded by `output: {max_bytes}` when the action declares it.
- Streams: `io.stream(name, opts, (session, params) => …)` and
  `io.open(node, app, action, params)` exchange framed messages over two
  legs, with `io.mjpeg(readable)` cutting JPEG images as they arrive.
  Writing never blocks the producer: past a 100 ms window the oldest
  message that is not a KEY is dropped and counted (`session.stats`).
- `io.node` is this host's own node (`{id, cluster_id, server,
  linked_at_ms}` from the agent's `link.toml`, `null` when not linked).
  A live session cannot stay on the host, so `io.open(null | 'self', …)`
  goes through the fleet to that node (`node_offline` when unlinked);
  `io.call('self', …)` does the same, `io.call(null, …)` stays local. A
  `CallError` prints as `code: message`.

**From the command line.**

- `3pm call <app> <action> [params]` places one call on one instance and
  waits for the one answer it owns: `--file` attaches a file of this host
  (nothing is copied — the handler is handed that very path), `--out DIR`
  puts whatever the action produced where you want it, and `--node`
  places the call on another node of the bucket. `--json` prints one
  document: `{call_id, ok, node?, results, output?}`.
- `3pm trigger --file` is the same call with the trigger's own report, so
  it reaches one instance (`--instance`, 0 by default) instead of every
  one; `3pm trigger --out DIR` needs no `--file` and delivers the file of
  every instance that left one. The listing (`3pm trigger <app>`, and
  `3pm show`) now says what each action takes: `kind` (`call`, `stream`,
  `×N` when several may run at once) and `input` (the size it accepts and
  the extensions it lists).
- `3pm fleet nodes | apps | inspect | call | top` is sugar over the
  bundled agent (like `3pm link`): the arguments travel untouched, the
  module owns the exit code, and 3pm refuses early and plainly when there
  is no agent module or no link to read this host's node identity from.

**Fleet identity.**

- A linked node gets its own bearer for the backend's HTTP routes in
  every `welcome` (`3pma_<10 hex>_<64 hex>`, re-minted per handshake,
  valid for the session plus a grace window): `3pm fleet …` and a
  node-placed call authenticate with it, so the owner's account token
  never has to sit on a server. The token never reaches a log — its
  `Debug` shows `3pma_<id>_<redacted>` — and the backend keeps only
  `sha256(token)` in memory, so a revoke bites on the next request.
- A remote action can carry bytes: the link names blobs
  (`action {input, output, stream}`), never the bytes themselves, and
  the file an action produced comes back as `output {blob, bytes, ext,
  sha256}` for the caller to pull.
- `call_result` reports a call THIS node placed on another node of the
  bucket; the calls that settled while the agent was away come back as
  `welcome.pending_calls`.
- `3pm fleet nodes` (alias `3pm fleet ls`) `| apps [node] | inspect
  <node> [app]` on a linked
  host, authenticated as *that node*: the fleet's hosts with their
  state and how many apps each runs, every process of the bucket (or of
  one node) with its cpu, memory and restarts, and one node in full —
  identity, capabilities, its own grant, its process table. `--json`
  everywhere gives a stable document, not the backend's own.
- The `nodes` table also says **whose** each box is and where it is
  reached: `user` (the account that node's daemon runs as), `public`
  (the address its link comes from, as the backend sees it) and
  `private` (its interfaces' addresses, container bridges left out).
  `—` means the backend did not say, which is what a node whose agent
  predates the columns reads. Every fleet table is `3pm ls`'s
  (`threepm-table`): box style, colored state, and on a narrow terminal
  the lowest-priority columns drop first — `os/arch`, `seen`, `apps`,
  `user`, `private`, then `public` for nodes; `pid`, `uptime`, `inst`,
  `↺`, `mem`, `cpu` for processes — while `node`, `host` and `state`
  (`name`, `status` and the node) stay. Only when those alone overflow
  is the `3pmn_…` id (the app name) cut, with a `…`, to no fewer than
  twelve (eight) columns. Widths are terminal columns, so a full-width
  name gets its on-screen width rather than a second line, and a
  box-drawing glyph in a name a neighbour chose prints as `·`, never as
  a border. A pipe or a file is never cut — every column, whole names,
  no color — like `3pm ls`; `--json` carries
  `user`, `peer_ip` and `private_ips` untruncated, plus `sessions` —
  the accounts logged in on that node — which the backend sends only to
  a node granted `fleet:read`, and `inspect` shows `user` on its card. A node
  granted nothing reads nothing: the first answer names the missing
  scope and where its owner grants it. With `fleet:read` the card also
  carries the host, and an app's own series and declared actions.
  `--ca-file <PATH>` adds a certificate authority to the built-in roots,
  for a private CA or a root this build is too old to know.

**`3pm fleet top`.**

**`3pm fleet top`** — the bucket live in two panels (nodes, then every process
of the fleet), fed by the backend's `events` stream with the node's own bearer
and `fleet:list` alone. The column names and units of `3pm fleet nodes` /
`apps`, fewer columns (no addresses, no `pid` or `uptime`),
redrawn once a second whatever the stream is doing; `q`, `Esc` and Ctrl-C
leave. Reopens the stream on a `reset` or a cut, takes one second chance at a
bearer the resident agent has since refreshed, and closes naming the missing
scope when the backend refuses for good. Refuses `--json` / `THREEPM_JSON` and
a stdout that is not a terminal, before opening anything.

**The data socket and remote calls.**

**The data socket** — an app of this host that calls another node never
speaks to the network. It `PUT`s its input and `GET`s its output on
`agent/data.sock`, with a token the daemon put in its ticket, and the agent
moves the bytes: TLS, the backend's address and the node's bearer stay in the
agent. A token is worth one blob in one direction until its call settles; it
is checked when the request opens and never again, so a day-long tail
survives a rotated one. Nothing is buffered and nothing is hashed on the way
through — a hundred megabytes cross for 240 KiB of the agent's memory — and
the head of a `GET` is flushed before the first byte, so the receiver knows
it is connected before the sender has started.

A call that names its target by **hostname** rather than by node id is
resolved in the fleet gate, and the answer is kept for ten seconds: one
name costs one read of the bucket instead of one per call, which is what
`io.open` pays inside its establishment latency. An id the backend says
it does not know is dropped at once, so a node that re-enrolled under
the same hostname costs exactly one refused call.

**What the socket is, and is not.** There is one `data.sock` per agent,
mode 0600 in a 0700 directory: only a process running as the daemon's
user reaches it. A root daemon that hosts users (`[users] allow`) adds
one socket per hosted user, `agent/data-<uid>.sock` in that user's
hosted home (`/var/lib/3pm/users/<name>/`), owned by the user, 0600, in
a directory root:gid 0750: the daemon names them on the fleet gate each
time it opens (`hosted_socks`), the agent binds them on the same server
as `data.sock` — same tokens, same relay — and the SDK of a hosted app
finds its own with no `THREEPM_DATA_SOCK`. The SDK finds the hosted home
under the default `/var/lib/3pm/users` only (or `$THREEPM_USERS_DIR`,
which a hosted app does not inherit): with a custom `[users] dir`, set
`THREEPM_DATA_SOCK` in the app's `env`. An app run as another account
by a daemon that does not host it (a non-root daemon's `user:`) still
has no data plane and places no call that carries bytes.
The data sockets keep 512 requests in flight **together**, `data.sock`
and the hosted ones sharing one descriptor budget: the hosted sockets
take at most half of it between them (256) and 128 each, so `data.sock`
keeps its half and one hosted user leaves the others theirs. A caller
past the ceiling reads `503`; at most 64 of those answers are written
at once, past which the connection is closed unanswered, and a failed
`accept` (out of descriptors) pauses 100 ms rather than spinning.
Connections that have not yet spoken a header are counted apart, and
past 64 of them the agent's patience for a head drops from ten seconds
to a quarter of one — a crowd of silent connections costs its author
forty times the reconnects, and never costs an honest caller its request. Under systemd,
give the agent `LimitNOFILE` (or `ulimit -n`) of at least 4096: the 512
plus the warm connections plus the link. The SDK keeps three agents:
every write, sized or chunked, goes on one with no socket ceiling that
keeps its connection alive (so a refusal that arrives mid-body keeps
its reason instead of dying with the reset); a read of a blob the
caller does not know settled takes a connection of its own; and the
keep-alive pool of eight serves what fits in one round trip. A `PUT` is
answered when its body ends, so it has no
header budget at all — sixty seconds without a byte moving on the way
out ends it instead. When the backend's name resolves to several
addresses the relay gives each a quarter of a second before the next,
so a published address that is not routed costs that and not the whole
dial; on a link slower than that round trip with several addresses, a
dial can cost up to half a second more, once. Opening a call is about
seven round trips cold (TCP and TLS, `nodeAction`, the relays composing
theirs, the head of the `GET`) and three warm; a `live` frame, once the
session is open, crosses the relay in two round trips and under half a
millisecond, which is the figure the plan quotes. The `?k=` a `viewBlob`
mints is a bearer for sixty seconds, bound to one blob and nothing else
— not an address, not a session.

**Concurrency.** An agent runs at most four plain commands at once and,
counted apart, sixteen sessions — the daemon's own
`STREAMS_PER_DAEMON_MAX`, so a session this agent admits is never one
the daemon then turns away. Each refusal names its own budget ("4
commands already in flight", "16 sessions already in flight");
`CALLS_IN_FLIGHT_MAX`, the wire figure the backend sizes a session's
blobs on, is unchanged at four. A session whose call is refused after
the ticket ends on that reply: both legs close, `session.stats` keeps
its last values, and `frames()` throws the backend's `CallError` —
waiting for the reading leg's idle guard used to turn every late refusal
into a sixty-second timeout.

**Streams across the fleet** — `io.open` on one node and `io.stream` on
another exchange framed messages through the same socket: a chunked
`PUT` for what a side sends, a `GET ?live=1` for what it receives, two
per side and no state anywhere else. The agent cuts the stream it
relays on **message** boundaries, never on a byte count, so a reader is
never handed half a header; and when the link to the backend cannot take
the frames as fast as the app writes them, the agent throws its oldest
ordinary messages away rather than queue them — keeping the `INIT`, the
last `KEY` and the `END`, which are what a reader joining later needs,
and never the only message it is holding, however big: the queue's
ceiling bounds a backlog, not a frame. That is the second abandon sink
of a live stream (the SDK's is the first, at the source): in `live`
there is no back-pressure, by design, and a hundred megabytes of frames
cross the agent for no more memory than a hundred megabytes of file.
For that sink to see the congestion, nothing below it may hold much: a
`live` leg's upload rides **a connection of its own kind**, never one a
file used, with 16 KiB unsent in the kernel (`TCP_NOTSENT_LOWAT`, Linux
and macOS; a 64 KiB `SO_SNDBUF` where it is missing or refused), one TLS
record in rustls and hyper's 8 KiB minimum write buffer — against
64 KiB, 64 KiB and ~400 KiB for a file, which keeps its throughput. On
Linux it also asks for BBR (taken when the agent may: `CAP_NET_ADMIN`
or BBR allowed), whose window does not fill a modem's queue. A LAN
neighbour's direct leg is served the same way. Measured in process
(Linux, 1 MiB/s offered to a 100 KiB/s reader): the oldest frame to
arrive was 380 ms late, against 1.14 s on a file's connection — 30 s at
the 5 kB/s of a congested hotspot. The backend dates a message on
arrival, so this is where a frame late from upstream can still be told
late and dropped.
Bytes that are not `live` messages, or that stop in the middle of one,
are the writer's own mistake and are refused `422` — never `503`, which
would blame a node that is answering perfectly well. A session's tokens
live as long as the session, the 24 hours the backend holds it open,
whether or not the backend named the deadline itself; and a session
never carries an `output`, because its answer is its second leg — the
SDK, the agent and the backend all refuse one.

In the Node SDK a session ends the way an application expects: `close()`
cuts the reading leg and a `for await` over it simply returns, `end()`
half-closes (sends `END`, keeps reading until the other side's), and
`send(readable)` feeds a source — `io.mjpeg(ffmpeg.stdout)` — and ends
the leg cleanly when that source ends, is stopped or is killed, never
with an uncaught error. A received frame can be written straight onto
another leg, flags and capture time intact. Because frames are dropped
at three places and only the first one is local, the reading side counts
what never arrived from the holes in `seq` (`s.stats.missing`), and
measures latency from each frame's capture stamp (p50/p95/max over the
last 256 frames, clock skew included between two hosts); the sink's
`drop` event lets a producer lower its rate the moment it is too fast.

**`3pm agent fleet call <node> <app> <action>`** — one call on another node
from the command line, with this node's own identity: `--file` sends bytes,
`--out` keeps what came back and checks its digest, `--timeout` bounds the
wait. It talks to the backend directly, because the command line has no data
socket for the daemon to mint it a ticket on, and the call shows as
`node:<id>/-`. A hostname is resolved here; `--stream` is not from
here. The output lands through a temporary
beside it that only this call can have made (`O_EXCL`): a link someone
else left at that name is refused rather than written through, and so
is the leftover of a crash.

**One pass over the bytes instead of three (`opts.sha256`, `input.stream`).**
A hundred-megabyte call spends most of its time on SHA-256 in JavaScript
and on a copy to disk, not on the road: measured end to end, the source is
read once by `io.call` to find its digest (about 200 ms), written once by
the callee into a staging file, and read back a third time by a handler
that wants the bytes. Two opt-ins, both off by default, remove two of
those passes and leave the wire untouched.

*The caller that already holds the digest* passes it: `io.call(node, app,
action, params, { file, sha256, bytes })` declares those instead of
reading the file to find them again. The size is checked against the file
— one `stat`, and it catches the one mistake this invites, a digest kept
after the file changed under it. The digest is taken at the caller's word:
verifying it is the very work being skipped, and the callee checks it
against what lands anyway. Both or neither, lower-case hex, refused as
`refused` otherwise.

*The handler that reads the body* declares `io.action(name, { input: {
max_bytes, stream: true } }, …)`: nothing is staged and
`ctx.input.stream()` yields the bytes once, as they arrive from the
agent. A handler that never calls it never fetches them at all. The
declaration the daemon reads is unchanged — what the caller may send is
the same — because the choice is about this process, not about the
contract. The body is read once, a second `stream()` being `refused`.

Three differences to know. **`ctx.input.path` is `null` on a call from
another node**, there being no file; a call placed on this host still
names one, and `stream()` reads the daemon's own file, so one handler
serves both. **The size and the digest the call declared are checked at
the end of the body**, so a truncated or altered input throws out of the
iteration rather than refusing the call before the handler ran — and
nothing has vouched for the bytes until that iteration ends, so a
handler that stops early has used unchecked bytes (the SDK says so on
its debug channel). **The transfer counts against the handler's
`timeout_ms`**, which starts before a byte has moved, where the default
path hands the handler its whole budget with the bytes already in.

### `3pm fleet ssh`: a login shell on a node

**What you see.**

```
$ 3pm fleet ls
$ 3pm fleet ssh envision-surface-3997     # or 3pmn_…, or a unique prefix
root@envision-surface-3997:~# htop        # full-screen programs, colours, resize
root@envision-surface-3997:~# exit 3
$ echo $?
3
```

The login shell of the target agent's own unix user (root on the
tablets), on a real pty, in its home. `~.` after Enter disconnects
(`~~` types a `~`); the exit code is the shell's, `128 + signal` when a
signal ended it, 255 when the session ended any other way, with one
line on stderr saying how. Who may open one: the bucket's owner, or a
node holding `invoke` on the target whose `callers` rules — when it
has any — name the action `3pm:shell` (app `-`); `*` never covers it.
A target refuses with `node:shell` taken out of the `caps` of its
`link.toml`. Every opening and end is written to VictoriaLogs by the
backend, every command line by the agent (`kind = shell`).

**How it works.** `nodeShell` places a call whose answer names two
`pipe` blobs, lossless and back-pressured: the command line `PUT`s the
keyboard into one and `GET`s the screen from the other (one reader,
never retried), the target the reverse; both carry `attach` messages in
`live` framing. Either leg ending ends the session — the target hangs
the shell up, the command line exits 255. The command line pings every
20 s (a keyboard leg silent for 90 s is a dead one to the target), and
never awaits the keyboard leg: messages queue, the terminal is read no
more past 64 of them, and the screen keeps flowing, so a paste into a
shell that is not reading cannot wedge the two legs against each
other. While the screen leg is silent, the call is read back every 3 s:
a target that refuses (no `node:shell`, an agent too old, sixteen shells
already open) never writes on it.

**What the node does under a shell.**

- *The link reconnects* (backend restarted behind a proxy, a network
  blink): the shell is not touched. Its legs are HTTP requests of their
  own, not the WebSocket; output keeps flowing, and its end is answered
  on the new connection.
- *`3pm restart agent` or `3pm update` typed in the shell* goes through:
  the daemon restarts the agent, and the agent, before it closes its
  link, hangs up every shell it carries (`SIGHUP`, then `SIGKILL` 0.5 s
  later for one that ignores it — not the 5 s of a leg that ended: the
  shell is a session of its own, the daemon's kill of the agent would
  not reach it) and delivers their end. The session
  ends as a hang-up — the command line exits **129** (`128 + SIGHUP`),
  the call settles `exited` with signal 1 — and the next `fleet ssh`
  opens on the restarted agent. The shell's own `3pm` command does not
  print its result: it dies of the hang-up with its shell, after the
  daemon took the order. The wait is bounded (1.2 s, under the daemon's
  1.6 s kill timeout); an agent killed outright (`SIGKILL`) closes the
  pty and the kernel hangs the shell up, and the command line exits 255.
- *A macOS target* works like a Linux one (the account's login shell,
  zsh by default).
- *Between LAN neighbours* the session's bytes may go agent to agent
  instead of through the backend. The command line asks for it when
  its host's agent runs (it finds that agent's `data.sock`); the backend
  agrees only when both nodes are set to `direct: all`. A session
  sealed end to end (the default, below) is recorded by the target's
  agent, so it goes direct **recorded or not**; a session in clear
  (`--no-e2e`, an old agent) is recorded off the relay, so on a bucket
  that records shells it always takes the relay. The command line then hands its
  two legs to its agent (`POST /legs` on `data.sock`, the agent's own
  socket, never a hosted user's) and `PUT`s/`GET`s them there; the
  agent dials the target for the screen, the target dials it for the
  keyboard, each with a 1 s hold before the relay takes the leg —
  whole, never split. The call, its audit and `exited` stay on the link
  as always; the call says `path: direct` once a leg went that way.
  When the backend named a neighbour but this host's agent would not
  take the legs (an agent too old for `POST /legs`, one with no LAN
  listener), the command line says so once on stderr and goes through
  the relay; `3pm fleet ssh -v` says which way every session's legs go,
  and why the relay (no socket at its path, no neighbour named, the
  agent's refusal).
  Same for `-- <cmd>`, `fleet attach` and `fleet cp`; `-L` forwards
  keep the relay for now.

**Where it comes from.** Command line: `modules/agent/src/fleet/ssh.rs`
(placement, legs, the raw terminal with its panic hook and the
signals that end a session from outside, refusals by code), `fleet/ssh/session.rs` (the loop, with no HTTP and no tty, which
the tests drive on a paused clock), `fleet/ssh/escape.rs`. Target:
`modules/agent/src/shell/` (`host.rs` permits and replies, `open.rs`
the login, `pty.rs`, `pump.rs`, `audit.rs`, `legs.rs` the LAN way of
the legs). LAN direct, caller side: `fleet/ssh/lan.rs`, `data/legs.rs`. Wire: `node:shell`,
`AgentRequest::Shell`, `AgentResponse::Exited`, `AgentEvent::Shell` in
`threepm-link` (`docs/backend-agent/link-protocol.md`).

### `3pm fleet ssh` end to end: the backend relays what it cannot read

**What you see.**

```
$ 3pm fleet ssh web-1
The authenticity of node 'web-1' (3pmn_…) can't be established.
Node key fingerprint is SHA256:Qm9v….
Continue connecting (yes/no)? yes
root@web-1:~#
$ 3pm fleet ssh web-1 --accept-new -- uptime   # scripts: pin without asking
$ 3pm fleet known-nodes ls                      # id, fingerprint, name, date
$ 3pm fleet known-nodes rm web-1                # forget it (ssh-keygen -R)
$ 3pm link status                               # on the node: "shell key SHA256:…"
```

Every mode — the shell, `-- <cmd>`, `fleet cp`, `fleet attach`, `-L` —
is sealed between this 3pm and the target's agent by default: the
backend relays keystrokes, screen, `cp` files and forwarded bytes it
cannot read, and cannot change a byte in flight without the session
ending (255, "the encrypted channel broke"). The node's key is pinned
at the first session (trust on first use) in `~/.3pm/known_nodes`,
one line `<node_id> <fingerprint> <name> <date>`, keyed by the node id:
the first contact asks on `/dev/tty`, or pins silently under
`--accept-new`, and with neither (no terminal) is refused. A key that
**changed** is always refused, the offending line named, exit 255 —
ssh's `NODE IDENTIFICATION HAS CHANGED`; `known-nodes rm` and the next
session asks again. The file is never filled from the backend: it is
what the pin guards against.

The id alone would let the backend dodge the pin by answering the
hostname you typed with **another node id**: a key it shows there is
a first contact. So a hostname typed in full (case aside) that is
already pinned under another id **of the same cluster** is refused like
a changed key, before any question and under `--accept-new` too: the
banner names both nodes and says `3pm fleet known-nodes rm <pinned id>`
(by id: `rm` by name is case-sensitive) for a node that was really
re-linked. One file serves every bucket, so each line carries a fifth
column, the cluster id; a line of an older 3pm has none and is compared
in every cluster, and an older 3pm reads the new lines. The limit: only
a hostname typed **in full** is held to its pin — a prefix (`3pm fleet
ssh we`) names whichever node the bucket answers, and an id is checked
by its key alone. `fleet cp` pins the node as its paths name it, like
`ssh` and `attach`.

**Old agents, old command lines.** An agent of this version refuses
every session in clear, with no setting to relax it: a 3pm that
predates the channel is told "this node requires end-to-end
encryption: 3pm >= x.y" (the agent's code `e2e_required`). The other
way round, a node whose agent cannot encrypt is refused before
anything opens (`409 ERR_NODE_NO_E2E`, naming its agent's version):
update it, or pass `--no-e2e`, which exists for that case only — the
session goes in clear, as every session did before, after a warning on
stderr; an up-to-date node refuses it. Both refusals exit 255; the
command line tells them by **code** (the backend's `ERR_NODE_NO_E2E`,
the agent's `e2e_required` / `e2e_broken` in the call's `result.code`),
never by their words.

**Recording, by the node.** Off unless the caller passes `--record`;
then, on a bucket that records shells, the target's agent writes the asciicast **before** sealing — the same
file, by the same writer, as the backend's for a session in clear:
the screen only, never the keyboard, never a forward, 64 MiB at most,
the session never waits on it — and uploads it as the call's artifact
before it answers `exited {recording}`. An agent killed mid-session
leaves no recording: the backend's `ended` line says `recording:
lost`. A sealed session goes LAN direct recorded or not.

| A compromised backend can… | in clear (`--no-e2e`) | end to end |
|---|---|---|
| read a session live (keys, screen, `cp` files, `-L` bytes) | yes | **no** |
| inject keystrokes, change a file in flight | yes | **no** (AEAD: the session ends) |
| pose as the node | yes | **no**, except at the very first contact (TOFU) |
| read the stored recording and audit lines | yes | yes (by choice: readable) |
| see metadata, sizes, timing; refuse to relay | yes | yes |
| **open** a session on a node with its fleet token | yes | **no** (the node's `authorized_keys`) |
| change what an admitted session runs (argv, `--user`, `host:port`, `record`) | yes | **no** (bound to the handshake) |

The last two lines are the next section's: a node opens a session only
for a caller key in its own `authorized_keys`, and only for exactly the
request that caller placed.

What "readable" covers: the recording is what the node wrote on the
screen leg. A copy (`3pm fleet cp`, up or down) is never recorded —
a download's screen is the file — by the backend nor the agent: both
recognise its fixed scripts byte for byte
(`threepm_link::cp::is_cp_transfer`: `sh -c <script> 3pm-cp <paths…>`),
and an argv one byte off is an ordinary command, recorded. Nothing is
hidden by it: the scripts only move bytes between the legs and the
named paths, and the audit keeps the command line, paths included.
`-L` is never recorded either.

**How it works.** `Noise_XX_25519_ChaChaPoly_BLAKE2s` (crate `snow`,
pure-Rust primitives), prologue = a label (`3pm fleet ssh e2e v2`), the
call id, the target node id and the canonical encoding of the request
(next section), so a handshake cannot be replayed on another call, two
sessions spliced, nor the request changed on the way. Three messages before any session byte: `e` on the
keyboard leg, `e, ee, s, es` on the screen leg, then `s, se` once the
node's key is checked against `known_nodes`; the node starts the shell,
the command or the forward only after it. Then each leg carries records
`[u16 BE length][Noise message]` (at most 65 519 clear bytes each) in
place of its raw bytes; the legs are lossless and ordered, which gives
the nonces. Keys: `shell.key` (X25519, 0600) in the agent's state
directory, `cli.key` beside `known_nodes` in the caller's `$THREEPM_HOME`
(moved there from the agent's state directory on first use, so `3pm link
delete` no longer changes it), each created when first needed.
Wire (additive): `e2e` and `record` on `shell`/`exec`/`attach`/
`forward`/`run_as`, `shell_e2e` in the agent's `hello`, `cli_key` on
the `open` audit event, `recording` on `exited`.

**Compression, inside the channel.** `3pm fleet cp` compresses by
default (`--no-compress` to send the bytes as they are), and `3pm fleet
ssh -C -- <cmd>` asks for it when the command runs without a pty (`-T`,
or stdin not a terminal); `-v` says whether the session is compressed,
and why not. Both directions, zstd level 1, one frame per sealed record
(a flag byte, then the bytes raw or compressed, at most 65 518 clear
bytes): each frame stands alone, so nothing waits for more bytes, and
one that would not come out smaller goes raw — a random or already
compressed file costs one byte per 64 KiB. A frame never opens to more
than its 65 518 bytes (no decompression bomb). Measured on one core
(EPYC 7281): source text 3.0× at 210 MB/s, a system journal 10×.

Never a shell, a pty or `fleet attach`, `-C` or not: a keystroke would
wait on the codec, and a compressed stream that mixes what you type or
read (a password, a token on screen) with bytes someone else chooses
leaks it through its length (CRIME). Never `-L`/`-R`/`-W` either (`-C`
is refused beside `-W` and `-N`): their bytes are mostly TLS or already
compressed. `--no-e2e` has no channel, so no compression. The recording
of a recorded `-- <cmd>` is written from the clear bytes, before sealing
and compressing, so it does not change.

Negotiated in the handshake, never through the backend: the caller's
first message carries what it asks (`zstd`) as its payload, the node's
answer what it grants. Both payloads are hashed into the handshake, so a
backend that strips or adds the ask breaks it (`e2e_broken`), as any
other change does. An older agent ignores the ask and answers nothing,
an older 3pm never asks: either way, no compression and nothing else
changes.

**Where it comes from.** `modules/agent/src/e2e/` (the handshake,
records, keys, `compress.rs`), `shell/legs.rs` and `shell/host.rs` (the responder),
`shell/record.rs` (the node's recording, writer
`threepm_protocol::cast`), `fleet/ssh/e2e.rs` (the initiator),
`fleet/known_nodes.rs`, `fleet/ssh/refusal.rs` (refusals by code).

### `3pm fleet ssh` access by key: the node's `authorized_keys`

**What you see.**

```
$ 3pm fleet key                              # this host's caller key (no link needed)
SHA256:Qm9v…  (/Users/me/.3pm/cli.key)

On a node, to admit it:
  sudo 3pm link authorize SHA256:Qm9v…
$ 3pm fleet key new --out /Volumes/USB/rescue.key   # an offline rescue key
$ 3pm fleet key push web-1 SHA256:Zx… --comment rescue   # or --all
$ 3pm fleet key revoke --all SHA256:Zx…
# on the node itself:
$ sudo 3pm link authorize SHA256:Qm9v… --comment mac-unitech
$ sudo 3pm link deauthorize mac-unitech      # by fingerprint or comment
$ 3pm link status                            # the authorized keys, or why none opens
$ 3pm link 3pmc_… 3pmk_… --authorize SHA256:…   # at link time (or THREEPM_LINK_AUTHORIZE)
$ 3pm fleet ssh web-1 --record               # record it (off by default)
```

A node opens a `3pm fleet ssh` session — shell, `-- cmd`, `fleet cp`,
`fleet attach`, `-L`, `-W`, `-R`, `--user` — only when **(a)** the
caller's key is in the node's own `authorized_keys` and **(b)** what it
runs is exactly what that caller asked for. The fleet backend, a stolen
`3pmu_` token or another node can no longer open a session, nor change
the command of an admitted one. A refusal exits 255 for every mode
(`-L`/`-W`/`-R` stop at the first one), with the refused fingerprint and
the `sudo 3pm link authorize …` to type on the node; the backend's
`ended` audit line carries the same message.

**The file.** `<agent state dir>/authorized_keys` — `$THREEPM_HOME/agent/`,
or `THREEPM_AGENT_STATE_DIR`; `/root/.3pm/agent/` on a root node. One
`SHA256:<fingerprint> [comment]` per line, `#` comments; a line that does
not parse is skipped with a warning, never read as "everyone". Checked
like sshd's `StrictModes`: owned by the agent's user and not writable by
group or others (0644 is fine), else refused. **Absent or empty: every
session is refused** — a node is closed until someone authorizes a key.
Re-read for each session (at most 64 KiB): no restart. `link authorize`
writes it atomically under a lock, idempotently; `3pm link delete`
keeps it, and `shell.key`.

**The caller's key.** `cli.key` (X25519, 0600) beside `known_nodes` in
the caller's `$THREEPM_HOME`. `THREEPM_CLI_KEY=<file>` uses another
one as it is (a rescue key from `fleet key new`). This key is the fleet
access credential; it is not the release signing key of `just publish`
(`~/.3pm-release/`), which is another pair entirely.

**The request is bound.** The handshake's prologue holds the canonical
encoding of every field that decides what runs: the operation, the argv
and `pty`, the `--user` account, the attach target/mode/`read_only`, the
`host:port` of `-L`/`-W`, the listen and connection of an `-R` takeover,
the `bind:port` of `-R`, the shell's `term` and first size, and
`record`. The command line encodes what it asked for, the node what it
received; one byte apart and the handshake fails before anything
starts. An `-R` connection is taken over only by the key that opened
the listen.

**Recording is the caller's to ask for.** A session is recorded (by
the node, see above) only when the caller, the bucket and the node all
agree: the command line asks with `--record` — without it the body says
`record: false`, the default (`--no-record` is still accepted and says
the same; an older 3pm that says nothing leaves it to the bucket) — the
backend may only lower it (bucket `record_shells` off, `cp`, forwards) and returns
the effective value before the handshake, which binds it; the command
line says "this session is recorded (screen and command lines)" before
opening. A node with `[shell] record = "never"` in `link.toml` records
nothing whatever was asked, and says so in its answer. The command
lines go with the recording: a session that is not recorded (any of the
three said no) sends no `line` and no `screen` audit event — the node
does not even read them off the screen — only `open`, `exec` or
`attach`, and `close` with `lines: 0`.

**What it does not cover.** Remote actions of apps and modules
(`nodeAction`, `remote: true` by default), `3pm:profile`,
restart/stop, the first contact's TOFU, other unsigned files of the
download site (SDKs), an app running as root writing `authorized_keys`.

**Locked out?** With local access (real ssh, console, `sshpass` on a
tablet): `sudo 3pm link authorize SHA256:…`. From a machine that holds
the rescue key: `THREEPM_CLI_KEY=/Volumes/USB/rescue.key 3pm fleet key
push <node> SHA256:…`.

**Where it comes from.** `modules/agent/src/e2e/authorized.rs` (the
file, `StrictModes`, the lock), `e2e/bind.rs` (the canonical request),
`shell/host.rs` (the check, after the handshake and before the
session's permit: 16 handshake slots, 8 per caller `from` so one
stolen token cannot hold them all, 5 s for the first message),
`authorize_cmd.rs` (`link authorize|deauthorize`), `fleet/key.rs`;
wire `AgentErrCode::KeyNotAuthorized` (`key_not_authorized`). Proven
end to end against the real agent by the backend's
`tests/it/real_agent_access.rs` and `real_agent_exec.rs`.

### `3pm fleet ssh <node> -- <cmd>`: one command on a node

**What you see.**

```
$ echo x | 3pm fleet ssh web-1 -- wc -c
2
$ 3pm fleet ssh web-1 -- 'journalctl -u nginx | tail -3'
$ tar -c ./site | 3pm fleet ssh web-1 -- tar -x -C /srv
$ 3pm fleet ssh web-1 -- cat /var/log/big.bin > big.bin   # bytes, unchanged
$ 3pm fleet ssh -t web-1 -- htop                         # a pty, from anywhere
$ 3pm fleet ssh web-1 -- false; echo $?
1
```

The command runs through the target's login shell, as ssh's does
(`<shell> -c`, not a login shell: no profile runs). Several words are
each quoted for that shell, so `-- cat 'my file'` reads one file; one
word is the whole line, so `-- 'ls | wc -l'` is a pipeline there.
**A pty or pipes**: the command gets a pty when stdin and stdout are
both terminals, or with `-t`; `-T` never gives one. On a pty it is a
shell's session (raw terminal, `~.`, the size followed). On pipes it is
bytes: stdin is streamed and then closed (`echo x | …` ends), the
command's stdout goes to stdout and its stderr to stderr, unchanged —
binary included —, with no raw mode and no `~.`; a command that stops
reading its stdin (`head -c1`) has the rest dropped, not the session
cut; after its exit, a tail still on its way to a paused reader
(`… | less`) is waited for, a background job holding its stdout
(`cmd &`) for 5 s of silence. The exit code is the command's
(`128 + signal`), 141 and nothing said when stdout stopped being read
(`… | head -1`, as `SIGPIPE`), 255 with one line on stderr when the
session ended otherwise (a leg cut, the node refusing, `Ctrl-C` on the
command line). The same rules as a shell:
the `invoke` grant, `3pm:shell` in the target's `callers`, `node:shell`
in its `caps`; an agent that predates this answers "too old".
**Audit**: the backend's `open_requested` and the agent's `exec` line
both carry the line the shell is given (`cat '>' /etc/x` for
`-- cat '>' /etc/x`).
Never the command's stdin; on pipes, nothing off its output either
(with `-t`, the screen's lines as for a shell).

**How it works.** `nodeShell` with `command` (and `pty`) places an
`exec` call — its own op, not a field of `shell`: an agent too old to
know it refuses it, where one ignoring a field would start a shell and
take a script's stdin for keystrokes. Same two `pipe` legs; on pipes
two more `attach` kinds ride them, `eof` (`0x05`, the end of stdin)
and `stderr` (`0x1d`). Back-pressure holds both ways with no buffer
that grows: the target reads a pipe only once its last message is
queued for the screen leg, and the keyboard leg only once its last
input is in the command's stdin. After the command exits, what its
pipes still hold is delivered (a background job holding them is left
after 5 s with nothing moving), then its end. 20 MB through `cat` there
and back: about 0.1 s on loopback (Linux, debug builds).

**Where it comes from.** Command line: `modules/agent/src/fleet/ssh/command.rs`
(the pty or pipes decision, the body), `fleet/ssh/session.rs`
(`run_command`, `Mode`). Target: `modules/agent/src/shell/pipes.rs`
(the command between its legs), `shell/open.rs` (`command_line`,
`spawn_pipes`, `spawn_command`). Wire: `AgentRequest::Exec`,
`ShellEvent::Exec` in `threepm-link`; `ClientMsg::Eof`,
`DaemonMsg::Stderr` in `threepm_protocol::attach`.

### `3pm fleet ssh -l <user>`: as another account of the node

```console
$ 3pm fleet ssh -l deploy web-1             # or deploy@web-1
$ 3pm fleet ssh -l deploy web-1 -- id -un
deploy
$ 3pm fleet cp -l deploy ./site.tar web-1:  # into deploy's home
```

- **Where the agent is root** (a tablet, a server installed by the
  package), the session becomes that unix account: `getpwnam` +
  `getgrouplist`, its home, its shell, `HOME`/`USER`/`LOGNAME`/`SHELL`
  from its `passwd` entry; in the child, after the new session and its
  controlling terminal, `setgroups` → `setgid` → `setuid`, then a check
  that uid 0 is out of reach, then `PR_SET_NO_NEW_PRIVS` on Linux — the
  daemon's drop for hosted users, copied
  (`modules/agent/src/shell/user.rs`). The pty is chowned to the
  account (`0620`), so `tty` and `sudo` work in it. No new right: a root
  shell could already `su`.
- **Where the agent is not root**, only its own name is taken; any
  other is refused (`forbidden`, "the agent runs as `pi`, not root"),
  never run as the agent. An unknown account is refused (`not_found`),
  before any leg opens.
- **The name** is a plain login, `[a-z_][a-z0-9_-]{0,31}`, checked by
  the command, the backend (422) and the agent alike. `-l` and `user@`
  together are refused. Never with `fleet attach` (the app's terminal is
  the app's).
- **Old agents are safe**: the request is a new op, `run_as {user,
  req}`, wrapping the `shell` or `exec`. An agent that predates it
  answers "unknown command" (the command says the agent is too old)
  instead of ignoring a field and running the session as itself.
- **Audit**: the backend's `open_requested` / `refused` carry `user`
  (the account asked for), the agent's `open` line the account it ran
  as.

### `3pm fleet ssh -L`: a local port through a node

**What you see.**

```
$ 3pm fleet ssh -L 5432:db.lan:5432 web-1          # a shell, and the forward beside it
$ 3pm fleet ssh -N -L 8080:127.0.0.1:80 web-1      # no shell: forwards only, until Ctrl-C
3pm fleet ssh: -L 127.0.0.1:8080:127.0.0.1:80: listening on 127.0.0.1:8080
$ 3pm fleet ssh -N -L '*:9000:[::1]:9000' -L 6379:cache:6379 web-1
```

`-L [bind:]port:host:hostport`, as ssh's: a listener on this host
(127.0.0.1 unless a bind address is given, every interface for `*`; an
IPv6 address in brackets; port 0 lets the system pick, and `-N` prints
what it bound), and every connection it accepts is opened from the node
to `host:hostport` — any destination the node itself can reach (a name
is resolved there). Repeatable. With a shell the forwards live as long
as the session; with `-N` (no shell, no terminal needed) until
`SIGINT`/`SIGTERM`, exit 0. A port already taken fails before anything
is placed; a connection that fails (the node refusing, the destination
refusing — `connect db.lan:5432: Connection refused` —, a leg cut) is
one line on stderr and that connection closed, the listener and the
other connections carrying on. A half-close crosses both ways (a
client's `shutdown(WR)` is the destination's EOF and its answer still
comes back), bytes are lossless and back-pressured. `-L` with
`-- <cmd>` is refused for now. Who may: the `invoke` grant, the action
`3pm:forward` in the target's `callers` (not `3pm:shell`: a forward
is its own permission), `node:forward` in its `caps` — on by default;
take it out to refuse. A listener off the loopback (`*`, an address
of the LAN) prints a warning: whoever reaches it goes through the node
with your grant; on a shared machine any local user can reach
127.0.0.1 too, as with ssh's. **Limits**: forwards count against the
target's 16 sessions in flight, with its shells — the command line
carries at most 16 connections at once, the next ones waiting accepted
(a browser's six stay well under it, but each open forward is a shell
fewer); each connection is one call and two blobs, so the backend's
blob quotas (100 open, 200 created a minute per owner) bound a client
that opens a fresh connection per request to ~100 a minute (`429`,
said on stderr) — keep-alive clients are unaffected. A leg cut ends the
forward at once (no wait for the destination to close). **Audit**: the backend's
`open_requested` / `refused` with `forward = host:port`, then `ended`;
the agent's `close` line carries the bytes each way.

**How it works.** Each accepted connection is one `nodeForward` call
(`POST …/nodes/{node}/forward {host, port}`) placing `forward {host,
port, input, output, from}` on the target: two `pipe` legs as a
shell's, but carrying **raw bytes** in `live` messages (no `attach`
kind byte), a message flagged `END` closing its way. The agent
connects (5 s), pumps both ways, and replies `exited {code: 0}` once
both ways ended by an `END` (1 when a leg or the connection broke); a
refused connect is the call's `err`, the legs never opened (the
caller's `GET` answers 408 and the command line reads the call back for
the reason). Code: `modules/agent/src/shell/forward.rs` (target),
`modules/agent/src/fleet/ssh/forward.rs` and `fleet/forward_spec.rs`
(command line).

### `3pm fleet ssh -W host:port`: the real `ssh` through a node

**What you see.**

```
$ 3pm fleet ssh web-1 -W localhost:22 --accept-new < request > answer
$ ssh -o ProxyCommand='3pm fleet ssh web-1 -W localhost:%p' envision@web-1
```

With this block in `~/.ssh/config`, every `<node>.3pm` host goes
through the node — `ssh`, `scp`, `rsync -e ssh`, `git`, ansible, VS
Code Remote-SSH:

```
Host *.3pm
  ProxyCommand 3pm fleet ssh $(echo %h | sed 's/\.3pm$//') -W localhost:%p
```

```
$ ssh envision@envision-surface-7caf.3pm
$ rsync -a ./site/ deploy@web-1.3pm:/srv/site/
```

ssh runs the line with `$SHELL -c`, never an interactive shell: `3pm`
must be on the `PATH` (an alias does not count), or written as a full
path.

`-W host:port`, as ssh's: no shell, no listener — stdin goes to
`host:port` as the node reaches it, what comes back goes to stdout,
one connection, and the process exits once both ways are closed. It is
`-L`'s connection with stdin and stdout for the local peer: the same
`nodeForward` call, sealed end to end the same way, the same grant
(`invoke`, `3pm:forward` in the target's `callers`, `node:forward` in
its `caps`), the same audit; nothing new on the node or the backend.
So the real `ssh` (its own keys, `authorized_keys`, agent forwarding)
reaches a node's sshd with no route to it, no port opened, through the
encrypted relay. **stdout is the destination's bytes only**: every
message (the first-use question, which goes to `/dev/tty`, a refusal, a
caution about the bearer) is on stderr. A node met for the first time
asks on `/dev/tty`; give `--accept-new` where there is none.
**Exit**: 0 when both ways closed cleanly, 255 (as ssh) for anything
else — the node unknown, a refusal, the destination refusing
(`connect localhost:22: Connection refused`), a leg cut — with one line
on stderr. `-W` with a command, `-L`, `-N`, `-t` or `-T` is refused by
the parser. **An idle tunnel stays up** (`-W`, `-L` and `-R` alike): a
local peer silent for 20 s sends an empty message on the input leg, a
keepalive the node ignores, so the relay no longer gives the legs up at
its one-minute idle bound. The output leg is not pinged: while the
destination says nothing, a reverse proxy in front of the backend
still cuts it at its read timeout (`proxy_read_timeout` /
`client_body_timeout`, 1 h on fleet.edgecr4ft.net, 60 s for a default
nginx). Code: `proxy` and `carry` in
`modules/agent/src/fleet/ssh/forward.rs`; e2e
`3pm-backend/tests/it/real_agent_proxy.rs` (20 MB through stdin and
stdout, the same hash; the host's own sshd through `ProxyCommand`).

### `3pm fleet ssh -R [bind:]port:host:hostport`: a node's port, served from here

**What you see.**

```
$ 3pm fleet ssh web-1 -N -R 0:localhost:8000
3pm fleet ssh: Allocated port 41237 on web-1
$ 3pm fleet ssh web-1 -R 8080:localhost:8000      # with a shell on web-1
web-1$ curl localhost:8080                         # reaches this host's :8000
```

`-R`, as ssh's: the node listens on `bind:port` (`bind` left out: its
loopback; port 0: one it picks, printed as above), and every connection
it accepts comes back here, connected to `host:hostport` as this host
reaches it. Repeatable, beside `-L`, with a shell or alone with `-N`;
refused with `-W` or a command. **This side places every call**: one
sealed `nodeListen` per `-R` (its legs carry `bound`, one `accepted`
per connection, and this side's keepalive every 20 s), then one sealed
`nodeForward` naming each accepted connection — so the end-to-end
channel, the node's key on first use, the grant (`invoke`,
`3pm:forward` in the target's `callers`, `node:forward` in its `caps`)
and the audit are `-L`'s; the node reaches nothing of this host.

**Where the node listens is its own decision**, as ssh's `GatewayPorts
no`: the loopback, unless its `link.toml` has

```toml
[shell]
gateway_ports = true
```

— written by hand on the node, never by the backend, kept across a
re-link. Without it, `-R 0.0.0.0:…` (or `*:…`, or a LAN address) is
refused with that sentence.

A connection the command line does not take over within 10 s is closed
on the node; a local connect that fails is one line on stderr and the
node's connection is taken over and closed at once (its peer reads the
end now rather than after the 10 s). The listen ends — the port closed
and every connection still waiting with it, those already carried left
to their own end — on Ctrl-C or the shell's end (the command line sends
its `END`), a leg cut, a minute without a keepalive, or the agent's
stop. Code: `modules/agent/src/shell/listen.rs` (node),
`modules/agent/src/fleet/ssh/reverse.rs` (command line); e2e
`3pm-backend/tests/it/real_agent_reverse.rs` (3 connections opened on
the node, 20 MB each echoed by a server on the command line's side, the
same hash; the port closed within milliseconds of `SIGTERM`).

### `3pm fleet attach <node> <app>`: a `tty` app's terminal on a node

**What you see.**

```
$ 3pm fleet ssh web-1 -- 3pm start bash --tty --name work   # once
$ 3pm fleet attach web-1 work            # ~. after Enter: detached, the app runs on
3pm fleet attach: detached from work on web-1
$ 3pm fleet attach web-1 work --peek     # the screen as text, then out
$ 3pm fleet attach web-1 work --raw --read-only --instance 1
```

`3pm attach`, from another node: the terminal of an app started with
`tty = true` there — persistence for a remote session (a `bash` left
running in a `tty` app survives every detach). `--raw`, `--peek`,
`--read-only` and `--instance` mean what they mean for `3pm attach`;
`~.` after Enter detaches (in place of `Ctrl-Z` locally), and the app
runs on. The screen, its title, its bell and its alternate screen are
drawn as `3pm attach` draws them; the app exiting is a status line, not
the end — it may come back. Exit 0 when the session was detached (`~.`,
a peek served, the app deleted), 255 with one line on stderr otherwise
(the node refusing, a leg cut). Refusals are the daemon's: an unknown
app is `not_found`, one without a terminal `not_tty`. Same rules as
`fleet ssh`: the `invoke` grant, `3pm:shell` in the target's `callers`,
`node:shell` in its `caps` (what the app runs, a shell could run); an
agent that predates this answers "too old". **Audit**: the backend's
`open_requested` carries the app, the agent writes `open`, `attach`
(the app, its instance, the mode), each raw line typed (`line {alt:
true}`: the agent relays the screen without reading it, so a password
typed into the app is logged as typed) and `close`.

**How it works.** `nodeShell` with `attach` places an `attach` call
(its own op). The target's agent opens a **fresh** connection to its
daemon's `rpc.sock` (never one of its pooled ones: the connection
leaves NDJSON), sends the daemon's own `attach`, and once it is
answered `attached` relays the messages as they are between the legs
and that socket — the same `attach` codec the shell's legs carry. A leg
that ends, or an agent that stops, sends the daemon `detach`; the
daemon's `detached` is forwarded, then the leg's `END`; the call
settles `exited` with no status.

### `3pm fleet cp`: files to and from a node

**What you see.**

```
$ 3pm fleet cp ./build/app.tar.gz web-1:/srv/releases/
$ 3pm fleet cp web-1:/var/log/syslog .
$ 3pm fleet cp -r -p ./site web-1:/srv            # → /srv/site, modes and times kept
$ 3pm fleet cp -r web-1:~/data ./data-web-1       # a new directory here
$ 3pm fleet cp a.conf b.conf web-1:/etc/app/      # several sources: a directory
$ 3pm fleet cp web-1:/nope .
cat: /nope: No such file or directory
[3PM][ERROR] the copy of /nope failed on web-1 (exit 1)
```

As scp has it: exactly one side is `<node>:<path>` (an id, a hostname
or a unique prefix, as for `fleet ssh`; `<node>:` alone is its home,
`~/x` is under it); a `:` before any `/` makes a node, so a local name
with one is written `./a:b`. A destination that is a directory takes
the source under its own name, else the copy is that name. `-r` copies
directories (a file under `-r` is a file); without it a directory is
refused, `Is a directory (use -r)`. `-p` keeps each file's mode
(setuid bits included) and modification time, and a directory's;
without it the receiving side's umask applies to the source's mode and
the time is the copy's. **Never half a file under its name**: a file
goes up into `<dst>.3pm-tmp-<rand>` (owner-only, `umask 077`, until
its `chmod`: a secret is never readable by others mid-copy), then `chmod`, `touch -t` with
`-p`, `mv` over the destination; it comes down into a hidden temporary
file beside its target, renamed only once the node's command exited 0
(a replaced file keeps its mode). A local read error hangs the node's
command up instead of closing its stdin, so the temporary file is
removed there, never renamed. A file is never written over a
directory, either way. Inside a tree, symlinks stay links; a single
file is followed (`-p` too: `tar -h`), as `cat` follows it; on the way
down, a member that climbs out (`..`, or through a symlink) is never
written. A destination that is a link to a file is written through
(the link stays, as with scp). A tree lands as `tar` lands one: merged
into an existing directory of that name, not atomic, and the node's
`tar` gives the directories it unpacks (an existing one included) the
source's mode under the umask, even without `-p`. One session per source, one after
the other; a failing source does not stop the next (a session lost, a
refusal or a destination that is not a directory does). Exits 0 when
every source was copied, else 1: one line per failure, then `n of m
sources not copied`. The node's own stderr is shown as it comes; with
stderr a terminal, one progress line (bytes, rate) per source. Nothing
on stdout. **Audit**: each copy is a `fleet ssh -- <cmd>` session, and
its command line is what the audit shows — `sh -c '…' 3pm-cp
/etc/app/ a.conf 0644 '' 1 0 1f2e3d4c`: the node saw a file written
into `/etc/app/`. Same grants as `fleet ssh`.

**How it works.** No new op and no new route: each source is an
`exec` on pipes. What runs is one of four fixed POSIX scripts, `sh -c
'<script>' 3pm-cp <args>`, the paths only ever its arguments (a name
with a space, a quote or a `$` cannot become code, and a relative one
starting with `-` is made `./-…`, never an option; `sh` is named so a
fish or csh login shell does not read the script): a file up is `cat >`
the temporary file, a file down is `cat`, a tree up is `tar -x -o
-p|-m -f - -C <dst>`, a tree (or a file with `-p`) down is `tar -c` of
the directory's content (`./…`) or of the one file. Here the archive
is the `tar` crate, never the local `tar` binary: the same bytes from
macOS and Linux (bsdtar would add `._` members), no child process, and
an unpacking that checks every path; there, `tar` with only the flags
GNU tar, bsdtar and BusyBox agree on (`-o`: owners are not taken from
the archive). The bytes are the command's stdin up and its stdout
down, both lossless legs. 100 MB up and back on loopback (Linux, debug
builds): about 0.2 s up, 0.6 s down, the same sha256 at each end.

**Where it comes from.** `modules/agent/src/fleet/cp.rs` (the command
line, the scripts, one session per source), `fleet/cp/local.rs` (this
side: the stream in, the archive built and unpacked, temporary files),
`fleet/cp/progress.rs` (the progress line); the session is
`fleet/ssh/session.rs::run_command`. End to end:
`3pm-backend/tests/it/real_agent_cp.rs`.

### Between LAN neighbours: the file skips the backend

**What you see.** Nothing changes in an app: `io.call(node, action,
{file})` behaves exactly as before. When the two nodes sit on one LAN,
the file goes from the caller's agent to the target's directly — the
backend still places, audits and settles the call, and shows which way
the bytes went on it (`path: direct` or `relay`). A neighbour that
cannot be reached (a firewall, another network behind the same public
address) costs the reader at most 300 ms per address and the writer at
most one second, then the relay carries the file as it always did; that
pair is left alone for a minute. `direct = false` in the agent's
`link.toml` turns the listener off.

**Where it comes from.** `modules/agent/src/data/direct/`: `tls.rs`
(TLS 1.3, raw Ed25519 keys — `node.key`'s seed behind the fixed PKCS#8
header — the dialler pins the key its ticket named, the listener checks
the key per blob), `peers.rs` (who may read which blob here, whom to
dial for one, what the last attempt with each neighbour came to),
`meet.rs` (the rendezvous: reader first or writer first, one lock, one
copy), `listener.rs` (the LAN port: `GET /blobs/<blob>` from the named
key, `404` for anything else, `409` when late), `dial.rs` (the reader's
`GET`). The hooks: `data/handler.rs` (`put` → `Direct::put` before the
relay, `get` → a neighbour before the relay), the two points that mint
local tokens (`fleet_gate/place.rs` for the caller's input,
`commands.rs::open_blobs` for the target's), and `run_fleet.rs` (the
listener bound with the data socket, its port in `hello`, the reports
forwarded to the link). Only a call's input: its output is read after
the reply, which is after its writer finished, so nobody could meet it.
A session (`io.open`) between two nodes both set to `all` sends both of
its legs the same way — the leg's frames still go through the abandon
sink (`data/sink.rs`) first, and the writer reports every 5 s how far
it got (`blob_stat`): the call's `live` counters, and "direct · no
preview" on the dashboard, since the relay never sees a frame.

## A virtual Ethernet between nodes (`lan`)

**Installed by default, harmless alone.** `lan` is marked `auto`: every
3pm installs it by itself (docs/modules.md). Where it cannot work — not
Linux, not a root daemon, no `/dev/net/tun`, a `lan.toml` it cannot use,
a host not linked yet — it says why in one log line and stays up, idle,
`lan.status` answering `{"state":"idle","reason":…}`: never a crash
loop. Its peers never call it: an idle `lan` declares no `link` action, and
the discovery keeps only nodes whose `lan` declares it (read per node
with `fleet:read`, `3pm fleet inspect <node> lan --json`, kept per `lan`
process for 10 min: a steady discovery is 2 `3pm fleet` calls); a `lan`
of 0.50/0.51 that still refuses with `lan-idle` waits the longest
backoff. Every failed open backs off per peer, 5 s doubling to 10 min
with ±20 % jitter, reset only by a session that came up, and past 6
failed opens a minute all dialling pauses (`lan.status` `warnings`):
each call, even refused, holds backend blob quota for a minute. Where it can, it still makes **no device until a peer shows up**:
the TAP (and the namespace) is created when a discovery first finds
another `lan` node, and removed when none has been found for 10
minutes, so a server alone in its bucket gets no interface. Before
creating it, the module checks the network against the host's
addresses and main routes: a range already in use (a VPN's `10.0.0.0/8`)
is not shadowed — a default route split in `/1` halves (OpenVPN's
`def1`) or written `/0` is no range in use — one warning, no device, `lan.status` `reason`, looked
at again at each discovery. To turn it off on a host, or on a whole
fleet through provisioning, write in `<home>/config.toml`:

```toml
[modules]
off = ["lan"]
```

The next `3pm update` (or daemon the CLI spawns) uninstalls it — apps
deleted, copy removed — and no later one puts it back; take the name out
and it comes back. `3pm uninstall lan` is the per-host refusal
(`modules.off`), as for any auto module.

**What you see.** A fourth bundled module, `lan`: every node of a bucket
that runs it as root gets a TAP device `3pmlan0` with an address on
`10.77.0.0/16`, and reaches every other one on it as if on one switch —
ARP, broadcast, UDP, anything Ethernet — wherever they are. Nothing to
list: each node finds the bucket's other `lan` nodes by itself. Frames
ride the fleet's own stream sessions (`io.open`): agent to agent when
both nodes are LAN neighbours (the node's `direct` setting, `all` by
default; `files` keeps live legs on the relay), through the backend relay
otherwise. No VPN, no port to open: each node needs its `3pm link`
and two grants. Made for LAN games run under Wine on the tablets
(discovery by broadcast, then UDP between peers).

```console
$ 3pm module list                        # lan: installed by itself (auto)
$ ip -br addr show 3pmlan0               # 3pmlan0  UNKNOWN  10.77.143.12/16
$ 3pm trigger lan lan.status             # peers, sessions (dir, up, call, stats), MACs, warnings
$ ping 10.77.201.7                       # another node of the bucket
```

**Grants.** A node finds its peers with `3pm fleet nodes --json` and
`3pm fleet apps --json` (its own bearer, `getBucket`): it needs
**`fleet:list`**. It opens sessions with **`fleet:invoke`**. Both are
given by the bucket's owner with `setNodeAccess` (`POST
/api/buckets/{cluster}/nodes/{node}/access`, `{"scopes":["fleet:list",
"fleet:invoke"],"direct":"all"}` — the list replaces the old one). A node
without `fleet:list` logs the backend's refusal once, names both grants,
and keeps running with the peers it knew (none at first); `lan.status`
shows the error under `discovery`. A peer whose `callers` hold rules
must admit `{from:"*", from_app:"lan", app:"lan", action:"link"}`.

**Who talks to whom.** At start, 5 s later (the bucket lists this
node's own app only once it is up, and the others find it then), every
15 s while it found nobody, then every minute, and after a failed open
(at most every 10 s) the module reads the bucket: the nodes `online` with an `online` app `lan` of the
module `lan`, itself excluded. `allow` in `lan.toml` (formerly `peers`,
still read) narrows that to an allow-list (node ids or hostnames);
empty, the whole bucket. Identity is
the **node id**: the lower id dials (`io().open(peer, "lan", "link",
{from: <own id>, mac, address})`), the other serves the `link` stream
action. A session whose `from` the last discovery did not list starts
a discovery at once and waits for it (≤ 3 s) before it is judged again;
only a `from` still not a `lan` node of the bucket is refused, before
any metric series or log line of its own is made for it. If both dialled, each keeps the lower id's session. Hostnames
are only shown. A peer that left the listing is not dialled again, but a
live session to it stays: it is its own proof.

**Trust boundary: the bucket.** Only a member of the bucket holding
`fleet:invoke` can place a session (the backend checks), but the module
reads the caller's id from the session's params: until the daemon hands
actions a verified `caller` (protocol work to come), a bucket member can
claim another member's id. A bucket is one trust domain for `lan`.

**At most 14 peers dialled.** A daemon holds 16 sessions. Past 14 `lan`
nodes this side dials the 14 nearest (a symmetric distance of the two
ids' FNV hashes, stable whatever the order of the listing), serves the
others while it holds fewer than 15 sessions (the 16th is kept for a
renewal), warns in the log and in
`lan.status` (`warnings`), and dials anyway a peer that stayed 30 s
without a session — the nearest relation is not symmetric.

`lan.toml` (optional) is read from `/etc/3pm/lan.toml`, else
`$THREEPM_HOME/lan.toml`: `iface` (`3pmlan0`), `network` (`10.77.0.0/16`),
`address` (default: a host of `network` derived from the node id —
stable; two nodes that collide refuse each other's sessions, the
conflict is logged and listed once in `lan.status` `warnings`: set
`address` on one), `allow` (the allow-list; `peers` still read),
`broadcast_route` (default
`false`: adds `255.255.255.255/32 dev 3pmlan0`), `rate_kbit` (default
`4000`, what one peer may be sent, `0` = no limit), `expect_direct`
(default `false`), `netns` (absent: off) and `netns_address` (see
below). An unknown key is a warning. The MAC is locally
administered and derived from the node id too.

**A namespace for programs that will not use the lan.** Some bind to
the interface of the default route whatever address they are told (AoE3
under Wine: `0.0.0.0%wlp1s0:2299`), so a host never answers over the
bridge. With `netns = "3pmlan"` the module also makes a network
namespace holding only the lan, for such a program to run in:

| | |
|---|---|
| path | `/run/netns/<netns>` (`ip netns add`: what `ip netns exec <netns> …` or a launcher's `setns` enters) |
| interface | `3pmlan1`, a macvlan of `3pmlan0` in `bridge` mode, MTU 1500, up; `lo` up |
| address | `netns_address`, else derived from the node id with a salt (never the host's), on the lan's network, `broadcast +` |
| MAC | locally administered, derived from the node id (its own, not the TAP's) |
| routes | the lan's network on `3pmlan1`, `255.255.255.255/32 dev 3pmlan1`; **no default route** |

The host keeps `3pmlan0` and its address. The namespace's frames leave
through the TAP like the host's own, and the switch carries them the
same way; a frame for its MAC written into the TAP reaches it. A macvlan
in `bridge` mode does not talk to its lower device: the host's `3pmlan0`
and the namespace do not reach each other locally (each reaches every
peer). The namespace lives with the device: made only once a peer
shows up, deleted with it — 10 minutes after the last peer left, or
when the app stops (a program still inside keeps an empty one). One
left by a previous run is reused and its port remade, so a program
inside survives a `lan` restart only if it re-binds. `lan.status` shows `netns` (`name`, `path`,
`iface`, `address`, `mac` as read from the device). The expected use: a
tabloid frame launches its Wine game inside `/run/netns/3pmlan`. The
namespace's address is not part of the hello: a collision with a peer's
is not detected — set `netns_address` when in doubt.

**The MAC is read back.** udev (`MACAddressPolicy=persistent`) renamed
a fresh TAP's MAC just after boot on one tablet. The module waits for
udev (`udevadm settle -t 5`, when there is one), sets the MAC, reads it
back and sets it once more if it moved — the TAP and the namespace's
port alike; `lan.status` reports the MAC the device has, and the frames
a peer sends back with any of this node's own MACs are dropped.

**It needs a root daemon.** Module apps run as the daemon's account, and
a TAP is created by root (`CAP_NET_ADMIN`). Under a user daemon the app
stays idle and `3pm logs lan` says so once; on macOS it says Linux only.
A host not yet linked waits for its link and goes on by itself. The
device is not persistent: it goes with the app (and with the last peer,
10 minutes later), and every session goes with it — given up, its peer
announced down. A session from a peer while this node has no device
first waits (≤ 3 s) for a discovery that may make one; refused then, it
says why — as `lan-idle` when the device cannot be made here (the
network overlaps, the TAP will not open). A TAP held by another process
(`EBUSY`: the previous `lan`, still exiting) is not that: no `lan-idle`,
tried again at the next discovery. The mesh dials from the moment the
device is made and stops when it goes, whatever a discovery under way
concluded.

**Sessions.** Reopened with backoff (5 s doubling to 10 min, ±20 %,
back to 5 s once a session carried a frame) and renewed after 23 h, the renewal
respecting the backoff: the new session takes over at its first frame
back, then the old one is retired. The serving side does the same: a new
session from a peer whose served session works (up, a frame within the
last 50 s) waits beside it and takes over at its first frame — one
renewal at a time, a third replacing the second; one that never comes
up ends with no backoff, and an old one that ends first leaves the new
one in its place — so neither side announces the peer down (no
`lan.peer_down`/`lan.peer_up` pair) at a renewal. A served session
that is silent, or never came up, is replaced at once. A daemon reconnect does not end a
dialled session (the reply grace is set past a session's day). Frames
written before the other side's reader joined are lost, so the opener
sends a hello (12 bytes, `3pmlan/1` + its address) every 250 ms until a
frame comes back, and is not routed to before; the server answers every
hello; an address in a hello is checked, either direction, before the
session counts as up. Every 25 s, all sessions together, each one quiet
for 12.5 s gets its keepalive — a ping, echoed (a leg is cut after a
minute without a byte); an empty frame, an older peer's keepalive, is
still skipped. A session this side dialled that has been silent for 50 s, or
never came up within 5 s, gives way to a new one from the same peer (it
hard-reset). Anything shorter than an Ethernet header (14 bytes) is
control: skipped when unknown.

A learning switch sits between the TAP and the sessions: a frame from a
peer teaches it the source MAC (300 s ageing, 4096 entries) and goes to
the TAP only; a frame from the TAP goes to the peer its destination was
learned behind, or — broadcast, multicast, unknown — to every session
up, one buffer shared by all (`send_shared`). The mesh is full, so
nothing a peer sent is ever sent on: no loop.

**Direct or relay, and the rate limit.** Measured a9b4 ⇄ Mac on one LAN
(300 B × 1000/s): direct p50 ~11 ms, p99 ~100 ms, loss under 0.2 %;
through the relay p50 46 ms, p99 ~480 ms. Bulk through the relay
collapses (1400 B × 3000/s: 10 s RTT, 72 % loss), because the backlog
builds downstream of the SDK's sink where this side cannot see it. So
what goes to each peer passes a token bucket (`rate_kbit`, 4 Mbit/s by
default, a 100 ms burst of at least 16 KiB; no bucket at all for `0`):
the excess is dropped and counted (`lan.dropped`, and `limited` per
session in `lan.status`), never queued. The module cannot see which
path a session took: at each `peer_up` it warns, unless `expect_direct =
true`, naming the call id — the call's `path` says direct or relay. The
`lan.peer_up` event carries `call_id` too. Known gap: with
`expect_direct = true`, nothing reports a session that fell back to the
relay; look the call up (`getCall`, its `path`).

**The latency guard: a peer's round trip cuts its rate.** On the relay
path a slow uplink keeps frames *in flight*, where no socket option
reaches: between a tablet on a 5G hotspot and one on Wi-Fi (2026-10-05),
ping over the lan climbed 0.18 s → 0.9 s → 2.8 s → 34 s under an AoE3
game and some ssh, until the uplink stalled. So each session measures
its round trip and sends less while it is long:

| | |
|---|---|
| probe | a ping (`3pmp?` + an 8-byte stamp, 13 bytes) the peer echoes as a pong (`3pmp!`): measured on this side's clock alone, the two hosts' clocks never compared |
| when | every second while the session carried an Ethernet frame within the last 5 s; idle, the 25 s keepalive **is** the ping — an idle node still wakes once per keepalive, and every session, up or not, still gets a frame at least that often |
| round trip | the last pong's, or the age of the oldest ping still out when longer (a stalled uplink answers nothing); nothing until a first pong — a peer older than the guard skips both frames and is never throttled |
| over 1 s | the peer's `rate_kbit` is halved, at most once per round trip, down to 64 kbit/s; logged at each cut |
| under 400 ms | it grows back by a tenth of `rate_kbit` per ping (≈ 10 s from the floor) |
| `rate_kbit = 0` | measured and shown, never acted on |

`lan.status` shows per session `rtt_ms` (smoothed), `rtt_last_ms` and
`rate_kbit` (in force); the histogram `lan.rtt` (ms, label `peer`)
records every sample. An idle keepalive's ping measures but never cuts.
Below the module, the agent's own `live` legs keep their queues shallow
(see "Fleet RPC": a `live` leg's connection), so what the rate lets
through is not then held for seconds in a socket.

**Closing without breaking the reply.** A session given up (replaced,
renewed, a duplicate, a conflict) is ended by a reaper thread, read
until the other side's `END` by its own reader, and closed 2 s later: in
direct mode a `close` right after `end` makes the peer's reply fail
(502, "the reader went away").

Metrics `lan.frames` and `lan.bytes` (Ethernet only — hellos, pings
and pongs are not counted; labels `peer` = node id, `dir` = `in`/`out`),
`lan.dropped` (no session up, over the rate, a refused send, the TAP
refusing a write), `lan.peers` (gauge), `lan.rtt` (histogram); events `lan.peer_up` and
`lan.peer_down` (with the reason), once per peer whatever its sessions
do underneath.

**How it works.** `modules/lan/`, a standalone workspace on `threepm-io`
(streams, metrics, events, actions), `libc` for the one `ioctl`,
`rustix`, `serde` + `toml`: `src/main.rs` (the app: the two actions
declared at once, then the setup on its own thread, or idle with the
reason), `src/config.rs` (the file, the defaults, the
derived address and MAC), `src/discovery.rs` (the two `fleet` documents
joined, the `3pm` binary found: `$THREEPM_EXE`, the daemon's own
`/proc/<ppid>/exe`, the shipped layouts, `PATH`), `src/tap.rs`
(`/dev/net/tun`, `TUNSETIFF` with `IFF_TAP | IFF_NO_PI`; the only file
allowed `unsafe`), `src/mesh.rs` (one slot per peer:
current and pending session, who dials, cap, backoff, renewal,
duplicates, conflicts, idle peers, and the switch — pure, one lock),
`src/ip.rs` (the `ip` commands: the TAP's MAC read back, address,
broadcast route, the namespace and its macvlan), `src/switch.rs` (the
learning table, pure), `src/bucket.rs` (the token
bucket, pure), `src/guard.rs` (the latency guard, pure),
`src/wire.rs` (hello, ping and pong, an older peer's empty keepalive),
`src/link.rs` (one session: bucket, guard, series), `src/links.rs` (the
data path), `src/status.rs` (`lan.status`), `src/sync.rs` (locks that
survive a panic), `src/device.rs` (the device made with the first peer and removed 10 min after the last, the network checked against the host's routes first, the TAP reader polling so it can stop), `src/sched.rs` (a scheduler asleep until the next deadline
— an idle node wakes once per keepalive —, the reaper, the discovery). Unit tests per file on any OS, including every
interleaving of a crossed dial; `tests/e2e.rs` against a real daemon
(the refusal under a user daemon, the binary run by hand), and two
root tests (`#[ignore]`, run with `sudo unshare -mn`): the TAP carries a
frame, and a namespace's limited broadcast leaves on the TAP while a
frame written into the TAP for its MAC reaches a socket inside.

## Bench fleet

**What you see.** One command fills the table with a realistic mix so
every column has something to show: 4 node apps (an HTTP server, a CPU
burner, a memory ballooner, one that crashes every 15 s), 2 python apps
(an HTTP server, a JSON-ticking worker) and 2 shell loops (`ls -la /`,
a poor man's `watch`). HTTP apps get `PORT=0` so the port column shows
the OS-assigned port; the crasher has `min_uptime 1000` / `restart_delay
500` so its restart counter climbs without ever tripping the crash-loop
cutoff. Rerunning restarts the same apps in place; `--delete` (alias
`--clean`) removes them, tolerating a fleet that is already half gone.
A missing `node` or `python3` skips those apps with a warning instead of
failing the run.

**How to get it.**

```console
$ 3pm bench                       # writes ~/.3pm/bench/*.{js,py,sh}, starts 8 apps
$ 3pm bench --dir ./demo          # scripts somewhere else
$ 3pm logs bench-py-worker        # {"tick": 42, "pid": …}
$ 3pm bench --json                # same Affected contract as `start --json`
$ 3pm bench --delete
```

**Where it comes from.** `commands/bench/scripts.rs` is the single
source of truth (names, files, bodies); `bench/fleet.rs` turns it into
`StartSpec`s through `SpecPatch::apply` and `start::start_specs` runs
the same start loop as `3pm start`. Every app is in the `bench`
namespace. Covered by unit tests on the table/specs and
`tests/e2e_bench.rs` (real node/python/bash, restart of the crasher,
delete idempotence).