# Features

User-facing features of the fleet backend, one section each: what you
see, how to get it, and where it comes from. Internals live in
OV `3pm-backend-docs-archived/backend`; the wire contract with the agent is in
3pm's `docs/backend-agent/`.

## Buckets and the pasted line

**What you see.** A bucket is a fleet: a name and a `3pmc_…` id. `/`
lists and creates them; `/b/<id>` mints an enrollment token and prints
the one line to paste on a host:

```
$ 3pm link 3pmc_<cluster> 3pmk_<token> ws://127.0.0.1:9616
```

and, for a host without 3pm, the same link through the installer
(`install_line` beside `paste_line` in `createToken` and `bucketToken`;
under *Link a host* the bucket page shows it with the permanent token,
and the settings page too):

```
$ curl -fsSL https://3pm.edgecr4ft.net/install.sh | sh -s -- link 3pmc_<cluster> 3pmk_<token> wss://…
```

A token is short-lived and counted: `{ttl_s: 3600, uses: 1}` by
default, up to 30 days and 10 000 uses. The secret is shown once and
never stored — only its SHA-256 — and `DELETE …/tokens/{id}` revokes
it. `pre_approved` is decided at creation (`uses == 1 && ttl_s <=
3600`): a single-use, short-lived token admits its host at once, a
reusable or longer-lived one produces a *pending* node an operator has
to admit. Minting refuses a local advertise address outside `--dev`
(409 `ERR_ADVERTISE_LOCAL`), so a pasted line never points at
`127.0.0.1` by accident. The bucket page shows the address the line
ends with (`ws://…` or `wss://…`).

**Behind a TLS proxy.** The backend speaks clear text; nginx (or any
proxy) terminates TLS in front of it. `--advertise` takes a URL whose
scheme wins, so the pasted line says what agents must dial:

```
$ 3pm-backend --listen 127.0.0.1:9616 --advertise wss://fleet.example.com --trust-proxy 127.0.0.1
$ 3pm link 3pmc_<cluster> 3pmk_<token> wss://fleet.example.com:443
```

The agent checks the proxy's certificate against the public roots,
nothing to configure. A bare `host:port` keeps the old rule — `ws://`,
since the backend serves no TLS of its own — and outside `--dev` the
backend warns at startup that agents refuse clear text to a remote
host without `--insecure`. An invalid URL stops the start.
`--trust-proxy <ip>[,<ip>…]` believes those peers' `X-Forwarded-For`
(the rightmost hop that is not a trusted proxy), so the per-address
budgets (enrollment, link handshakes) and the link logs name each agent
rather than the proxy; the proxy must set it (`proxy_set_header
X-Forwarded-For $proxy_add_x_forwarded_for;`). Without the flag the
header is ignored. **Behind a reverse proxy the flag is not optional:**
without it every agent is the proxy's address, and the whole fleet
shares one address's budget — 64 pending link handshakes, 16 pending
enrollments.

**Where it comes from.** `crypto/tokens.rs`, `db/enroll.rs` (one
transaction: consume the use and upsert the node — no burnt use, and
two hosts racing for the last one never both win it), `http/paste.rs`,
`advertise.rs`, `http/client_ip.rs`. `POST /api/v1/enroll`
(`http/api_enroll.rs`) is the agent's side: a token, a public key and a
signature over both in, a node id out. A host is
`(cluster_id, node_id, Ed25519 key)` on its row — there is no
certificate and no CA, so nothing expires and nothing is re-issued: the
row is read again at every handshake, and an approval or a revoke lands
on the next connection.

## Admitting and revoking hosts

**What you see.** Every node of a bucket is a row: its id, host, OS,
agent version, state, last seen and receive rate.
`Approve` admits a pending host, `Revoke` cuts it — the live session is
closed 4403 `ERR_NODE_REVOKED` and the agent parks itself and probes
hourly. Both are idempotent, and approving again re-admits a revoked
node with the key already on record.

`Delete` is offered on every node that is not `online` — offline,
stale, pending or revoked; a node that is talking to the backend shows
no such button, because deleting it cuts a link its agent cannot mend
by itself (it would retry forever under an id nobody knows). The route
itself refuses nothing: a script may still delete an online node.
`Delete` forgets a node, after a confirmation: its row, its calls and
its artifacts (the files too) go, its session is closed 4401
`ERR_NODE_UNKNOWN` and its bearer dropped. It is the opposite of a
revoke in one respect: nothing is kept against the key, so the host may
enroll again — as a new node, with a new id. Use it for a machine that
is gone, or a test enrollment; use `Revoke` to keep a host out. Every
open page of the bucket redraws without the node (the feed sends
`reset`, the page asks for a fresh `init`). The metrics and events the
store already holds for it stay until the store's retention takes them.

A host that comes back under a new key — reinstalled from scratch, its
`node.key` gone with the disk — enrolls as a new node, and its old row
is **deleted on its own** when the new one becomes active (enrolled
with a pre-approved token or approval off, approved, or admitted by
turning approval off): every older row of the bucket with the same
hostname (case ignored) that has no live session and is `offline` goes,
exactly as `Delete` would. A row someone is linked through, a `stale`
one (it may be reconnecting) and a revoked one are kept; a pending
node replaces nothing until it is admitted; of several twins admitted
at once, the newest stays. The hostname is the agent's word, not a
proof: two homes of one machine linked to the same bucket share it, and
the offline one is deleted when the other enrolls.

A host that knocks while pending or revoked leaves `last_auth_error`
on its row (`ERR_NODE_PENDING`, `ERR_NODE_REVOKED`, with a timestamp),
so the page shows who is waiting instead of failing silently — one
click admits it.

**Where it comes from.** `http/api_nodes.rs` and the hub
(`hub/registry.rs`): the database row changes, the hub's view is
replaced, and a `node` event goes out on the feed. A delete is
`db::nodes::delete` (the row and the paths of its files, one
transaction), then `Hub::forget`, whose `reset` `http/sse.rs` turns into
the end of every stream. The automatic delete is `http/supersede.rs`
(`is_replaced_by`, the rule; `sweep`, called by enrollment, `approveNode`
and `setBucketSettings` before the node is published), through
`api_nodes::forget`, the body `deleteNode` shares.

### Bucket settings: approval, on or off

**What you see.** `settings`, in the header of a bucket's page, opens
`/b/{cluster_id}/settings`: one box, *Require approval for new hosts*.
On (the default), a host linked with a reusable token waits for
`Approve`; a one-use token of an hour or less is active at once, as
always. Off, every host that links with a valid token of the bucket is
active at once — what enrolling twenty tablets with one reusable token
wants — and the token is then the only gate. Turning it off approves
the hosts that were already waiting (the page says how many before you
do); turning it back on leaves them in. The bucket page follows the
setting: the note next to *reusable* and the line under a new token say
whether the host will wait.

**Calls between nodes** is a second box: *Require nodes to specify
access rights*, ticked by default. Ticked, a node places calls only once
you tick *Place calls* in its Access panel. Unticked, every active node
may run any action of any other and list the bucket's nodes to find
them by hostname, and read each other — as if each held `fleet:invoke`,
`fleet:list` and `fleet:read` —
with nothing to grant node by node: a tablet that enrolls tomorrow
talks to the others at once. It opens *Read the fleet* too (hosts,
metrics, the catalog of actions: a caller checks what a peer declares
before calling it), it writes nothing on the nodes (their
Access panels still show what you granted, and ticking the box again
takes back exactly what it gave), and a node's own *Who may call this
node* rules still apply. Both ways are felt on a node's next request.

Below it, the bucket's **permanent enrollment token**: one
`3pm link …` line with a `copy` button, the same every time you open
the page. It never expires, enrolls any number of hosts, and cannot be
renewed or revoked — paste it on a host, or give it to the script that
provisions your machines. It goes through the same approval as any
reusable token: with the box ticked its hosts wait for `Approve`, with
the box unticked they are active at once and the token is the bucket's
only gate, so keep it like a password. The one-use and reusable tokens
of the bucket page still exist next to it.

Over the API: `bucketToken(clusterId)` answers `{ token_id, token,
paste_line, created_at }`; `setBucketSettings(clusterId, { approval: false })`
answers the `BucketView`, which carries `approval` everywhere a bucket
is read. A node's bearer never sees it.

**Where it comes from.** `http/api_settings.rs` over
`db::buckets::set_approval` (the setting and the queue it empties in
one transaction), read at enrollment in `http/api_enroll.rs`; `require_access_rights`
is read by `db::effective_scopes`, applied where a node's scopes are read: the
node gate (`auth/node_gate.rs`, on the bucket row of that request) and
the `call_result` push (`calls/result.rs`, which reads the setting
fresh); the
permanent token is `db::tokens::permanent` (created once, behind a
partial unique index; `SQL_CONSUME` takes a use from it without
spending one, `SQL_REVOKE` leaves it out), the only token whose secret
is stored; the page
is `assets/settings.html` + `assets/pages/settings.js`.

## The live dashboard

**What you see.** `/b/<id>` opens one SSE feed and stays current
without polling: presence goes `online` → `stale` (3 s without a
frame) → `offline` (30 s), each node's process table refreshes at 1 Hz
with CPU, memory (of the app's whole process tree, as 3pm reports
them), restarts and uptime, `+N` after an app with other processes
and a **Process trees** panel that draws them (its own section
below), the last raw `telemetry` frame
sits under a `<details>`, and lifecycle events (`process`,
`daemon_shutdown`) scroll in a log. Each node also shows what it costs:
`rx_bps` (bytes received over its link, averaged over the last 5
seconds) and the total since the backend started.

Next to each card's hostname, the node's addresses: `public <ip>`,
where its link comes from as this backend sees it (behind a
`--trust-proxy` proxy, the forwarded client; kept while the node is
offline, forgotten when the backend restarts), and `private <ip>, …`,
the private addresses of its interfaces (RFC 1918, CGNAT `100.64/10`,
IPv6 ULA; loopback, link-local and container or VM bridges — `docker*`,
`br-*`, `veth*`, `virbr*`… — left out), from its last complete host
snapshot. The node view carries them as `peer_ip` and `private_ips`.

At the bottom of the page, **Metric keys** lists every key the bucket
holds — the store's own telemetry and the application metrics seen
here over the last thirty days — with its kind, unit, default time
function, the labels a query may filter or group on, and where the key
comes from (`static` or `app`). Applications come first, the count
next to the title says how many of the keys are theirs, and the list
is asked again every 30 s while the tab is visible, so an app's first
sample shows up on its own.

**Live JSON.** Every row of the node table has a **Live JSON** button,
and every app of a card a `JSON` one: a dialog shows the node's
documents exactly as the API serves them, refreshed every second while
it is open — the host now (`GET …/nodes/{node}/host`, `nodeHost`, what
the node button opens on), the last telemetry frame as the feed sent
it (no request), `nodeAppMetrics` (custom metrics),
`nodeProcessTrees`, `nodeActions`, or one app composed from the three
(`{ process, app_metrics, tree }`, what the per-app button opens on).
A second selector narrows any document to one app. The document is a
collapsible tree (json-formatter-js, vendored under `assets/vendor/`):
the top-level keys open, everything under them folded behind a `►`,
a preview of what a fold holds on hover, keys, strings, numbers,
booleans and nulls each in their colour; what the reader unfolds stays
unfolded across the one-second repaints (every row carries its path,
and the fresh tree is unfolded along the same paths). A **search**
box cuts the document down to every key or scalar value holding the
text (case-insensitive): each hit with its parent whole — every
attribute of the parent, as it is — under the chain of keys that
leads there, everything unfolded down to the hit and the hit's key on
an amber ground; an array on the way shows as its indices, so they
stay true. The count says how many hits; the search follows the live
document. **text** shows
the raw JSON instead, **stop** freezes the document to study it (the
title says when it was frozen; changing the source or the app loads
that document once and stays frozen; **resume** follows it again),
`copy` puts the text on the clipboard whatever the view, and the
reader's scroll survives a repaint.

The frame and the host differ on purpose. To save 5 KiB a second per
node and viewer, the feed carries the host's identity — hostname, OS,
kernel, arch, boot time, CPU brand and counts, memory and swap totals,
the interfaces' names, MACs and IPs, the mounts, the GPUs' models —
only on a node's first frame, when it changes, and once a minute
(`host_full`); every other frame is a tick whose identity fields read
blank (`host_tick`). `nodeHost` answers the host whole — the one the
backend rebuilds from the node's state (see "The host, received as a
diff"), `identity` always `patched`. 404 before the node's first frame.

**Events and Metric keys, folded and filtered.** Both lists fold
behind their title (closed by default, the count next to it: `(200
events)`, `(218 keys, 3 from applications)`), with a filter box
inside: words, every one of which must appear in the line — `api
error` keeps the error events of `api`; `counter iface` the counters
carrying the `iface` label (a key's id, kind, unit, labels, source and
description are all matched). The count says what is shown of the
whole (`(12 of 200 events)`).

**SDK guide.** At the bottom of the page (**SDK guide** in the
header), six folds: start an app with the SDK, custom metrics, custom
events, custom actions, profiling, and the `3pm` commands to know
(`start`, `ls --tree`, `show`, `events -f app` / `-f proc_action`,
`trigger`, `profile`, `link`). The examples are runnable as they are:
the `curl` and JavaScript-client snippets name this bucket, its first
online node and that node's first app, and follow the feed when they
appear (`assets/pages/guide.js`, whose escaping and filling
`tests/it/dashboard_render.rs` holds). Every example is coloured by
highlight.js (vendored under `assets/vendor/hljs/`: core and the
`bash`, `javascript` and `json` grammars, loaded after the guide is
painted — `assets/pages/highlight.js`), one language per block.

A page that draws a few hosts asks the feed for less
(`?history=0&match=<hostname part>&view=live`, `http/feed_query.rs`):
only those hosts, no per-process history in `init`, and each frame cut
to the gauges a live chart draws — a tenth of a frame, and the other
nodes never leave the server. The generated clients send `history=0`
by default, the dashboard at `/b/<id>` included (it never read the
history); it asks for nothing else.

The feed replays what a reconnecting client missed (`Last-Event-ID`,
the last 4 096 events per bucket); a client that fell further behind is
told to `reset` and starts over from a fresh snapshot. Presence
timings are tunable (`--stale-after-ms`, `--offline-after-ms`,
`--history-points`).

**Where it comes from.** `http/sse.rs` over `hub/` (registry,
presence, per-process rings, per-bucket event log); `assets/` for the
pages — `app.js` discovers the route table (`GET /api/v1/routes`) and
hands the generated client to `pages/index.js` / `pages/bucket.js`,
which know route names and never a path. The key list is
`metricsCatalog` rendered by `catalogTable` (`pages/bucket.js`), whose
escaping `tests/it/dashboard_render.rs` holds.

## The telemetry, persisted

**What you see.** Start the backend with `--metrics-url
http://127.0.0.1:8428` (VictoriaMetrics) and `--events-url
http://127.0.0.1:9428` (VictoriaLogs) and every linked node's
telemetry outlives the process: one sample per series every 5 s
(`--store-interval-ms`) — `threepm_process_cpu_ratio` and
`threepm_process_memory_rss_bytes` (the app's whole process tree: the
spawned pid and every descendant, RSS summed — an upper bound when
members share pages — as 3pm reports them; the spawned pid alone from
a 3pm predating trees), `threepm_process_procs` and, per group of
the tree's members sharing an executable name, `threepm_subprocess_*`
(label `proc`, never a `pid`),
`threepm_process_up`,
`threepm_process_restarts_total`, `threepm_host_cpu_ratio`,
`threepm_host_memory_used_bytes`, `threepm_host_network_receive_bytes_total`,
`threepm_host_disk_used_bytes{mount}`, `threepm_node_up`, and what 3pm
itself costs the node as its host agent measured it —
`threepm_daemon_cpu_ratio`, `threepm_daemon_memory_rss_bytes`,
`threepm_host_agent_memory_rss_bytes`, `threepm_daemon_apps_online`,
`threepm_daemon_restarts_total`, `threepm_daemon_handoffs_total`…
(catalog ids `daemon.*`, `host_agent.*`) — labelled
`cluster` (the bucket), `node`, and per process `app`, `instance`,
`ns`. Process lifecycle events (`start`, `online`, `exit` with code and
signal, `restart`, `errored`…) land in the logs engine, one line each.
`GET /api/v1/info` → `store` says whether the engines are configured
and counts what was `sent`, `dropped`, `retries` and `errors`, with the
kind of the last failure. Without the flags nothing is persisted and
nothing changes.

Every number `3pm hostmetrics` collects is kept, not a chosen few: the
CPU per core and by state, the kernel counters, the memory
decomposition and the page-fault counters, PSI, processes and TCP
connections by state, the socket and conntrack counters, the daemon's
cgroup, then one series set per network interface, block device,
temperature sensor, GPU, RAID array and battery — with `core`, `iface`,
`blockdev`, `sensor`, `gpu`, `array` and `bat` as query labels — plus
when each sampling tier last landed, so a tier that stopped is visible.
A state is always part of the metric's name, never a `state` label that
would collide with an application's own. `GET …/metrics/catalog` lists
the 292 lines and the dashboard's **Metric keys** table shows them.

A node's **top processes** (`top_processes`: the heaviest processes of
the host by CPU, resident memory and network rate, at most 30 rows) are
live state, not series: `nodeHost` and the feed's host carry the table,
nothing is stored per process, and `system.sample.top` dates it.

A laptop's or a UPS-backed box's packs are twelve of those lines
(`system.battery.*`, one set per pack under `bat`): charge, energy,
what the pack holds today and held new, the health those two make, the
draw, the voltage, the cycle count, the temperature, whether the bay is
filled, and two codes — `system.battery.state` for what sysfs `status`
says (nominal: 0 unknown, 1 charging, 2 discharging, 3 not charging, 4
full, 5 a status this backend has no name for), `system.battery.level`
for the coarse level a gauge with no numeric capacity falls back to
(ordinal, 1 critical … 5 full). The direction the energy flows is the
state, never the sign of the draw, which is a magnitude. `system.on_ac`
is the host's own answer, not a pack's, so it carries no `bat`. Every
reading is optional and an absent one stays absent: no `status` file
writes no state, no sensor writes no temperature, and a full pack
drawing nothing writes a real 0.

**Extended Monitoring** (a 3pm node's `[hostmetrics] extended`, on by
default) adds the kernel's own counters on Linux, ~1 000 to 1 600
series per node, stored **one series per family**:
`threepm_hostx_<family>_total` for a counter family,
`threepm_hostx_<family>` for a gauge one — `vmstat` and
`vmstat_gauge`, `netstat`, `snmp`, `diskstats`, `cpuidle`, `hwmon`,
`thermal`… — with `field` the kernel's own spelling of the reading and
the row under the label that already means it (`core`, `iface`,
`blockdev`, `array`) or a generic `row` (a PSI resource, a protocol, a
chip, a zone, a C-state). A chart names the family and filters the
field: `{"metric":"host.ext.vmstat","filters":[{"label":"field","op":"eq","value":"pgsteal_kswapd"}]}`;
an unfiltered query of the line shows the fields that family holds,
while `metricLabelValues` on `field` lists every field of every family
in the bucket (and an application's own `field` label, if it has one);
the inventory lists neither `field` nor `row`. The live feed carries
none of it — ~40 KB a node at every tick, drawn by no dashboard — and
`nodeHost` serves the node's current `extended` map whole. A slow-tier
counter (`host.ext.cpuidle`, read every 5 minutes) rates over at least
10 minutes. The catalog has one hand-written line per family
(`host.ext.*`, 42 with the three eBPF short-lived-process families),
each saying where the fields'
meaning is written — the man page, the kernel doc, the RFC. `field` and
`row` are custom labels, like `bat`: an application's own `field` label
still works. Per group of a process tree come sixteen more
(`subprocess.faults.*`, `subprocess.context_switches.*`,
`subprocess.disk.*`, `subprocess.io.*`, `subprocess.syscalls.*`,
`subprocess.threads`, `subprocess.fds`, `subprocess.fds.usage_max`,
`subprocess.memory.{pss,footprint,swap}`), each present only where the
platform gives it — a gauge summed only when every member of the
group gives it, a counter accumulated by the backend (see "Process
trees") — and per app `process.memory.pss`, the tree's PSS —
what it really costs where `process.memory.rss` double-counts shared
pages, absent unless every member's could be read.

Network traffic per app, from a 3pm node's own meter (not Extended
Monitoring: on wherever the node has a source — Linux, macOS):
`process.net.received` / `process.net.sent` per app (its whole process
tree, each socket once; `threepm_process_net_{received,sent}_bytes_total`,
read with `rate`), and `subprocess.net.received` / `.sent` per group,
accumulated like the other group counters. The counter restarts with
the app and with a daemon update — a counter reset `rate` handles. On
Linux an outbound connection opened and closed between two of the
daemon's 5 s windows is not in it;
on macOS loopback is not. The live figure is on the record
(`RedactedRecord.net`: `r`/`t` per second, `rb`/`tb` totals) and on each
tree line (`TreeProc.net`).

**Where it comes from.** `src/store/`: the writer pulls the hub's last
frame per node on its own tick (nothing per frame, the live feed is
untouched), encodes off the runtime with the label blocks of every
series rendered once and kept (`prefix.rs`), gzips the bodies side by
side, posts Prometheus text and NDJSON with a short retry ladder; the schema and the sizing are in
OV `3pm-backend-docs-archived/backend` → Metrics store and
`3pm-metrics-plan.md` in `~/3pm-rd-doc-2026-09-09.tgz`.
The workspace `ecosystem.toml` starts the engine on loopback with a
dedup interval equal to the store interval.

## The node's whole state, received as diffs

**What you see.** Nothing on the pages, the feed or the API says how a
node's state travels. On the link there is one mechanism: the backend
asks its agents (`welcome.pace`, from `--state-every-s` /
`THREEPM_BACKEND_STATE_EVERY_S`, default `1`, `1` at least — `0` is
refused at start) for **one `state` frame a second** carrying only what
moved in the node — the host, the process table, each instance's
application metrics, each process tree. Measured on real traces,
everything included: 357 B/s for a dev Mac with 11 apps, 463 B/s for a
Linux server with 8. It is the only way in: the link is at **version
4**, and an agent that predates it is refused at the handshake with
`ERR_LINK_VERSION` (close 4426) — or at the upgrade with 400, if it
offers the older sub-protocol — until the node runs `3pm update`; the
frames of a stream this build no longer knows, should one ever come,
are unknown tags — skipped, never an error.

`GET /api/v1/info` → `link` is what says the fleet and this backend
agree on the wire: `frames_invalid` counts agent frames whose tag this
build knows and whose **body** it could not read (a field one side
spells differently: skipped, the session kept), `crc_full_mismatch` the
sessions in which a full state section failed its checksum. Both `0`
is the normal reading; either one moving names a bug in one of the two
builds, and the backend's log says which frame or which section. Only
an **established** session is counted or logged that way: before its
`hello` a peer has proved nothing — the `Authorization` header is a
claim over two public ids — so what it sends is skipped quietly, eight
frames at most, and the ninth closes the connection.

**Where it comes from.** `src/hub/node_state.rs`: `NodeState`, one per
node, kept across a session replacement, applies every patch of a frame
on its own with `threepm_link::NodeReader` — **the** patch reader, the
only one — a gap leaves its section untouched and the session sends
`state_resync {secs}` (at most once a second, the one `ws/resync.rs`
coalescer of a session), a **full** patch with a wrong checksum is kept
and turns the checks off until the node's next session (one `error!`
line naming the section, and `info.link.crc_full_mismatch`); a
section past 4 MiB, or the one that takes the node's sections past
32 MiB together (both tracked as bounds, measured only past them), is
dropped and counted — from its typed holder too, the host included —
and comes back whole at the agent's next patch of it; a 405th section
is refused and counted, with one warning a session, until one held
goes. A frame of **another incarnation** of the agent (`gen`: it
restarted) forgets every section that frame did not bring again: a new
agent owes all it has in full and never says `gone` for what went away
while it was down, so nothing of its former self is served, or piles
up restart after restart. What the reader refused so far is the
`dropped` of the feed's telemetry and the store's
`threepm_telemetry_dropped_total`. The process table is decoded when
its section moves, and read at every frame's clock.
`hub/registry_state.rs` hands what moved,
typed, to the holders the consumers read: the host section to
`NodeHost` (the view's addresses and accounts following it), an
instance's metrics to `NodeApps::set`, a tree to `NodeTrees::set`.
Both refuse what is over the daemon's caps or off the label grammar —
counted, and the state held for the instance dropped rather than left
behind its section — and an instance goes with its section: `gone`, or
the agent restarted without it, at once. Its row of the process table
is a safety net only (absent from two frames in a row: forgotten, its
section with it, so the next patch is a gap and not a patch onto what
nobody serves) — and never on the table of a **daemon that is down**,
which lists nothing: the agent says no `gone` then, and an instance
that never moves (one that only declares actions) is still there when
the daemon answers again. A node **revoked** keeps its view and none
of what it sent. A section that does not decode,
or names another instance than its own, counts as gone. The trees held
are current as of every frame, and their clock says so. Then
`Hub::state` publishes, for **every** frame, the `telemetry` the feed
always carried (`hub/feed_snapshot.rs`, the backend's own type now) —
numbered by the backend, the table as of the frame's clock (a process's
uptime is derived from it), the daemon, the host by the feed's rule —
so the feed, the rings and the store's freshness work as they did.
Tests: `hub::node_state` (typed hand-back, gaps, gone, the bounds and
the node's budget, the checksum rule, an agent restarted, the table
read at every clock, the refusal count), `ws::resync` (a section that
came in full is no longer asked), `hub::apps` and `hub::trees` (caps, grammar, the two
ways out), `hub::registry_tests`, `tests/it/state_e2e.rs` (the feed's
shape, `nodeHost`, the view's address, `nodeAppMetrics`, the trees, the
store's lines, a heartbeat, `state_resync`, gone, an older stream's
frame skipped), `dashboard_e2e` (link 1 refused) and every
`real_agent_*` test (a real daemon and agent).

## The host, received as a diff

**What you see.** The bucket page, the tablets, `nodeHost`, the charts
read a host that is whole and current, and a linked node costs little
on the link: the host is the `host` section of the node's state, so
only what moved in it travels. Measured on real traces against the
whole snapshot in every frame: 359 → 194 B/s for a dev Mac, 576 → 403
B/s for a Linux server, and static sections cost nothing after the
first frame. Numbers arrive at **3 significant digits**. `nodeHost`
answers the rebuilt host, identity included; its `identity` field is
always `"patched"` (it used to say `full`, `merged` or `tick` for a
host read off `telemetry` frames — that source is gone, the field stays
for the pages that read it), and a node that sends no host (its host
agent is down) is a 404 that says so.

**Where it comes from.** The section is rebuilt and bounded by
`NodeState` like any other (above). `src/hub/host.rs`: `NodeHost`, kept
across a session replacement, holds what the section stands for — the
typed `HostMetrics`, decoded once per revision when the section moves,
shared as a pointer — and the agent's clock on that frame. It is the
**only** host the backend reads: the store writes it as both the
figures and the identity (every host family, per interface and per
mount included); the view's private addresses, account and sessions
follow it (a patch touching them announces the node, a host gone leaves
the last answer); `nodeHost` serves it. The `telemetry` the feed
publishes gets it put in by the feed's rule — `host_full` on the first
frame, on an identity change and every minute, a stripped `host_tick`
otherwise (`hub/feed_snapshot.rs`: `FullPolicy`, `strip_static`) — so
dashboards and tablets see the shape they always did, at 5 KiB less a
second per node and viewer. Moving the feed itself to diffs is a later
piece of work. Tests: `hub::host` (the decode, a section that is not a
host, the feed's rule), `hub::feed_snapshot`, `tests/it/host_e2e.rs`
(the route whole while the feed's tick is stripped, the view, the clock,
the 404s), `tests/it/state_e2e.rs` and `real_agent_host` (a real daemon
and agent: the rebuilt host equals `3pm hostmetrics --json` at three
digits on everything that does not move within a second).

## Link traffic: what each node's link costs, by kind

**What you see.** A node card's "Link traffic" panel: per direction
(↑ agent → backend, ↓) and kind (`host`, `extended`, `top`, `procs`,
`app`, `trees`, `events`, `shell`, `calls`, `control`, `traffic`, `logs`,
`artifacts`, `blobs`, `api`), bytes/s and messages/s over the last
minute, the state kinds marked "est." (a pro-rata split of an exact
per-frame total, measured ±5–40 %: `rd/logs-bytes-counter/lot0-preview.md`),
and `rx_bps` — what this backend measured on the WebSocket — beside as
the cross-check. In charts: `link.bytes` / `link.messages`, filtered or
grouped by `link_dir` / `link_kind`.

**How.** The agent counts and ships the `traffic` section every 5 s
(`3pm/docs/backend-agent/link-protocol.md` §7). `hub/node_state.rs`
holds it like any section it has no typed view of (`traffic()`),
`Collecting::finish` reads it leniently (`traffic_view`), and
`store/link_samples.rs` writes `threepm_link_bytes_total` /
`threepm_link_messages_total` `{cluster, node, link_dir, link_kind}` at
every store tick of a collected node, `Traffic::known` only (≤ 30
pairs a node, ~60 series): a kind this build does not know is dropped
and said once per node. `link_dir` / `link_kind` are `Custom` query
labels carried by the `link.*` lines only, like `bat` and `field`: no
word reserved, an application's own `link_kind` coexists.

**Tests.** `store::tests_link` (the series and their labels, an unknown
kind dropped and warned once, the catalog naming exactly what is
written, the hub handing the section over), `query::tests_link` (the
labels filter and group on `link.*` only), `dashboard_traffic` (the
panel's query, rows, "est." and escaping, under `node`).

## Charts: the metrics query

**What you see.** `api.queryMetrics(clusterId, body)` — `POST
/api/buckets/{id}/metrics/query` — answers a chart in one call: a
range and a step, one or more queries naming a metric of the catalog
with filters, a time function and an aggregation, formulas over their
refs, and options. The answer is one `times[]` array and, per series,
numeric `values[]` aligned on it with `null` for a gap, plus a summary
(min, max, avg, last, count) — what uPlot or ECharts draw as is.

CPU per app in the `prod` namespace, last hour, 30 s:

```json
{ "range": { "from": "now-1h", "step": "30s" },
  "queries": [{ "ref": "A", "metric": "process.cpu.utilization",
    "filters": [{ "label": "ns", "op": "eq", "value": "prod" }],
    "aggregate": { "fn": "avg", "by": ["app"] } }] }
```

Peak memory per node over 6 h, the 20 largest by their last value:

```json
{ "range": { "from": "now-6h" },
  "queries": [{ "ref": "A", "metric": "process.memory.rss",
    "time": { "fn": "max_over_time" }, "aggregate": { "fn": "max", "by": ["node"] } }],
  "options": { "max_points": 360, "max_series": 20, "sort": "last" } }
```

Memory in use as a percentage of the host's, per node — two hidden
queries and a formula:

```json
{ "range": { "from": "now-1h" },
  "queries": [
    { "ref": "U", "metric": "system.memory.used", "aggregate": { "fn": "sum", "by": ["node"] }, "hidden": true },
    { "ref": "T", "metric": "system.memory.total", "aggregate": { "fn": "max", "by": ["node"] }, "hidden": true } ],
  "formulas": [{ "ref": "PCT", "expr": "U / T * 100", "unit": "percent" }] }
```

What a query may name is `api.metricsCatalog(clusterId)`: ids in
OpenTelemetry style (`process.cpu.utilization`, `system.network.receive`,
`node.up`…), each with its kind (gauge or counter), unit, labels
(`node`, `app`, `instance`, `ns`, `mount`, `window`, `src`, the
host dimensions `core`, `iface`, `blockdev`, `sensor`, `gpu`, `array`,
`bat`, and a process tree's groups, `proc`)
and the time function a chart wants by default — `rate` for a counter,
`avg_over_time` for a gauge — so `time` can be left out. The values a
label takes in the bucket are `api.metricLabelValues(clusterId, "app")`.
A dashboard or an LLM that starts from nothing asks
`api.metricsInventory(clusterId)` (`GET …/metrics/inventory`) once:
the catalog lines that actually hold data in the bucket over the last
two days, grouped by the labels they carry, with their descriptions
and, for an application's own line, the apps that emit it; what every
other label holds (`window`, `mount`, `iface`, an application's own
labels…, 50 values at most); and the hosts with their processes
(`{ app, instance, ns? }`, label values that go into `filters` as they
are) — every series carries `node` = a node_id, and `hosts` is where a
hostname becomes one. Its first field, `tokens`,
says what the answer costs an LLM before it reads the rest.
It answers TOON-style text by default, a header per list and a row per
entry (about a fifth fewer tokens than the JSON); the generated client
gets the JSON (it sends `Accept: application/json`), and so does
`?format=json`. `?detail=index` drops the descriptions (about 40 %
fewer tokens; more than half fewer as text). The bucket page links it
(**inventory**, next to the API links).
A question that starts from a host or an app asks
`api.metricsByServer(clusterId)` (`GET …/metrics/servers`) instead:
each server keyed by node_id with the ids of its host's own metrics,
then its apps keyed by name, each with the ids of the metrics its
series carry on that server — what can be queried with `node` (and
`app`) without drawing nothing; a Linux-only line shows under the
Linux host only. The metrics are described once, in `metric_groups`,
as the inventory describes them; present means a series over the last
24 hours. Same formats and `?detail=index` as the inventory. The
bucket page links it too (**by server**, next to **inventory**).
One that starts from an app asks `api.metricsByApp(clusterId)`
(`GET …/metrics/apps`): each app keyed by name with the servers it
runs on, keyed by node_id, and on each the ids of the metrics its
series carry there — `agent` on every host, a worker on two; `hosts`
turns a node_id into a hostname, and a host's own lines stay with
**by server**. Linked on the bucket page as **by app**.
A step left out is derived from `max_points` (600 by default) on a
ladder of round values; `max_series` (200 by default, 2 000 at most) keeps the top
series by average (or `last`); `mode: "summary"` answers without
values, for an assistant that wants numbers, not a chart.

A query that is wrong is refused before anything reaches the engine,
400 `ERR_BAD_REQUEST` with a message naming the field: an unknown
field (a typo would otherwise draw the wrong chart), an unknown
metric, a label the metric does not carry, `rate` on a gauge, a regex
that does not compile, a range past 31 days, more points than
`max_points`. Without `--metrics-url` the route answers 409
`ERR_STORE_DISABLED`; an engine that is down, slow or busy is 503
`ERR_STORE_UNAVAILABLE`.

The same chart refreshed within the same step, by however many
viewers, costs the engine one request: the answer is cached for a
step (`--store-query-cache-mb`, 64 by default) and identical requests
in flight share one; `GET /api/v1/info` → `store.cache` shows hits,
misses and coalesced requests. Answers are gzipped when the browser
asks, which every browser does.

**Where it comes from.** `src/query/` (the contract, the catalog,
validation into a plan, formulas, alignment and execution — engine
agnostic, a test holds it), `src/store/victoria/{metricsql,read}.rs`
(the plan as MetricsQL with the bucket first in every selector and
`extra_label` on the URL, the reads under `--store-query-concurrency`),
`src/store/cache.rs` (the answers kept a step, single-flight),
`src/http/api_metrics.rs`; the gzip layer sits on every JSON route in
`src/http/mod.rs`. The engine's language never crosses the
API in either direction: OV `3pm-backend-docs-archived/backend` → Metrics API.

## Logs: what the nodes ship, queried

**What you see.** With `--events-url` set, every linked node sends its
logs (3pm 0.55.0 and later): its apps' output, the system journal and the
log files of the other programs it runs, reduced on the node to templates
(a line with its variables masked), counts per template per 5 minutes and
the raw lines that carry news (a new template, a warning once an hour, a
spike, the 30 s before a new error). What the node sends and why is in
3pm's `FEATURES.md`, "Log shipping". `api.logsQuery(clusterId, body)` —
`POST /api/buckets/{id}/logs/query`, the owner's, like `eventsQuery` —
answers the bucket's logs in one call. What went wrong over the last six
hours, errors and worse only:

```json
{"range":{"from":"now-6h"},"filters":{"sev":"err+"},"limits":{"templates":50,"lines":100}}
```

`range` is the events query's (`from` / `to`, epoch ms or `now-<duration>`,
30 days at most). `filters` are exact and AND-ed: `node`, `src` (the
node's source id), `owner` (the writer's name), `tpl` (a template id,
unique per source of a node only: it needs `node` and `src`, else
`400`), and `sev` — a word
of `debug`, `info`, `notice`, `warning`, `err`, `crit`, a list, or
`<word>+` for that level or worse. `text` keeps the shipped lines that
contain it (case sensitive); counts have no text. `limits {templates ≤
500 (200), lines ≤ 1 000 (200)}`.

The answer is `{range, sources, templates, lines, truncated}`:

- `sources`: every source the nodes declared over the range, `{node,
  hostname, src, kind (app|journal|file), path?, app?, owner?, format?,
  lines, shipped, dropped, templates}`;
- `templates`: worst severity first, then by lines, `{node, src, tpl,
  sev, text, lines, first_seen, last_seen, params: [{var, distinct,
  top}]}` — `sev` is the worst counted over the range; the templates
  beyond a node's 2 000 busiest come back together per source and
  severity as `tpl: null`, `text: "(other templates)"`;
- `lines`: the shipped lines, newest first, `{ts, node, hostname, src,
  tpl?, sev, why, line, ident?, unit?}` (`why`: `new`, `error`, `back`,
  `spike`, `return`, `context`);
- `truncated`: what the limits cut, and `counts`: the nodes whose
  counts the engine would not give (their templates and source `lines`
  are missing; 0 when the answer is whole).

Counts come in windows of about 5 minutes stamped at their end: the
window still open at `to` is not counted yet, so the last 5 to 10
minutes lag behind the lines. Without a `node` filter they are asked
five nodes at a time, four queries at once, 20 s for all (VictoriaMetrics refuses a query touching more than
`-search.maxUniqueTimeseries`, 100 000 on the fleet); a chunk it refuses
is logged and counted in `truncated.counts`, the rest of the answer
stands. `params` come back empty rather than failing the answer;
`distinct` is the largest over the whole range (merged by the engine),
`top` the values most often in the tops of the latest 2 000 windows.
Bounds: `limits.logsQuery` of `GET /api/v1/routes`.

**Series budget.** At any moment a node writes at most 2 000 labelled
templates (`tpl` = its id, usually one `sev` each) plus, per source, one
`tpl="other"` series per severity (6 at most) and one
`threepm_log_dropped` series per drop reason: with the agent's 512 files
plus its apps and the journal, ~2 000 + a few hundred series in
practice (prod, 2026-10-06: 36–601 per node, 1 977 for 14 nodes). The
seats bound only the labelled templates: `other` and the drops are
outside them, so the worst case for one node is 2 000 + its 1 000
sources × (6 severities + 4 drop reasons), ~12 000 series. The
2 000 seats bound the series written *at once*, not over a range: a
rerank (every 10 min) that moves a seat starts a new series, so a range
counts every template that held a seat during it. Log series share
VictoriaMetrics' `-storage.maxHourlySeries=500000` with the host
metrics; a node at its seat cap adds at most one series per new seat
taken.

**How it arrives.** `pushLogs`, `POST /api/buckets/{id}/logs`, is the
agent's own route (`link-protocol.md` §6b in 3pm): a batch every 10 s,
gzip NDJSON, the node's fleet bearer, any active node of the bucket.
The `welcome` says `logs {apps, host, files}` only when `--events-url`
is set; without it the route answers `503 ERR_STORE_DISABLED` and the
agents ship nothing. Refused before the body is read: no
`content-length` or over 1 MiB (`413`), a batch of the node already in
flight, or the node over its budget — 128 KiB/s inflated, 4 MiB burst
(`429` + `retry-after`). A body inflating past 4 MiB is a `413`, one
that is not gzip or does not open with `batch` a `400`. Records that do
not read, are of an unknown kind or name an undeclared source are
skipped and counted. A retried batch (same `run` and `seq`) is stored
once, a line once per `(node, src, pos)` (`pos` is opaque to the backend:
hashed with `src`, never parsed). A store that does not take the
batch is a `503 ERR_STORE_UNAVAILABLE`; the node retries, and only the
half still owed is written. At most 32 batches of the whole fleet are
read and stored at once: past that, `503 ERR_STORE_UNAVAILABLE` +
`retry-after: 1` (the agent backs off and sends the same batch again).
A record whose identifiers or labels (`src`, `run`, `path`, `app`,
`format`, owner fields, `ident`, `unit`, `pos`, a parameter's `var`)
hold a control character is skipped and counted: they become series
labels, stream fields and listing cells (a line's own text may hold
any). New sources or templates past the node's catalog bound evict its
least recently seen rows — never one the batch names (a source a
template of the batch belongs to included); refused only when the batch
itself fills the bound, then said in the answer (`refused {sources: [ids],
templates: [[src, tpl]], reason: "catalog_full"}`, threepm-link
`LogsRefused`; each list sorted and cut at 200, the node re-sends the
rest) and logged. A body slower than 30 s is a `408`; the
fleet-wide slot (32) is taken only once the body is read. `GET /api/v1/info` → `store.logs` counts
batches, duplicates, records, lines stored, bytes, refusals, failures.

**Where it is kept.**

| data | store |
|---|---|
| sources | SQLite `log_sources` (1 000 per node, least recently seen evicted first) |
| templates (text, severity, first/last seen) | SQLite `log_templates` (50 000 per node, the same; a template counted is marked seen once a day, a re-declared one too) |
| counts | VictoriaMetrics `threepm_log_messages{cluster,node,src,tpl,sev}`, one point per window; per node the 2 000 busiest templates labelled, the rest `tpl="other"` (the seats survive a restart) |
| drops reported by the node | VictoriaMetrics `threepm_log_dropped{…,src,reason}` |
| shipped lines, parameter stats | VictoriaLogs tenant `0:1` (`kind=line\|param`), stream `cluster,node,src` |

Events stay in tenant `0:0`; logs share the events instance and its
retention (`-retentionPeriod=30d` in `ecosystem.toml`), so a range never
goes past 30 days. The catalog follows: once an hour the sources and
templates no node named for 30 days are removed (`src/logs/retention.rs`)
— a daily-dated file no longer holds a source seat for good.

**Where it comes from.** `src/http/api_logs.rs` and `src/logs/` (budget,
parse, dedup, tally, render, sink) for the ingest;
`src/http/api_logs_query.rs`, `src/query/logs.rs`,
`src/query/logs_assemble.rs`, `src/store/victoria/logs_read.rs`,
`src/db/logs_read.rs` for the query; `src/db/logs_upkeep.rs` (eviction,
retention, seats and last-seen writes); migrations V21–V22. Tests:
`tests/it/logs_e2e/`.

## Insights: what deserves a look

`GET /api/buckets/{id}/insights` answers the question a fleet keeps
asking: what is wrong, or about to be, and where. The incidents are
ranked by the insights apps, nine Python processes that read this
backend's store next to it (`insights/` in this repository): absolute rules (a
disk full, a reboot loop, a collector silent), resources running out
or leaking, a machine unlike its peers, a series unlike its own past,
dated level shifts, series profiles, crash loops. Their `triage` groups
what they find on one machine and metric family, or on the fleet,
into an incident scored 0 to 10, and says how to read it: `problem`
(bad in itself), `context` (only unusual, damped), `inventory` (a
standing fact, an old agent, a worn battery).

**What you see.** Text for an LLM by default, one row per incident:
`id, score, tier, hostname, metric, labels, sources, since, passes,
label, summary`. With `?format=json` (the generated client's
`insights(clusterId, query)`), each incident carries its catalog
`metric` and a `chart`, a `queryMetrics` body that draws its headline
series, so a page draws an incident without knowing the store.
`summary` is the detectors' sentence, in French, with the figures;
`lang` says so. `label` is a person's verdict (`true`, `false`,
`known`; a `known` incident is listed only with `known=true`).
Filters: `tier`, `node` (its incidents and the fleet-wide ones that
name it), `min_score` (on the score as served), `limit` (20 by default,
at most 200). A fleet-wide incident has no `node_id`: its machines are
in `hosts`, and its `chart` draws them.
`insight(clusterId, incident)` gives one incident with the findings it
groups (detector, series, score, weight, summary), the process events
of its node over 6 hours, and the incidents that started with it.
`stale: true` says the ranking stopped moving (no pass for two
intervals), `error` why the last pass failed; `sources` the state of
each detector at the last pass. An incident id goes URL-encoded in the
path (a fleet-wide one reads `flotte@<cluster>:<family>`); an id the
list does not hold is a 404 answered without asking triage.

**From the logs.** The `logs` detector (`insights/apps/logs.py`, port
9708) reads what the nodes ship: the counts per template in
`threepm_log_messages` and one shipped line per template (VictoriaLogs,
tenant `0:1`), kept in an exemplar index of the last 7 days. Every log
finding says which incident it belongs to (a failing disk's kernel lines
join the disk incident, an app's errors its `process.logs`).
- **Rules:** a process crashing in the kernel's eyes (`segfault`, 5–6), a
  systemd unit that failed (4, once in 24 h `inventory` 2), kernel errors
  (3–4, `inventory` 2 when the same template comes every day), an error
  template never seen in 7 days (3); a kernel hardware pack (Wi-Fi
  firmware, disk I/O, read-only remount, NVMe, hung tasks, memory
  errors, temperature, USB loops; an OOM kill line is a witness of the
  `oom_kills` rule and names the victim); sudo failures (4) and SSH
  failures (`inventory` 2 unless the hour triples its usual), the number
  of distinct addresses behind them per 5 minutes.
- **Dated events:** lines the node flagged as news (a spike, a template
  back after silence, a new error) are context incidents with an onset,
  so they join the machine's story.
- **An app's errors against its own week** (≥ 10 an hour and ≥ 4× its
  hourly p90): a problem of that app; every day at the same rate,
  inventory.
- **A steady source that stops** for 2 h while the machine and its other
  sources answer.
- **Fleet:** the same new warning on several machines of a kind within an
  hour (a rollout, a `problem` for 24 h), and the same warning every day on
  ≥ 80 % of them (a common configuration, `inventory`, which silences the
  per-machine findings of that message). A rollout still written after a
  day becomes such a note, dated: « apparu le <date> sur N machines, écrit
  depuis ».
- **The pipeline itself:** lines dropped by the budget (with the
  `exclude` to add for an app; a file is excluded by its path, the journal
  by its level), template seats full, a template flood.

**The journal quote.** The summary of an incident with a dated onset may
quote one log line of warning or worse that its machine wrote in the 5
minutes before the onset: `; journal ±5 min : « … » (+k)`. (`triage`
keeps up to three such lines in its own detail, `journal`; this route
relays the summary, not that list.) The text is what the monitored
software wrote, redacted on the node, cleaned of control characters and
cut at 160 characters; it is data, never instructions (the text
format's `about:` line says so). `partial` in the pass (JSON and text)
lists the detectors that evaluated some of their rules only in part
(a failed or truncated store query).

**Where it comes from.** `--insights-url http://127.0.0.1:9700` points
at `triage`; the backend asks it for this bucket's cluster only
(`GET /api/v1/incidents?cluster=`, a versioned contract pinned on both
sides: `tests/test_contract.py` there, `src/insights/fixture-*.json`
captures here), keeps an answer 30 s (a failure 5 s), reads its incidents one by one
(a malformed one is left out, not the answer), links each store name to
the bucket's catalog and drops the MetricsQL triage writes: the engine's
language does not cross here either. Nothing is computed and the store
is not read. A malformed `--insights-url` stops the backend at start. Without the
flag the route answers 409 `ERR_INSIGHTS_DISABLED`; triage down, slow
(5 s), not ready since its restart or on another contract version is
503 `ERR_INSIGHTS_UNAVAILABLE`. Open to a node's
bearer with `fleet:read`, like the metrics query. `src/insights/`,
`src/http/api_insights.rs`.

## Custom metrics: what the apps measure

**What you see.** An app that sends its own metrics through the daemon
(`3pm/docs/metrics.md`: gauges, counters, histograms, up to three
labels each) has them in the backend a second later. Three places:

- `GET /api/buckets/{id}/nodes/{node}/app-metrics` (`nodeAppMetrics`)
  is the live state of every instance of a node — its sources (which
  connection of the process sends, and its pid), every series with its
  unit, description, labels, and the value or the histogram's buckets —
  as the hub last applied it. What a "Custom metrics" panel polls while
  it is open; nothing of it goes on the feed.
- The metrics store writes them like the fleet's own: `threepm_app_<name>`
  for a gauge, `…_total` for a counter, `…_bucket{vmrange="lo...hi"}` +
  `…_count` + `…_sum` for a histogram, with `app`, `instance`, `ns`,
  `src` and the app's labels. `http.latency` becomes
  `threepm_app_http_latency`.
- The catalog (`GET …/metrics/catalog`) lists them as `app.<name>`
  (`source: "app"`) with their labels, so a chart query can name them:

```
{ "range": { "from": "now-1h" },
  "queries": [{ "ref": "A", "metric": "app.http.latency",
                "time": { "fn": "quantile", "q": 0.99 },
                "filters": [{ "label": "route", "op": "eq", "value": "/api/bucket/:id" }],
                "aggregate": { "fn": "sum", "by": ["app"] } }],
  "options": { "explain": true } }
```

A histogram takes `quantile` (with `q`; the median when `time` is
omitted), or `rate` / `increase` of its count; `raw` and the
`*_over_time` family are refused on it, as is a label the metric never
carried. `options.explain` returns `results[].expr`, the expression
exactly as compiled for the engine — to read, never to send back.

**Events.** An app's events (`io.emit(...)` and friends: a name, a
level, attributes, an exception, a fingerprint and a repeat count)
reach the feed as `emit` (fifty a second per bucket at most) and the
events store as `kind: "app"` lines dated by the app's clock, with the
attributes flat under `a.` and `exception.*`. `POST
/api/buckets/{id}/events/query` (`eventsQuery`) reads them back —
newest first, a range, exact filters (`kind`, `app`, `level`,
`event_name`, `fingerprint`…), a limit — without the engine's language
ever showing. A node may emit a hundred application events a second
(burst 500); the rest are dropped and counted.

**On the dashboard.** Each node card has a "Custom metrics" panel,
closed by default: open, it polls `nodeAppMetrics` every second and
shows each instance's sources and series (value, unit, description,
labels; a histogram's count, sum and approximate p50 / p90 / p99).
The "Events" list shows the live feed on top and the last day from
the store underneath.

**How it behaves.** An instance's metrics are the `app/<id>` section of
its node's state (see "The node's whole state, received as diffs"): the
agent sends every `--app-metrics-every-s` seconds (1) what moved in it,
the backend rebuilds the section and replaces the instance's state
whole each time — so an instance a restart gave a new incarnation
keeps nothing of the old one, and a lost frame is that section asked
again (`state_resync`), at most once a second per node. A state over
the caps (1 000 series, 128 buckets a histogram, 100 instances a node)
or whose name is off the label grammar is refused and counted
(`nodeAppMetrics.refused`), and nothing is held for the instance. It
goes with its section (`gone`) and with its row of the process table.
The catalog remembers a metric for 30 days after it was last seen and
holds 5 000 per bucket (`info.store.catalog_over_cap` counts the rest).

**Where it comes from.** `hub/node_state.rs` (the section), `hub/apps.rs`
(the state per instance, the caps), `store/app_samples.rs` (the lines),
`query/dynamic.rs` + `db/app_metrics.rs` (the catalog and its table),
`http/api_app_metrics.rs`; the wire is the `app/<id>` section of
`AgentFrame::State` in `threepm-link` (`3pm/docs/backend-agent/link-protocol.md`).

## Process trees: every process of an app

**What you see.** An app is every process it spawned — `npm run dev`,
the `node` it starts, Electron and its helpers — and 3pm reports it so
(`3pm ls --tree`). The backend has the same picture, in three places:

- The process table of each node card writes `saw +13` after an app
  with other processes (`proc_count` on the telemetry's record, the
  spawned pid included); the link opens the card's **Process trees**
  panel on that app. Open, the panel polls `nodeProcessTrees` every
  second and draws, per app with more than one process, the folded
  tree `3pm ls --tree` draws — a chain of single children on one row
  (`npm › bash › node › Electron`), twins grouped (`Electron Helper
  ×4`), `⚠` on a process that left the app's process group and that
  `3pm stop` will not reach — with pid, CPU, memory and uptime (no
  ports: they do not travel on the link, the process table's
  `listen_ports` is their union), then every process under a fold.
  The title says how many trees, and how many were refused.
- `GET /api/buckets/{id}/nodes/{node}/trees` (`nodeProcessTrees`) is
  what the panel reads: one entry per instance the hub holds a tree
  for, each with `procs` (every line: `procs[0]` the spawned pid, a
  parent before its children) and `compact` (the folded rows, computed
  by the same code as the CLI's), the totals of the same frame, and
  the `refused` count. `GET …/procs/{proc}/tree`
  (`nodeProcessTree`) is one instance, 404 while the backend holds no
  tree for it: an instance running alone, or before its first tree.
  Nothing of it goes on the feed: at 300 KB a second for two hundred
  apps of fifteen processes, a broadcast would cost every open
  dashboard the whole fleet.
- The metrics store writes one series set per **group** — the members
  of an instance's tree sharing an executable name — for as long as
  the agent sends the tree: `threepm_subprocess_procs` (the members),
  `threepm_subprocess_cpu_ratio` and `…_memory_rss_bytes` (summed),
  `threepm_subprocess_detached` (how many left the process group), on
  the app's labels plus `proc` (the executable's name), and the tree's
  count as `threepm_process_procs`. There is no `pid`: keyed by pid, a
  kiosk's shell loop forking a `sleep` every 3 s made 17 000 series
  sets a day per tablet (735 000 new series on 2026-10-01, the engine's
  daily cap within reach of 25 tablets, and past it every node loses
  its new series). By group, the count follows the programs. Extended
  Monitoring's gauges are summed only when every member gives them (a
  partial sum would read as a drop), the fd limit becomes
  `threepm_subprocess_fd_usage_ratio_max` (the highest open / limit
  among the members: the `EMFILE` question, which a sum of limits could
  not answer), and the ten counters are accumulated per group by the
  backend: a member is `(pid, start time)`, its first reading a
  baseline when it was already running at the group's first sight
  (a backend restart then reads as a counter reset, not a spike) and
  whole when it started after, and a member that exits keeps its share
  — the counters never go down while the group lives (a group unseen
  for a minute starts again from 0). What is lost: the history of one
  member among homonyms (which of eight `electron` grew), and what a
  process that lives and dies between two store ticks did; the member
  by member detail stays live (the tree panel, `nodeProcessTree`). The
  catalog lists them as `subprocess.*` and `process.procs`; a fleet
  query wants an `aggregate.by`, like the host rows. `proc` and `pid`
  stay reserved label keys for applications, in 3pm too, and a query
  naming `pid` is refused (400).

**How it behaves.** An instance's tree is the `tree/<id>` section of
its node's state (see "The node's whole state, received as diffs"): the
agent sends the *shape* whenever an instance's membership moved, and the
*figures* every `--trees-every-s` seconds (1 by default; `0` = the
shape only, when it changes) for every instance with more than one
process. The hub keeps the last tree per `(node, instance)`, replaced
whole each time its section moves — a restart's new incarnation
included — drops it when the section is gone (the app runs alone now),
and forgets it with the instance's row of the process table. Nothing is
asked by heuristics any more: a lost frame is a gap of that section,
answered `state_resync`. Guards: 100 trees per node, 256 lines a tree
and 64 bytes a name (the daemon's own caps, enforced again), and a tree
for an instance the process table does not list is refused; each is
counted in `refused`, and the tree held for the instance goes rather
than stay behind its section — never an empty tree, which takes no
place. Member names are cleaned again on the way in, as the daemon
does — control characters, bidi overrides and line separators blanked —
so no agent can bend the panel's lines. An older 3pm reports the
spawned pid alone. The history's resolution is the store's
(`--store-interval-ms`, 5 s by default); the panel is live at 1 s
whatever it is. A tree held is current as of the node's every frame (an
agent sends it only when it moved), so the store keeps writing its
members' figures while the node is heard — with `--trees-every-s 0`
those are the figures of the last shape frame — and stops a minute
after the node's last frame; the count (`threepm_process_procs`) comes
from the process table.

**Where it comes from.** `hub/node_state.rs` (the section),
`hub/trees.rs` (the state per node, the guards), `http/api_trees.rs`
(the two routes over `threepm_protocol::tree::compact`),
`store/tree_samples.rs` (the lines) + `store/tree_groups.rs` (the
groups, their sums and accumulators, carried by `prefix.rs`), `query/catalog/lines/{process,subprocess}.rs`,
`assets/pages/fleet.js` (`processTreesPanel`, `treeGuides`, whose
escaping `tests/it/dashboard_render.rs` holds); the wire is the
`tree/<id>` section of `AgentFrame::State` and `welcome.pace.trees` in
`threepm-link` (`3pm/docs/backend-agent/link-protocol.md`). Tests:
`hub/trees_tests.rs` (replacement, the guards, the two ways out),
`store/tests_tree.rs`, `tests/it/trees_e2e.rs` (a fake agent: served,
stored, refused, cleaned, gone, the row gone) and
`tests/it/real_agent_trees.rs` (a shell and two sleeps under a real 3pm
and its agent).

## Remote actions and profiles: run what the apps expose

**What you see.** An app declares actions through its SDK (`io.action`
in `@3pm/api`); the daemon publishes them with the sources of each
instance, the agent relays them, and this backend lets the operator run
one from the dashboard or the API and read the result — or the file it
left: the SDK's built-in `3pm:profile` takes a CPU or heap sampling
profile, stored here as an artifact with its hot frames.

```console
$ curl -sS -H "Authorization: Bearer $TOKEN" -X POST \
    http://127.0.0.1:9616/api/buckets/3pmc_…/nodes/3pmn_…/actions \
    -d '{"app": "api", "action": "cache.flush", "params": {"all": true}}'
{"call_id":"kq3Z…","node_id":"3pmn_…","request":{"op":"action","target":{"name":"api"},"action":"cache.flush","params":{"all":true},"timeout_ms":5000},"state":"sent","sent_at":1726300000000,"deadline_at":1726300020000,"artifacts":[]}
$ curl -sS -H "Authorization: Bearer $TOKEN" http://127.0.0.1:9616/api/buckets/3pmc_…/calls/kq3Z…
{"call_id":"kq3Z…","state":"replied","result":{"res":"triggered","results":[{"proc_id":3,"name":"api","instance":0,"source":"main","action":"cache.flush","duration_ms":12,"ok":true,"data":{"flushed":4}}]},…,"artifacts":[]}
$ curl -sS -H "Authorization: Bearer $TOKEN" -X POST …/nodes/3pmn_…/actions \
    -d '{"app": "api", "action": "3pm:profile", "params": {"kind": "cpu", "duration_ms": 10000}}'
$ curl -sS -H "Authorization: Bearer $TOKEN" …/api/buckets/3pmc_…/artifacts?node=3pmn_…
[{"artifact_id":"a9Xf…","app":"api","instance":0,"source":"main","action":"3pm:profile","ext":"cpuprofile","bytes":184322,"gz_bytes":41210,"sha256":"…","summary":{"kind":"cpu","samples":9500,"top":[{"function":"hot","location":"/srv/app/worker.js:6","self_value":6120000,"percent":62.4},…]},…}]
$ curl -sS --compressed -H "Authorization: Bearer $TOKEN" -o hot.cpuprofile …/artifacts/a9Xf…/file
```

In the dashboard, an app that declares actions has them right under
its row of the node card's process table — under its CPU and memory,
as `3pm ls` would list them: a button per action and per source (`CPU
profile 10 s` and `Heap profile 30 s` for `3pm:profile`, with a
countdown; a params field for the rest — the button's tooltip is the
declared description and timeout). Click one: the **Action results**
dialog opens on the call — `sent…`, "waiting for the node…", then
`replied` with the handler's JSON folded under it (or the error in
words), fetched with `getCall` as the feed reports each step; the last
10 calls stay in the dialog. The rows are fed by `GET
…/nodes/{node}/actions` (`nodeActions`) — the declarations alone,
nothing of the metrics — polled every 3 s for every node that is not
offline; a row is repainted only when the declarations move, so what
is typed in its params field survives the process table's one-second
refresh (the table is kept by app id, not rebuilt). The buttons run
as the logged-in account, which owns the bucket. The
"Artifacts" panel lists what the node's actions left — the top 5
frames of a profile, raw and gzip sizes, a download per file. The feed
carries every step (`call` events: `sent`, `acked`, `replied`, or
`unacked`, `refused`, `lost`) and every file (`artifact` events), but
only states, ids, timings, sizes and error codes: every open page of
the bucket reads it, so params, results, messages and summaries stay
with `getCall` and `listArtifacts`.

**How it works.** `nodeAction` builds the link's `action` command
(`nodeCall` sends `restart`, `stop` or `ping`; an `action` there is a
400), records the call (`calls`, at-most-once) and hands it to the
node's live session (offline: 409; its queue full: 503
`ERR_NODE_BUSY`, the call recorded `refused`); the agent acknowledges,
asks its daemon (`trigger`, always `remote: true`, so an action declared
`remote: false` answers `refused`) and replies with each instance's
outcome. A handler's `data` travels gzip + base64 **in the reply**,
while the call's results stay under 256 KiB together; a result past
that fails its own outcome (`too_large`) and is a file to return
instead. A **file** an action returns never travels on the link at
all: before its reply, the agent uploads it to `PUT
…/calls/{call}/artifacts` with the bearer of its `welcome` — raw
bytes, at most what the action declared it may answer (32 MiB when it
declares none), one per instance and sixteen per call — and the reply
names the artifact the upload minted, with the size and SHA-256 of the
raw bytes; a reference that is not the upload's fails that outcome, and
an upload no reply claims — or whose call settled while its bytes were
arriving — is removed. A bucket takes 2 GB of files in flight at most,
whatever its nodes declare; retention keeps it at 1 GB. The bytes
are gzipped and hashed as they arrive and stored under `--data-dir`
(`<db>.d/` by default) as
`artifacts/<bucket>/<node>/<id>.<ext>.gz` — never in SQLite. The call
is settled first; a profile (`cpuprofile`, `heapprofile`) is summarised
right after, two at a time (`threepm_core::profile_summary`, the same
code `3pm profile` prints with), so its `summary` reads `null` for a
moment. Retention: 30 days and 1 GB per bucket, oldest out; a call
never answered by its deadline (the caller's `timeout_ms`, else the
timeout the app declared, 30 s when none, plus `--call-margin-ms`,
15 s) is settled by the sweeper — `unacked` when the agent never saw
it, `lost` when it did; one a reply has started to settle gets 60 s
more.

**The locks.** The host must allow `proc:action` — its `link.toml`
`caps`, declared in every `hello` and shown on the node's row, the
agent itself refusing what it does not carry; the action must be
declared by the app,
and declared `remote`; the routes are `guarded` — from another site a
session cookie (or `Basic`) is refused 403, an `Authorization: Bearer`
passes (see "From another origin: CORS is open") — and answer only the
account that owns the bucket — another account gets the 404 of a
missing bucket, nobody a 401 — or, for `nodeAction`, `getCall` and the
artifacts, a node of that bucket whose bearer carries `fleet:invoke`
(see "Node calls"), since they run code on the nodes and serve what it
produced (a profile shows an app's code). The upload is the one route
no account may call: only the node the call was **placed on**, which
needs no scope for it — answering a call must not cost the right to
place one.
Every call is audited
on the node (`proc_action` on its bus) and here (`kind: action` in the
events store, the feed's `emit`).

**What there is not.** No roles, and no permission per action, per app
or per node for an account: its tokens (`3pmu_…`, see "Accounts") carry
no scope, and whoever holds one — or a session it opened — can run
every `remote` action of every app on every node of every bucket the
account owns, restart or stop any app, and read every profile ever
taken there. Without a credential the routes are closed, not open. The
finer grains are elsewhere: a node calling another is held to the
target's `callers` rules (by calling node, application and action, see
"Node calls"), and the app has its own — `remote: false` keeps an
action off this path altogether, and a handler is expected to validate
its `params`. Access finer than the account, if it comes, is a backend
feature of its own; nothing on the wire assumes it.

**What a profile is.** `3pm:profile` takes a *sampling* profile, never a
dump: `kind: cpu` samples the stack (a `.cpuprofile`, default 1 ms),
`kind: heap` samples allocations (a `.heapprofile`, `HeapProfiler`
sampling, default every 32 KiB allocated) and says which functions
allocate — not what the heap holds. A heap snapshot is deliberately
not offered from the fleet: it freezes the JavaScript thread while it
is taken, weighs hundreds of MB on a large renderer and carries the
app's data, secrets included.

## Node identity and fleet access

**What you see.** Each node's card has an **Access** panel: three boxes —
*List the fleet*, *Read the fleet*, *Place calls* — and a list of who may
place a call on that node. Nothing is granted when a node enrolls, so a
freshly linked host can read nothing of its bucket. The node table shows
a badge per scope a node holds. What you grant or take away is felt on
that node's very next request; nothing has to reconnect.

**Where it comes from.** Every `welcome` hands the node its own bearer
(`3pma_<10 hex>_<64 hex>`), so `3pm fleet …` on a server authenticates as
*that node* and the owner's account token never has to sit on it. The
bearer says which node is calling and nothing more: the node's row —
its state and its scopes — is read again on every single request, which
is what makes a grant and a revocation both take effect at once. The
bearer lives as long as its WebSocket session plus ten minutes (a short
cut does not break a command in flight), is dropped on the spot by
*Revoke*, and is never persisted: a backend restart re-mints at the next
handshake. Only `sha256(bearer)` is ever kept in memory, and the token
never reaches a log.

Serving a call needs no scope at all. `getBlob` and `putBlob` admit a
node that is **party to the blob** — the call's writer or its reader —
whether or not its row carries anything, because the blob is the
authorisation and always was: one that is not this node's reads as a
404, exactly like a blob that does not exist. Asking a target for
`fleet:invoke` before it could read the bytes sent to it would have
handed it, in exchange, the right to call every one of its neighbours.
`fleet:invoke` is what it takes to **place** a call, and nothing else
opens that way: a surface test names the two routes.

A node's bearer reaches fourteen read routes and nothing else — never
`approveNode`, never an enrollment token, never an account, and never
the panel that would grant it more. What those routes answer depends on
the scope: with `fleet:list` a node reads the fleet as a list — every
node's identity and its process table, as `3pm ls` shows it — and its
own grant, never a neighbour's (who may invoke what on whom is the map
you would need to move sideways, and it stays with the owner). With
`fleet:read` it also reads the hosts, the daemons, the per-process
history and the applications' own events. Of its calls it sees only the
ones it placed itself: `getCall` answers as if a neighbour's call did
not exist, the artifact list leaves them out, and the live feed is
filtered and projected the same way — one rule, whether a node asks a
route or reads the stream. `GET /api/v1/routes?route=<name>` names the
scope each route needs.

## Node calls: one host asking another

**What you see.** A node places a call on another node of the same
bucket with its own bearer, through the same `nodeAction` an operator
uses. The caller needs `fleet:invoke`, and that is enough: it may run
any action of any other node of the bucket, `3pm:profile` included. The
**target** can narrow it in its Access panel ("Who may call this node"):
empty, it accepts every node that may call; once it holds a rule, only
what a rule admits goes through, and an action of the `3pm:` prefix has
to be named in full there. A rule no call matches, `{"from": "*",
"app": "-"}`, closes the node to all of them. On the dashboard a call reads `by 3pmn_…
(edge-7)/edge` — the node, and the application its daemon said was
asking; `-` stands where no application was named, which is what `3pm
fleet call` from a terminal looks like. That application name is
declared by the **calling node's own daemon**: it is an operational
guard rail for reading a fleet's traffic, not a boundary — the boundary
is the node, and any node a rule admits can send any `from_app`.

A node may also call **itself**, under the same rules (`fleet:invoke`,
and a rule that admits it once its Access panel holds one): the bytes
loop through this backend and back into the agent that placed the call.
It is how two applications of one host share a session, since a daemon
never opens a `live` leg on its own — `io.open('<own node id or
hostname>', …)`.

**Linking two machines from a script.** Everything the Access panel
does is one route, `setNodeAccess`, sent with the owner's account token.
To let the node A call `kiosk/show` on the node B:

```bash
T=3pmu_…; U=https://fleet.example.net
curl -s -H "Authorization: Bearer $T" $U/api/buckets                          # the bucket: cluster_id
curl -s -H "Authorization: Bearer $T" $U/api/buckets/$C/nodes                 # node_id by hostname, scopes, callers
curl -s -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
  -X POST $U/api/buckets/$C/nodes/$A/access -d '{"scopes":["fleet:invoke"]}'  # A may place calls
curl -s -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
  -X POST $U/api/buckets/$C/nodes/$B/access \
  -d '{"callers":[{"from":"'$A'","app":"kiosk","action":"show"}]}'            # B accepts A for kiosk/show
```

Each list sent replaces the node's list whole (read it from `listNodes`
and send the old rules back with the new one); a list left out is
kept. Both answers are the node's new `NodeView`, and the change holds
from the node's next request. The same two bodies are the "Link two
machines" recipes of `GET /api/v1/routes`.

A call can carry bytes. `input` (how many, their SHA-256, their
extension) opens a blob the caller writes and the target reads; `output:
true` opens one the other way. Neither application ever speaks to the
network: the SDK writes to its own agent's socket, the agent relays, and
what comes back is a file on disk. The answer to `nodeAction` names both
blobs, so nothing has to be parsed out of the request. A call that
carries bytes either way has to name its `instance` — sixteen instances
would each pull the same hundred megabytes, or write over one another's
answer — and `deadline_at` allows for the transfer as well as for the
handler: half a gigabyte is given time to travel, a small file reserves
a minute, not ten.

Both blobs are charged to the node that **placed** the call, including
the one its target writes: a caller you accept spends its own allowance
and never yours. The output blob reserves what the target's action
declared it may answer — `io.action(name, {output: {max_bytes}})`, as
the node published it, never a size the caller names — and 256 MiB when
it declares nothing (an older agent, or an action without the bound):
a 2 MB screenshot costs its caller 2 MB of quota, so dozens can run at
once where eight did. `nodeAppMetrics` and `nodeActions` show it as
`output_max_bytes`. The blob is given back the moment the call is over
and nobody claimed it. A node has at most a hundred blobs in flight at
once — a blob stops counting the moment it has been relayed, not when its
retention ends, so a caller downloading output after output never runs
out of room — and opens at most two hundred a minute; a blob that **carried nothing** — no byte written,
no reader ever attached — gives its creation back when it is forgotten,
so a call refused by a rule, by a node that was not there or by a target
that said no costs the retry nothing. One byte, or one reader, and it is
spent for the minute like any other. An input bigger than `--blob-max-mb` is refused before
anything is reserved, because a blob holds what it declares against its
bucket's quota for as long as it is open.

**Where it comes from.** The target's answer is pushed into the output
blob and announced on the outcome. The backend keeps no copy: the
answer stays the call's blob, and `getBlob` reads it until the relay's
own retention takes it. The outcome must name the blob this call opened,
with a well-formed digest and extension, or it fails; so does a second
instance trying to claim the one blob the call opened. The backend never
reads those bytes — the relay never hashes either — so the digest the
target announced is its claim, relayed as is.

Once the call is over, `getCall` says `result.ok` (the action ran; each
instance still carries its own `ok`) and fills the output in: how many
bytes, which digest, which extension. That is what lets the side that
pulls the blob check it against something rather than trust the relay.

When the call settles, the node that placed it is told on its own link,
without asking — the backend re-reads its row first, so a node whose
`fleet:invoke` was taken away in the meantime hears nothing and reads
the result through `getCall` if the scope comes back. If it was not
connected at that moment, its next `welcome` hands it the calls it
placed that are over (the last twenty-four hours, at most 256 ids,
newest first); the agent ignores the ones it already knows and reads the
rest with `getCall`. Nothing is queued, and nothing is replayed: a call
happens at most once.

### Between LAN neighbours: the bytes skip the relay

**What you see.** A node calling another node that sits on the same LAN
(same public address, as this backend sees both links) gets `peer` in
the answer that placed the call — the target's node id, public key and
private addresses — and the target's ticket names the caller the same
way. The two agents move the call's file between themselves over TCP +
TLS (each presents its node key; only the key the ticket names is
accepted). The call is placed, audited and settled here exactly as
before, and its blobs stay open as the fallback: when the direct
connection does not come up, the bytes take the relay without the
caller noticing. Once the writer knows, `path` on the call (`getCall`,
the feed's `call` events) says `direct` or `relay`.

**Where it comes from.** `calls/direct.rs`: the rule (both ends are
nodes, not the same one, both settings allow it, both announced a port
in `hello.direct_port`, same public address, a private address each),
the peers built from the hub's view of each node (`Hub::direct_of`:
the session's port, the address the link came from, the host's private
interfaces — at most eight), and `path_report` (`blobs/path.rs`: a
`direct` report from the blob's writer takes it off the relay and gives
its creation back — a file is forgotten, a `live` leg stays in the
registry, served by nobody, reserving nothing, alive on its writer's
`blob_stat` — unless the relay carried a byte of it or has a reader). Per node:
`setNodeAccess.direct` — `off`, `files`, `all` (default: a stream's
live legs too, on both ends: the relay then has nothing to show a
viewer; set `files` on a node whose sessions must stay watchable). `--direct off` turns it off for the whole backend.

A session (`stream: true`) goes direct only when both nodes say `all`.
Its legs then never reach the relay; what `live` shows on the call is
what each writer reports every 5 s (`blob_stat`), marked `direct`, and
the dashboard has no Watch button for it — "direct · no preview". A
leg gone direct is over 15 s after its writer's last report — an end
and an abandon read the same — and it ends alone: the other leg, if it
went by the relay and is still writing, runs on (a short question, a
long answer); one that is idle or never started ends with it.

## Shells: `3pm fleet ssh`

**What you see.** `nodeShell` (`POST /api/buckets/{id}/nodes/{node}/shell`,
body `{ "term": "xterm-256color", "cols": 120, "rows": 40 }`) opens a
login shell on the node — the login shell of the unix user its agent
runs as, on a real pty of that size — for the bucket's account or for a
node holding `fleet:invoke` whose call the target's `callers` admits
for the action `3pm:shell` (`app` `-`; an empty list admits every node
that may call, a wildcard action never reaches it). A node the rules do
not admit reads 403 `ERR_SHELL_NOT_ALLOWED`, one without
`fleet:invoke` 403 `ERR_FORBIDDEN`, a bad `term` 422; a page on
another site is refused even with a `Bearer`, which `nodeAction` lets
through. The answer is the call: `input.blob`, the keyboard, which the
caller `PUT`s as one long streamed body, and `output.blob`, the screen,
which it `GET`s — two `pipe` blobs, lossless and with back pressure,
one reader each, carrying the `attach` messages of `3pm attach` (input,
resize, ping; output, pong, exited). The call stays in flight as long as
the shell runs, a day at most, with `live` counting the bytes each way;
a node's link that reconnects meanwhile does not end it. It settles
`replied` with `res: "exited"` and the shell's `code` or `signal` (`ok`
on 0), or with the agent's refusal — `forbidden` when the node took
`node:shell` out of its `link.toml` or runs an agent that does not know
the command. A leg quiet for a minute (the command line pings every 20
s, the agent answers), a `3pm fleet ssh` that went away, or a backend that stops, ends both legs and the
shell is hung up.

**One command instead** (`3pm fleet ssh <node> -- <cmd>`): the same
body with `"command": ["wc", "-c"]` — at most 256 words, 16 KiB in all,
no NUL, the first not empty, else 422 — and `"pty": true` for a pty.
The backend places `exec` rather than `shell` (an agent that does not
know it refuses it, and the legs end at once as for any refusal); the
node runs the command through the same login shell, on a pty or on
pipes, where the screen leg carries its stdout as `output`, its stderr
as `stderr` and the keyboard leg its stdin, closed by `eof`. Same
gate, rules, quotas, legs and end; `open_requested` (and `refused`)
carry `command`, the agent adds an `exec` line.

Every session is audited in the events store (`eventsQuery`, `kind:
shell`, `call: <call id>`): this backend writes `open_requested` (who,
from which host, the terminal) when it places the call and `ended`
(the status, the agent's refusal, or `lost` / `unacked` when the call
settled without the agent) when it settles — two lines a node cannot
leave out —, and `refused` (who, why, no `call`) for an attempt it
turned away once it knew the target (`callers`, the node offline or
busy, a quota); a session the agent refuses has its two legs ended at
once. The agent adds `open` (user and shell), `line` (each
command line as the screen showed it, `[hidden]` where the terminal did
not echo, the raw input under a full-screen program), `screen` and
`close` (duration, bytes each way, how many lines it sent: a gap in the
store shows against it). `line` and `screen` come only from a session
that is recorded: without `--record` (the command line's default,
bound in the handshake),
`record_shells` off or a node with `[shell] record = "never"`, the
agent sends `open`, `exec`/`attach` and `close` with `lines: 0` — no
gap. None of them is rate-limited, and none goes on
the bucket's feed, which nodes holding `fleet:read` follow too. What
the agent writes is as complete as the screen lets it be, and no more:
the backend's lines are the ones to trust. A `line` names the call its
agent says it belongs to; the backend does not check that the call is
that node's, so read a session's lines with `node` as well as `call`.

**Where it comes from.** `http/api_shell.rs` (the route),
`calls/shell.rs` (the legs, the backend's audit lines, the shell whose
legs are gone), `db/calls_shell.rs` and `db/calls_abandon.rs` (a shell
outlives its connection), `store/shell_line.rs` (the lines), `blobs/`
(`pipe`).

**Attach.** With `attach: {app, instance?, mode?, read_only?, peek?}`
(never with `command`, an app of 1 to 256 bytes without a control
character, `mode` `rendered` or `raw`, else 422), `nodeShell` places an
`attach` call (`3pm fleet attach <node> <app>`): the terminal of a
`tty = true` app of the node, relayed by its agent from its daemon's
own `attach`, on the same two legs, under the same rules. The app is on
the `open_requested` (and `refused`) line; the agent's `attach` line
follows its `open`. An attach call is a shell's everywhere a shell's is
found (`is_shell`, the sweeps of gone legs and of a reconnect).

**As another account.** With `user` (a unix login,
`[a-z_][a-z0-9_-]{0,31}`, never with `attach`, else 422), `nodeShell`
wraps the shell or command in `run_as {user, req}` (`3pm fleet ssh -l
<user>`): a new op, so an agent that predates it refuses it as unknown
rather than ignoring a field and running as itself. The target's agent
becomes that account only when it runs as root, and otherwise takes
only its own name. `user` is on the `open_requested` and `refused`
lines. Every reader of a stored request reads the wrapped one
(`calls::shell::unwrapped`: `is_shell`, the legs, the call's view, the
claimed output; the SQL of the shells in flight and of an abandoned
link lists `run_as` beside `shell`/`exec`/`attach`).

**Port forwarding** (`3pm fleet ssh -L [bind:]port:host:port <node>`).
`nodeForward` (`POST /api/buckets/{id}/nodes/{node}/forward`, body
`{host, port}`: a host of 1 to 255 bytes without space, a port 1 to
65535, else 422) places one `forward` call per TCP connection the
command line accepted: the node connects to `host:port` (any
destination, the owner's switch is the node's `node:forward`
capability, on by default) and pumps the bytes. The answer and the
legs are a shell's, carrying `live` frames of raw bytes; an empty
frame flagged `END` is a half-close of that way. The same guard as a
shell (account, or a node's `fleet:invoke` the target's `callers`
admits), but its own action, `3pm:forward` — a `3pm:shell` rule does
not open a tunnel — and its own refusal, 403 `ERR_FORWARD_NOT_ALLOWED`;
never a `Bearer` from another site. A connection the node cannot make
settles the call with the agent's `err` (`connect host:port: …`).
Audit, `kind: shell`: `open_requested` / `refused` carry `forward` =
`host:port`, `ended` when it settles, the agent's `close` the bytes each
way. A forward call is a shell's for every sweep (`is_shell`, gone legs,
a reconnect).

`-W host:port` (`3pm fleet ssh <node> -W host:port`, ssh's
`ProxyCommand`) is one such forward whose local side is the command
line's stdin and stdout: nothing of its own here.

**Reverse forwarding** (`3pm fleet ssh -R [bind:]port:host:port
<node>`). `nodeListen` (`POST /api/buckets/{id}/nodes/{node}/listen`,
body `{bind, port, e2e}`: `bind` empty for the node's loopback, else a
plain host of 255 bytes at most; `port` 0 for one the node picks; `e2e`
must be `true`, else 422; 409 `ERR_NODE_NO_E2E` for a node that cannot
encrypt) places one long `listen` call: the node binds, and its sealed
legs carry `bound {port}`, then one `accepted {conn, peer}` per
connection, the caller's keepalive every 20 s the other way — what
keeps the quiet legs under the relay's idle minute (no sweeper
change). Whether a non-loopback `bind` is allowed is the node's own
`[shell] gateway_ports`, which this backend neither reads nor sets.
Each accepted connection is taken over by a `nodeForward` with
`{accepted: {listen, conn}, e2e: true}` and neither `host` nor `port`
(422 otherwise): the node pumps that connection instead of dialing.
That forward is refused with **one 404** (`no running listen <id> of
yours on node …`) unless `listen` is a `listen` call on this node,
placed by this caller, still `sent`/`acked` — the same words whichever
failed, so a guessed call id says nothing of another caller's listen;
`3pm:forward` is checked first, so a caller it turns away learns
nothing of the listen either. Guarded exactly as `nodeForward`
(`3pm:forward`, `ERR_FORWARD_NOT_ALLOWED`, never a cross-site
`Bearer`). Never recorded. Audit, `kind: shell`: the listen's
`open_requested` / `refused` with `bind` and `port`, `ended`; each
connection taken over a forward's own line with `listen` (the call id)
and `conn` in place of `forward`. A
listen call is a shell's for every sweep (`is_shell`, the SQL of the
shells in flight and of an abandoned link). Code:
`src/http/api_listen.rs`, `api_forward.rs` (`listen_of`, `accepted`).

**Sessions are recorded when the caller asks.** Every shell, command
(`-- <cmd>`) and attach session placed with `--record` (`record: true`;
a current 3pm sends `false` without it, an older one that sends nothing
counts as asking) on a bucket whose `record_shells` setting is on — the
default, and every bucket that existed before the setting — is kept as an asciicast v2 file (`.cast`, gzipped on
disk) attached to its call: it is in the call's `artifacts`, in
`listArtifacts`, downloadable with `artifactFile`, and goes with the
artifacts' retention. What is recorded is **what the screen
displayed**: if a session runs `cat .env`, the secrets it prints are in
the recording. What is typed and not echoed — a password at a `sudo` or
`passwd` prompt — is not: the keyboard leg is never copied. The header
carries the terminal (`TERM`, columns, rows) and the title `<node>
<caller>`; events are output (`o`, stdout, stderr, an attach's screen
and diffs) and resizes (`r`), timed from the session's opening. The
backend's `ended` audit line names the recording's artifact
(`recording`). Bounded: 64 MiB of recording per session, then a marker
(`m`) and nothing more; the recording never slows a session — a
recorder that falls behind marks a gap and stops there. Not recorded:
forwards (`-L`, raw TCP) and copies (`3pm fleet cp`, whose screen is
the file: `threepm_link::cp::is_cp_transfer`, below). The other way round, a recorded session never
goes agent to agent when the backend is the one recording it (a
session in clear): only with `record_shells` off may such a session's
legs go LAN direct (`nodeShell` `direct: true`, both nodes `direct:
all`); an end-to-end session (below) goes direct recorded or not. The
owner turns it off on the settings page ("Record shell sessions"), or
`setBucketSettings {record_shells: false}`; recordings already kept
stay until retention.

**End to end: the backend relays what it cannot read.** A 3pm of this
version asks `e2e: true` on every `nodeShell` and `nodeForward` (shell,
command, `cp`, attach, `-L`): the session's bytes are sealed between
the command line and the target's agent (Noise XX), and the backend
relays opaque records on the same `pipe` legs — it holds no key, does
not tee them, and a byte it changed ends the session. A node whose
live session did not announce `shell_e2e` (an older agent) is refused
at once, 409 `ERR_NODE_NO_E2E` naming its agent's version; the caller
may retry in clear with `--no-e2e`, and that session is the old one,
teed and recorded here. An up-to-date agent refuses every session in
clear: the call settles with the agent's `err`, whose code the stored
answer keeps as it came (`result.code`: `e2e_required`, or
`e2e_broken` when the channel broke) under the action code
(`forbidden` / `failed`) — the command line reads that code.

The recording of a sealed session is made **by the node**: the request
says `record: true` when the bucket records shells, the agent writes
the asciicast from what it seals, uploads it with `putCallArtifact`
(cap `RECORDING_MAX_BYTES`, not an action's) and names it in `exited
{recording}`; the backend claims it as the call's artifact, and the
`ended` line names that artifact. An agent that died mid-session sent
none: `ended` says `recording: lost`. Audit: `open_requested` and the
agent's `open` carry `e2e` and `cli_key` (the caller's key
fingerprint — which machine opened the session, not just which token).

| A compromised backend can… | in clear (`--no-e2e`) | end to end |
|---|---|---|
| read a session live (keys, screen, `cp` files, `-L` bytes) | yes | **no** |
| inject keystrokes, change a file in flight | yes | **no** (the session ends) |
| pose as the node | yes | **no**, except at the first contact (TOFU) |
| read the stored recording and audit lines | yes | yes (by choice) |
| see metadata, sizes, timing; refuse to relay | yes | yes |
| **open** a session with the fleet token it holds | yes | **yes** |

What "readable" covers: the recording is what the node wrote on the
screen leg. A copy (`3pm fleet cp`, up or down) is never recorded —
a download's screen is the file — by the backend nor the agent: both
recognise its fixed scripts byte for byte
(`threepm_link::cp::is_cp_transfer`: `sh -c <script> 3pm-cp <paths…>`),
and an argv one byte off is an ordinary command, recorded. Nothing is
hidden by it: the scripts only move bytes between the legs and the
named paths, and the audit keeps the command line, paths included.
`-L` is never recorded either.

The last line is the known limit: the node does not authenticate the
caller's key (`cli_key` is audit metadata), so a backend that holds a
fleet token can still open a session of its own. Out of scope.


### Who may open a session: the node, not this backend

Holding the bucket's account, its token, or a node's bearer is not
enough to open a session any more. Every `nodeShell`, `nodeForward` and
`nodeListen` call is sealed end to end, and the node's agent admits it
only when the caller's key — proven in the handshake — is in the node's
own `authorized_keys`, and only for exactly the request the caller
placed (the handshake binds argv, `--user`, `host:port`, the `-R`
connection, `bind:port`, `term`/size and `record`). This backend places
and relays calls; it cannot forge a caller key, and a request it
changes on the way gets no session (the handshake breaks, nothing runs,
nothing is dialed). A refused key settles the call `replied` with
`err.code` `forbidden` and `result.code` `key_not_authorized`, the
message naming the refused fingerprint; the `ended` audit line carries
it. `record` in `nodeShell`'s body (default `true`) is the caller's: it only
lowers `record_shells`, and the value in the answer is the one the
handshake binds. Proven against the real agent by
`tests/it/real_agent_access.rs` and `real_agent_exec.rs`.

## Blobs

**What you see.** A call's bytes are relayed, not stored. `PUT
/api/buckets/{id}/blobs/{blob}` writes one once; `GET` reads it as it
arrives — the headers come back before the first byte, so the receiving
end knows it is connected before the sender has begun. A file keeps 2
MiB in memory and touches the disk only while a reader is further behind
than that, and only for as long as it is; a stream never touches the
disk at all, and a reader that falls behind jumps to the head. The `PUT`
stops while the reader does not read (`--blob-window-mb`), `Range` works
once the blob is complete, and `?k=` — sixty seconds, one blob, read
only — is what lets an `<img>` tag read one. A node's share of its
bucket's blob quota is **half** of it: past that its own `putBlob` and
`nodeAction` read `429 ERR_QUOTA` while its neighbours keep their room,
and there is no flag for it. Bounds: `--blob-max-mb` per
blob, `--blob-quota-mb` per bucket and **half of it per node**, so one
node's transfers never leave its neighbours nothing, `--blob-disk-mb`
for the whole backend, and sixty-four live `?k=` tokens per bucket.
Reading a blob is behind the cross-site guard like every other read of
what a call produced: a page on another site cannot spend the
operator's session on it, and a `?k=` — which had to be minted and put
in the URL — goes through from anywhere, which is what a browser tag
needs. A blob the relay gave up on ends its readers' responses **in
error** rather than cleanly, so a viewer can tell a cut stream from a
finished one.

## Streams

**What you see.** `nodeAction` with `stream: true` opens a **session**
rather than a call: two `live` blobs, one each way, that carry framed
messages instead of bytes with a length. A message is at most a
megabyte, carries a sequence number and a timestamp, and says whether a
decoder can start on it (`KEY`), whether it is the decoder's parameters
(`INIT`) and whether it is the last (`END`).

The relay never slows a camera down. It keeps the last
`--live-window-ms` of the stream in memory and nothing on disk; when it
has to forget something it forgets whole messages, and a viewer that has
fallen out of the window is sent on to the newest `KEY` rather than
handed a hole. A viewer that joins in the middle gets the stream's
`INIT` in front of it, so H.264 is joinable at all, and no reader is
ever handed half a message. Sixteen viewers may read one leg at once.

While a session runs, the call carries `live` — `bytes_in`,
`bytes_out`, `dropped_out` and `since`, counted in memory, on `getCall`
and on the feed's `call` events. `dropped_out` is what a viewer that
cannot keep up costs and nothing else: what the window forgets before
anyone has connected is not a dropped frame, and is not counted as one.

**What one stream carries over a distance.** A `live` leg crosses this
server between two agents, so a frame makes two TCP hops. Measured under
`tc netem delay 30ms` — 60 ms of round trip per hop, an ordinary distance
between two continents — one stream carries **5 MB/s at the latency
floor**: a hundred 50 KiB images a second arrive with a p50 age of
**61.0 ms** and a p90 of 61.3, against a floor of 60, and ten of 1 485
are lost. Thirty a second: p50 61.2, p90 61.5, **nothing lost**. The tail
is p99 92 ms at a hundred and 80 at thirty, which is the same tail the
link had before and is not the buffer's.

Above that, what the kernel's send buffer allows is **up to about
43 MB/s where the path permits it** — the buffer Linux sizes for that
round trip (2 626 560 bytes here) divided by it, read off the socket and
never measured end to end; the highest measured is the 5 MB/s above,
which is all the camera offered. A real intercontinental path, with loss,
keeps CUBIC well under buffer ÷ round trip.

Until this lot, 3pm pinned that buffer at 256 KiB to keep a frame's
latency down, and the same stream capped at **4 MB/s**: the same hundred
images a second delivered 4 of an offered 5, p50 216 ms, with 18 % thrown
away by the window behind a queue that stood at 200. `TCP_NOTSENT_LOWAT`
keeps the latency now (a frame's p99 is 1.5 ms on a bare loopback,
unchanged) and nothing keeps the ceiling. On a platform without that
knob (macOS, the BSDs, Windows) the buffer stands, and so does the
4 MB/s.

**What it costs the kernel.** Not resident memory — `sk_wmem_queued`,
which counts against `net.ipv4.tcp_mem`. Auto-tuning grows a socket's
send buffer to roughly twice the bandwidth-delay product, so a 5 MB/s
stream at 60 ms settles near 600 KiB and only a much faster path
approaches the `tcp_wmem` ceiling (4 MiB on a stock Linux). This server
tunes **every** accepted socket, so the count is three per linked node
(one link session and the agent's two warm relay connections) plus one
per dashboard, SSE or API client. Worst case at the ceiling that is
12 MiB per agent, about 1.2 GiB for a hundred nodes; `tcp_mem` is what
stops it, by putting the whole stack under pressure rather than any one
socket. Size for the real figure, not the ceiling, and watch `tcp_mem`
rather than the process.

A session's leg is charged to its bucket's quota for what it holds — a
256 KiB floor plus the window, the `INIT` kept outside it and the
message being read in, in 256 KiB steps — not ten megabytes up front;
a bucket with no room left forgets the window's oldest messages first,
then refuses the write (429). A call that fails (an error, or no
outcome that went well) gives both legs back at once, short of a
finished answer its caller is still reading. It ends when either leg has been quiet
for a minute (a leg that has not started is excused while the other one
is alive: a handler loading a model for thirty seconds has not gone
away), or after twenty-four hours, whichever comes first. Once both
legs are over and nobody is still reading them, the session's blobs are
forgotten on the sweeper's next tick: their reservation is given back at
once (a `file` blob, an answer somebody may still fetch, keeps its 15
minutes; neither holds one of the node's open slots once settled), and the call's
`live` counters go with them.

A browser watches one with two requests and no JavaScript: `viewBlob`
mints a sixty-second URL for the leg, and `getBlob?as=mjpeg` serves it
as `multipart/x-mixed-replace` — one `image/jpeg` part per message,
with its length, behind a fresh sixteen-byte boundary, `nosniff` and
`Content-Security-Policy: sandbox; default-src 'none'`. A message that
is not a whole JPEG is dropped without the connection going with it, and
the type on every part is the relay's, never the application's. The
dashboard's call card shows the counters and puts that URL in an `<img>`.

**Where it comes from.** A blob exists because a call opened it, never
because a request asked for one, and membership of that call is the
whole of the authorisation: the side that placed the call writes the
input and reads the output, the side that runs it does the opposite, the
account that owns the bucket may do either, and everything else reads as
a blob that is not there. No second bearer, no signed ticket, nothing
with a lifetime of its own.

### Behind nginx

A relay proxied by nginx needs its buffering turned off, in both
directions, or a `PUT` of a gigabyte is held in a temporary file and a
`GET` arrives in bursts. The whole backend goes behind **one**
`location /`: the agent's link is a WebSocket under `/api/v1/`, the
blobs are under `/api/buckets/`, and the dashboard's feed is
`/api/buckets/{id}/events` — splitting them is what leaves one of the
three on nginx's defaults.

```nginx
# Outside `server`, once per configuration.
map $http_upgrade $connection_upgrade { default upgrade; "" ""; }

location / {
  proxy_pass http://127.0.0.1:9616; proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;
  proxy_set_header Host $http_host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_request_buffering off; proxy_buffering off;
  proxy_max_temp_file_size 0; client_max_body_size 0;
  client_body_timeout 1h; proxy_read_timeout 1h; proxy_send_timeout 1h;
  gzip on; gzip_types text/event-stream; gzip_proxied any;
  gzip_comp_level 5; gzip_min_length 0; gzip_vary on;
  access_log off;
}

# `server` context, not `location`: nginx refuses to start if this sits
# inside the block above. Four kilobytes is for the first byte out.
ssl_buffer_size 4k;
```

Four of those lines are not negotiable.

**`gzip` on `text/event-stream` only.** The feed is JSON repeated
every second — seven times smaller gzipped — and with
`proxy_buffering off` every event the backend writes is a flush, so
each still leaves at once (the window kept across events). Nothing
else is added to `gzip_types`: the JSON routes are gzipped by the
backend itself (nginx leaves a response with a `Content-Encoding`
alone), and a blob or an MJPEG stream must go out byte for byte.

**`Upgrade` / `Connection`.** `/api/v1/link` is a WebSocket, and it is
under the same `location` as everything else. The `proxy_set_header
Connection "";` that an ordinary upstream wants strips its
`Connection: Upgrade`, axum's upgrade extractor refuses the request,
and **no agent links at all**. The `map` above
sends `Connection: upgrade` on a request that asked to upgrade and
nothing on one that did not, which is both answers at once.

**`proxy_http_version 1.1`.** Without it nginx buffers every chunked
`PUT` whole, whatever `proxy_request_buffering off` says, because an
HTTP/1.0 upstream has no chunked encoding to forward.

**`client_body_timeout 1h`.** This is the timer the relay's back
pressure runs into, not `proxy_send_timeout`. With
`proxy_request_buffering off` nginx reads the client only as fast as the
upstream takes it, and the timer armed on the **client** side while the
upstream is not taking any is `client_body_timeout` (60 s by default).
The relay stops reading a `PUT` while the reader is more than
`--blob-window-mb` behind; it resumes the moment that reader comes back
within a chunk of the window, so an ordinary slow reader never holds it
for a minute. The one long stop is a blob **nobody has attached to
yet** — the writer runs one window ahead and waits — which is exactly
where the default would cut the upload. `proxy_send_timeout 1h` is
harmless and worth keeping for the `GET` side of a stream nobody feeds
for a while.

**`Host` and `X-Forwarded-For`.** `--trust-proxy` reads what nginx puts
there and nothing else: without these two the backend logs and rate
limits every node under the proxy's own address. `$http_host` and not
`$host`: the cross-site guard compares the authority of an `Origin`
against the `Host` it received, and `$host` is normalised **without the
port** — on a vhost anywhere but 443 that turns `Origin: https://h:8443`
against `Host: h` into a 403 on every guarded request. On 443 the two
agree, which is what makes it a trap rather than an outage.

A note on HTTP/2 **from the client**: the backend is HTTP/1.1 only, so
nginx always talks h1 upstream, but `proxy_request_buffering off` on an
h2 request from the client was ignored by nginx's v2 module for a long
time — the body went to a temporary file first, which is the failure
this block exists to prevent. It was fixed in **nginx 1.25.1**; below
that, a browser `fetch` PUT of a gigabyte is buffered whatever this
block says, and the agents' relays are h1 and never take that path. Ask
`nginx -v` before believing a measurement. `proxy_max_temp_file_size 0;`
does **not** guard against it — it governs only the buffering of an
upstream *response* to disk, and is already moot under `proxy_buffering
off` — but it is free, and it keeps a large `GET` out of the temporary
directory if `proxy_buffering` is ever turned back on.

**A shell needs the whole block.** `3pm fleet ssh` is two requests
that last as long as the session: a `PUT` that sends a keystroke now
and then and a `GET` that may say nothing for a while. Without
`proxy_http_version 1.1` and `proxy_request_buffering off` the
keystrokes never reach the node (nginx waits for the end of the body);
with `proxy_buffering on` the screen arrives in bursts. The command
line's twenty-second pings (and the agent's pongs) keep a quiet shell
inside nginx's default minute (`client_body_timeout`,
`proxy_read_timeout`, `proxy_send_timeout`), with one missed ping of
margin and no more: keep the block's hour, or a stalled ping cuts the
session (the relay then ends both legs). Check with `nginx -T` before
opening the route on a proxied backend.

`access_log off` is not optional either: a `viewBlob` token travels in
the URL (`?k=`), and an access log is a file of live credentials.

One node relaying a wall of cameras holds a socket per stream in each
direction, so give the host the file descriptors and the sockets to
match: `ulimit -n 65535` for the backend's unit (`LimitNOFILE=65535`),
and `sysctl -w net.core.somaxconn=1024`.

## The JavaScript client, generated from the route table

**What you see.** The backend serves the client for its own API:

```js
import { Client } from "/api/v1/client.js";

const api = new Client();          // same origin
const buckets = await api.listBuckets();
const feed = api.events(buckets[0].cluster_id, {
  init: (detail) => render(detail),
  node: (n) => upsert(n),
  telemetry: (t) => plot(t),
});
feed.close();
```

One ES module, no dependency, no build step: a method per endpoint, a
`Subscription` that reconnects on its own (and turns a `reset` into a
fresh `init`), and `ApiError {status, code, detail}` carrying the
backend's own `ERR_*`. `fetch` and `EventSource` are taken from the
global scope, or passed in. `docs/examples/fleet.mjs` is a whole
interface — print the fleet, then follow it — in fifty lines.

**This client targets browsers.** Nothing in it touches the DOM when
the module is imported, but `connect()` below draws, and the feed
prefers the browser's own `EventSource`. A Node program uses the SDK's
fleet client (`require("@3pm/api/fleet")`), generated from the same
route table.

Live data is the SSE feed, two ways: handlers (`api.events`) or a loop
— `for await (const [kind, payload] of api.follow(clusterId))` closes
the feed when you leave it (`docs/examples/follow.mjs`). The two routes
the API also lists for the agent, `enroll` (a `POST`) and `link` (the
WebSocket), are its protocol (`3pm link`), not a feed: `api.enroll()`
and `api.link()` throw an `AgentOnlyError` that says so and names
`events`, and the `LINK` constant carries the sub-protocol
(`3pm.link`) and where the protocol is written down.

`GET /api/v1/routes` is the API describing itself: every route with its
method, path, argument names, kind (`json`, `sse`, `agent-ws`,
`agent-http`, `static`), who may call it (`auth`: `public` or `user`, an account;
`node`: the scope a node's bearer needs, when one may), whether the
cross-site guard stands in front of it (`guarded`), its body and response
types, and a line of documentation — plus a `realtime` block that says
how live data is had (the SSE feed and its event names, the agent's
WebSockets and their sub-protocol, each pointing at its route by name),
where the client lives and a runnable snippet.

A route whose body is a small language of its own — `queryMetrics`,
`eventsQuery` — carries more than that line: `details` spells out every
field, what comes back, and a runnable example body, and the `limits`
block gives the numeric bounds by route name (range, points, series,
queries, formulas, filters, value lengths, body size). Both are
generated from the code that enforces them — the limits are the
validator's own constants, and a test parses each example through the
very validation the handler applies — so reading the descriptor is
enough to write a valid query, without discovering the bounds one 400
at a time. The one bound not there is the step floor, which depends on
how the backend was started: `store.interval_ms` of `GET /api/v1/info`,
and the block says so.

The document opens with its own price, `tokens: { claude, o200k }` —
`o200k` exact (OpenAI's `o200k_base`), `claude` estimated from the
bytes with a ratio measured per shape through Anthropic's
`count_tokens` API — counted over the exact bytes the route sends and
put first on purpose: a consumer with a context budget reads what the
document costs before reading the document. For an LLM the route
answers text by default: an index, one TOON-style row per route it can
use, recipes (a body for each common question — the value now, a
total over a period, the top N, a percentage, a rate, one host by
name, one label value, the error events — and how to read its answer),
then `metricsInventory`, `queryMetrics` and `eventsQuery` given whole
with their types — what a dashboard starts from (10 741 Claude tokens,
against 8 497 for the JSON table) — and
`?route=<name>` for one route in full — its details, its bounds, and
the TypeScript of its body and answer with every type they name. The
JSON table is `?format=json`, or what an `Accept: application/json`
gets, which is what the generated client sends. `?version=` defaults to `latest`; a backend serves
its own table or refuses, never a guess. `client.discover()` reads it
and follows that backend's paths from then on, so a page built against
one version keeps working against another — and says so when the
versions differ (`onVersionMismatch`, a console warning by default;
the client's own `VERSION` is exported). `client.call(name, args,
body, options)` drives any route by name, and `options.signal` on it
or on any generated method abandons a request a page no longer needs.

### Asking the viewer for a token

**What you see.** A page of your own, served from anywhere, that wants
to read a backend it has no session on:

```js
import { connect } from "https://fleet.example.com/api/v1/client.js";

const api = await connect({ baseUrl: "https://fleet.example.com" });
const buckets = await api.listBuckets();      // ready to call
```

`connect()` asks only when it has to. With a token already remembered
for this page it checks it against the backend and goes straight
through; otherwise a small card asks for one — "Log in", an
`3pmu_…` field, the same wording as the dashboard's own login. Enter
submits, Escape (or a click outside) dismisses and rejects with a
`TokenPromptCancelled` error.

The card lives in a **shadow root** of its own, so the page's CSS
cannot deform it and its own cannot leak out, and it follows the
viewer's light/dark preference.

**Nothing is stored until the backend has accepted it.** A token that
does not start with `3pmu_` is refused on the spot, with a line saying
so rather than a flat 401 later; anything else is spent on one
`listBuckets` call — the cheapest authenticated read, and one the route
table leaves unguarded, so it answers from another origin. Only a token
that came back 200 is written to `localStorage`, under
`3pm.token:<location.pathname>` (`connect({ namespace })` overrides the
key — one for a whole app, or one per environment). Every read and
write is wrapped: in a private window or with site data blocked,
nothing is stored and the viewer is simply asked again.

Revoked or rotated under the page? The 401 clears the stored token,
reopens the card, and — once a new one is accepted — **retries the call
that met the 401**, so the page recovers instead of freezing.
`api.forgetToken()` clears it on purpose.

**Where it comes from.** `connect()`, `promptForToken()`, `tokenKey()`
and `Client.forgetToken()` in `src/api/client.tmpl.js`, generated into
`assets/api.js` by `just api` like the rest. `new Client(…)` is
unchanged: the prompt is opt-in and nothing draws until it is called.

### From another origin: CORS is open

**What you see.** A page served from anywhere can call this backend —
there is no proxy to put in front of it any more. Every origin, every
method, and the request headers a preflight asks for, so `Authorization`
really is allowed (`Access-Control-Allow-Headers: *` would *not* have
covered it: the Fetch spec excludes it from the wildcard on purpose).
A preflight is answered for ten minutes, so a burst of calls does not
preflight twice per call. Response headers are exposed with `*`, so a
cross-origin reader sees more than the six safelisted ones.

```console
$ curl -si -X OPTIONS https://fleet.example.com/api/buckets/$B/metrics/catalog \
    -H 'Origin: https://somewhere.example' \
    -H 'Access-Control-Request-Method: GET' \
    -H 'Access-Control-Request-Headers: authorization, content-type'
HTTP/1.1 200 OK
vary: access-control-request-headers
access-control-allow-origin: *
access-control-allow-methods: *
access-control-allow-headers: authorization, content-type
access-control-max-age: 600
```

**Credentials do not travel, and that is the point.** There is no
`Access-Control-Allow-Credentials`, because a wildcard origin forbids
one — and the session cookie is `SameSite=Strict; HttpOnly` anyway, so
it never leaves this origin whatever we answer. A cross-origin caller
authenticates with a token it puts on the request on purpose:
`new Client({ baseUrl, token })`, or `Authorization: Bearer 3pmu_…`.

**What CORS does not open is the cross-site guard — and what opens the
guard is a bearer.** The guard is about a page elsewhere driving an
operator's session, a credential the browser attaches by itself. So a
request carrying another origin is still answered **403
`ERR_FORBIDDEN`** when it rides the session cookie or `Basic`, and on
`register` / `login` whatever it carries. A request that carries
`Authorization: Bearer 3pmu_…` is the opposite — the page holds the
token and put it there on purpose — and passes from any origin on every
route behind an account: a dashboard served elsewhere can run a custom
action (`nodeAction`), follow it (`getCall`), read its artifacts, and
make every other write its token allows; a bad token is a 401. The 403,
when it comes, is *readable*: it carries the CORS header, so the page
sees the status and the `ERR_FORBIDDEN` body instead of an opaque
network error it cannot tell from the backend being down.

**Where it comes from.** `src/api/surface/` declares the surface once
as data: `mod.rs` says what an entry is (`Endpoint`), `table.rs`
assembles the table from one file per resource under `table/`
(`meta`, `buckets`, `metrics`, `calls`, `blobs`, `pages`). The axum router mounts exactly that table, and `just api`
generates `assets/api.js`, `assets/api.d.ts` and `assets/api.json` from
it — `just check` fails when what is committed no longer matches.
`tests/it/api_surface.rs` holds the table to the running server: every
entry mounted on the method it declares, every guarded entry really
guarded, `/api/v1/routes` equal to the generated file, and every field
the API serializes declared in the types. Adding an endpoint is one
entry, one handler arm, and `just api`. There is deliberately no
OpenAPI: half the surface (the SSE feed, the two agent WebSockets) is
not describable in it.

## Accounts: keeping the API to yourself

**What you see.** Every bucket belongs to an account, and everything
under `/api/buckets` wants one. An account is a name and its tokens —
no password, no email. `/` without a session shows a login card: paste
a token, or create an account, whose token is shown once with a Copy
button (it is the only way back in). Logged in, the header shows the
name and **Log out**, and `/` lists the account's tokens under
"Account tokens": one per laptop or CI job (the secret shown once),
each revocable — the sessions it opened close with it, and the last one
stays (409 `ERR_LAST_TOKEN`).

- **Registration.** `--registration closed` (the default) lets the
  first account in and no other (403 `ERR_REGISTRATION_CLOSED`); that
  first account adopts every bucket created before accounts existed.
  `--registration open`, or `--dev`, lets anyone who reaches the port
  register.
- **A token** reads `3pmu_<id>_<secret>`: only its SHA-256 is stored
  (compared in constant time), the id is what lists and logs show. A
  script sends it on every request — `Authorization: Bearer 3pmu_…`, or
  `curl -u :3pmu_…` (Basic, the token as the password) — and the
  generated client takes it as `token`.
- **The dashboard** trades it for a session (`login`): a `3pm_session`
  cookie, `HttpOnly`, `SameSite=Strict`, 30 days sliding, `Secure` with
  `--cookie-secure` (set it behind HTTPS). The feed (`EventSource`)
  runs on it. Log out closes the session, not the token.
- **Another account's bucket** answers 404, like one that does not
  exist, so ids cannot be probed; no credential answers 401
  `ERR_UNAUTHORIZED` with the challenge. The pages, `/api/v1/info`,
  `/api/v1/routes`, `register`, `login` and the agent's own routes stay
  open — an agent authenticates with its node key, over the enrollment
  token at `enroll` and over the server's nonce at `link`.
- **Cross-site refusal, always on**, in front of every mutating route,
  `login` included. A request whose `Origin` (or `Sec-Fetch-Site`) names
  another site than its `Host` is refused 403 `ERR_FORBIDDEN` before
  anything else, so a page elsewhere cannot drive a logged-in browser.
  Only a credential the page had to hold opens it: a minted `?k=` on
  `getBlob`, an `Authorization: Bearer` on a route behind an account.

Beyond that: enrollments are budgeted per client address
(`--enroll-pending-per-ip`, then 429 `ERR_RATE_LIMITED`), and so are
link handshakes until they are over (`--link-pending-per-ip`, 64 by
default, judged before the database is read; then 503 `ERR_LINK_BUSY`,
also the answer when the handshake nonce store is full) — an IPv6
client counted by its /64. 503 and not 429: a busy backend is a retry
within the agent's 1–60 s transient backoff, never its 15-minute
rate-limit park, so a flood of upgrades without a key no longer holds
the honest agents away. A /48 can still fill the shared nonce store;
the agents then retry within a minute. Slow header
senders are dropped (`--header-timeout-ms`), agent frames are capped at
1 MiB before inflation, and a silent link is closed (`--silence-ms`).

**Where it comes from.** The table's `auth` column (`public` / `user`)
decides which routes sit behind the account layer. `auth/`: the token,
the cookie and the layer (`require_user`: who is calling, and whether a
route's `{id}` is a bucket that account owns). `db/users.rs`,
`http/api_auth.rs`, `http/guard.rs` (pure `cross_site`, table-tested),
`ws/budget.rs`, `server.rs`.

## What is redacted

**What you see.** The process table the backend stores and shows never
contains environment variable *values*, script paths or home paths —
the agent strips them before sending, and an exit event carries its
code and signal but not the stderr tail. Token secrets are never
stored, and `TokenView` / `NodeView` carry no key material. Log lines
arrive already redacted: the node replaces the secrets it recognises
before anything leaves (a pattern list; 3pm's `FEATURES.md`, "Log
shipping"), and the backend stores what it receives.

**Where it comes from.** `threepm-link`'s `redact` (in 3pm), and the
views in `http/views.rs`, whose tests assert no secret survives
serialization.