# 3pm

A fast, robust process manager, written in Rust. `3pm` keeps your apps alive:
it daemonizes them, restarts them when they crash, aggregates their logs, and
tracks the CPU/memory of their whole process tree — with the pm2 workflow
you already know.

```console
$ 3pm start app.js
[3PM] Spawning 3pm daemon with home=/Users/you/.3pm
[3PM] App [app] launched
┌────┬──────┬───────┬────────┬───┬────────┬──────┬────────┐
│ id ┆ name ┆ pid   ┆ uptime ┆ ↺ ┆ status ┆ cpu  ┆ mem    │
╞════╪══════╪═══════╪════════╪═══╪════════╪══════╪════════╡
│ 0  ┆ app  ┆ 4021  ┆ 0s     ┆ 0 ┆ online ┆ 0.0% ┆ 12.3mb │
└────┴──────┴───────┴────────┴───┴────────┴──────┴────────┘
```

## Install

```console
curl -fsSL https://3pm.edgecr4ft.net/install.sh | sh        # any platform; rerun to update
curl -fsSL https://releases.3pm.dev/setup-apt.sh | sudo sh   # Debian/Ubuntu
curl -fsSL https://releases.3pm.dev/setup-yum.sh | sudo sh   # RHEL/Fedora
```

Or grab a tarball / `.deb` / `.rpm` from the
[releases page](https://github.com/Unitech/3pm/releases), or build from
source: `cargo install --path crates/threepm-cli`.

## Commands

| Command | What it does |
|---|---|
| `3pm start <script>` | Start any executable; interpreter inferred from the extension (`.js`→node, `.py`→python3, `.rb`→ruby, `.sh`→bash, `.pl`→perl, `.ts`→node); a quoted command line (`3pm start "npm run dev"`) runs under `bash -c` |
| `3pm start ecosystem.toml` | Start every app declared in a config file (`.json`, `.yaml`, pm2's `ecosystem.config.js` too; `-` reads it on stdin; no target = the current directory's `ecosystem.*`); `--env <profile>`, `--only a,b` |
| `3pm apply ecosystem.toml` | Reconcile with the file, idempotently: create / update / restart / leave unchanged; `--prune`, `--dry-run` |
| `3pm validate ecosystem.toml` | Check the file (fields, profiles, dependencies) without starting anything; exit 1 on errors |
| `3pm start app.js -i 4` | Cluster mode: 4 instances of one app (`-i 0` = one per CPU); each gets `THREEPM_INSTANCE`/`NODE_APP_INSTANCE` |
| `3pm scale <name> <n>` | Resize an app to n instances without restarting the running ones |
| `3pm stop\|restart\|delete <name\|id\|all>…` | Lifecycle control, one or more targets at a time (`3pm delete 0 1 api`); `stop` returns once the process is actually dead; `restart --update-env` re-reads the app's config file |
| `3pm reset <name\|id\|all>…` | Zero the restart counters (the process keeps running) |
| `3pm sendSignal <SIG> <name\|id\|all>` | Send a signal to the running process (`--group` for its process group) |
| `3pm ls` | The process table (`list`/`status` aliases) + pm2's `host metrics` summary line (cpu, load, ram, gpu, NIC/disk throughput) from the host agent's cache; `--watch [--interval MS]` redraws it in place, where the cursor is; `--tree` (`-t`) draws each app's process tree under its row (`--all` unfolds it), `cpu`/`mem` being the tree's totals |
| `3pm show <name\|id>` | Detailed view (`describe`/`info` aliases), every process of the app listed (with PSS/footprint, threads and fds under Extended Monitoring) |
| `3pm attach <name\|id>` | The terminal of an app started with `--tty` (`console` alias): its screen, your keys, `Ctrl-Z` detaches; piped, stdin goes to the app and its output comes out; `--peek [--json]` prints the screen, `--read-only` watches, `--raw` relays the app's bytes, `--instance N` |
| `3pm logs [name\|id…]` | Last 30 lines (`--lines N`, `--rotated` to reach into `.1`/`.2.gz`) then realtime stream; `--err` `--out` `--raw` `--nostream` `--timestamp[=FMT]` `--format` (logfmt) `--highlight TEXT` `--size` |
| `3pm flush [name\|id]` | Empty log files in place (all apps + `3pm.log` without a target) |
| `3pm rotate [name\|id]` | Rotate log files now (`<file>` → `<file>.1`, …) |
| `3pm reloadLogs` | Reopen every log file by path — the hook for an external logrotate in create/move mode (`kill -USR2 <daemon pid>` does the same) |
| `3pm logrotate [--print]` | Install (root) or print a logrotate(8) `copytruncate` drop-in, for hosts that centralize rotation |
| `3pm save` / `3pm resurrect` | Persist / restore the process list across daemon restarts |
| `3pm startup` / `3pm unstartup` | Install / remove a boot service so the saved apps come back after a reboot (prints a `sudo` command to run) |
| `3pm internals` | Daemon self-monitoring: memory/CPU, host load, fds vs limit, per-status counts, bus subscribers, log & dump gauges (`report` alias) |
| `3pm hostmetrics` | Host-wide OS metrics, sampled by a dedicated agent process (cheap gauges every second, walks/forks every 10s, NTP every 5min) and served from its cache: CPU (per-core, per-state incl. steal), PSI pressure, memory/swap + page cache + overcommit, OOM kills, cgroup limits, load, disks + inodes, RAID health, block-device I/O, network, TCP states & health (listen drops), sockets, conntrack, file handles, sensors, GPUs (utilization, VRAM/unified memory, temp, power), memory compressor (macOS), NTP clock offset, the top processes of the host by CPU, memory and network; `--extended`: the kernel's own counters, family by family (Extended Monitoring, Linux) |
| `3pm kill` | Stop the daemon and every managed process |
| `3pm update` | Upgrade the resident daemon in place (save → kill → respawn → resurrect) |
| `3pm completions <shell>` | bash/zsh/fish completions |
| `3pm serve [dir\|page.html] [port]` | Serve a directory (or one HTML page, opened in your browser) over HTTP as a managed app (pm2 `serve`): free port picked, local + LAN links and a QR code for your phone; `--spa`, `--ftp` listing, `--basic-auth-username/-password`, `--host`, `--homepage`, `--name` — a bundled module, see `docs/modules.md` |
| `3pm install <module\|dir>` / `3pm uninstall <module>` / `3pm module list` | Modules: optional features bundled with the install (`serve`, `agent`, `autoupdate`, `lan` — a virtual Ethernet between the root nodes of a bucket, with an optional network namespace holding only it for programs that bind to the default route's interface, which is its trust boundary: a member can claim another's node id until the daemon passes a verified caller); `install` copies one into `~/.3pm/modules`, starts its apps (shown under **Modules** in `ls`, shielded from `stop\|restart\|delete all`) and saves the list |

`3pm start` options: `--name`, `--interpreter` (`none`: run the script
itself), `--interpreter-args "--inspect"` (`--node-args`), `--cwd`, `--env KEY=VALUE`
(repeatable), `--namespace`, `--restart-policy always|on-failure|never`
(`--no-autorestart` = `never`), `--stop-exit-codes 0,3`, `--start-delay MS`,
`--depends-on APP` (repeatable; the app is parked until APP is online),
`--cron-restart "0 2 * * *"`, `--max-memory-restart 200M`,
`--kill-signal SIGTERM`, `--wait-ready`, `--shutdown-with-message`,
`--watch` / `--watch=PATH` (+ `--ignore-watch GLOB`, `--watch-delay MS`),
log destinations (`-o/--output PATH`, `-e/--error PATH`, `-l/--log PATH`,
`--merge-logs`, `--log-date-format FMT` / `--time`, `--disable-logs` — see
[Logs](#logs)), `--tty` (run on a pseudo-terminal, for interactive
programs: `3pm attach` connects to it), and `-- args…` passed to the app.

A bare word that names a managed app (or its id, or `all`) restarts that
app with its configuration, as pm2 does (`3pm stop api && 3pm start api`).
Otherwise the target is a file (relative to the cwd) or, when there is no such file
and it has no `/`, a command on your PATH: `3pm start npm -- run dev`. A
target with a blank that is not a file is a command line for `bash -c`,
named after its words (`3pm start "npm run dev"` → `npm-run-dev`). A
config file is recognized by its name (`.toml`, `.json`, `.yaml`/`.yml`,
`.config.js|cjs|mjs`); `-` reads the config on stdin, and no target at
all uses the current directory's `ecosystem.*`.

## Supervision behavior

- **Auto-restart**: crashed apps respawn at once (pm2 default); a crash
  loop reaches `errored` after 16 quick exits. `restart_delay` adds a
  fixed pause; `exp_backoff_restart_delay` (config or
  `--exp-backoff-restart-delay`) switches on exponential backoff
  (×1.5 per crash, capped at 15s, reset after 30s of stable uptime).
- **Restart policy**: `always` (default, pm2 behavior), `on_failure`
  (respawn only on a non-zero exit or signal death — a clean `exit 0`
  parks the app in `stopped`), or `never`. User `stop`/`restart` and the
  memory guard are never gated by the policy.
- **Crash-loop protection**: 16 consecutive exits below `min_uptime` (1s)
  mark the app `errored` and stop the loop; `3pm restart` re-arms it.
- **Stop sequence**: `SIGINT` (configurable per app) to the process group →
  1.6s (`kill_timeout`) → `SIGKILL`. When the incarnation's leader exits, the
  rest of its process group is swept — background children die with it.
- **Memory guard**: `--max-memory-restart 200M` gracefully cycles the app
  when the RSS of its whole process tree crosses the limit
  (`limit_scope = "leader"` judges the spawned pid alone).
- **Watch mode**: `--watch` restarts the app when files change under its
  cwd (or under `--watch=PATH`, repeatable). OS events (inotify / FSEvents
  / kqueue) with a trailing debounce (`--watch-delay`, floor 100ms);
  dotfiles and `node_modules` are always ignored, `--ignore-watch GLOB`
  adds to that list. Falls back to polling — and says so in `ls`/`show` —
  when the native watcher can't be set up. `3pm restart <app> --watch`
  toggles it, like pm2.
- **Orphan reconciliation**: if the daemon is killed uncleanly (`kill -9`),
  surviving app processes are identified by pid + start time on the next
  start/resurrect and reclaimed — never double-started. Until then
  `3pm ls` names them (`1 process from a previous daemon still running
  unmanaged: api (pid 4321)`) so a taken port is never a mystery.
- Every timing knob (`min_uptime`, `kill_timeout`, `restart_delay`,
  `exp_backoff_restart_delay`, `backoff_*`, `max_restarts`) is
  configurable per app.

## Graceful start & stop (control channel)

Apps can talk back to 3pm over a Unix socket — any language, four lines
of code ([examples/hello-ready.py](examples/hello-ready.py)):

```python
s = socket.socket(socket.AF_UNIX); s.connect(os.environ["THREEPM_CTL_SOCK"])
s.sendall((json.dumps({"msg": "hello", "id": int(os.environ["THREEPM_ID"]),
                       "gen": int(os.environ["THREEPM_GEN"])}) + "\n").encode())
s.sendall(b'{"msg":"ready"}\n')            # done booting — mark me online
```

- `--wait-ready`: the app stays `launching` until it sends `ready`
  (`listen_timeout`, default 3s, forces it online with a warning —
  pm2 semantics). `3pm start` returns when the app is actually ready.
- `--shutdown-with-message`: a graceful stop arrives as a
  `{"msg":"shutdown"}` line on the socket instead of a signal; clean up
  and exit (SIGKILL still follows after `kill_timeout` if you don't).

Node apps get pm2 parity for free later via a shim; everything is opt-in —
apps that never connect are supervised with plain signals.

## Boot persistence

Surviving a reboot needs the OS init system — nothing in userland restarts
itself after a power cycle. `3pm startup` prints the privileged command to
register a boot service (via the [`service-manager`](https://crates.io/crates/service-manager)
crate — launchd, systemd, OpenRC, rc.d):

```console
$ 3pm startup
[3PM] Init system: launchd
[3PM] To set up the startup script, copy/paste the following command:
sudo env PATH=$PATH THREEPM_HOME=/Users/you/.3pm /usr/local/bin/3pm startup launchd -u you --hp /Users/you
```

Run that once (it's the only step needing root); the service is system-level
but runs the daemon **as you** — it runs `3pm __daemon --boot`, which restores
whatever `3pm save` last recorded. The init system supervises the real daemon
and restarts it on crash. `3pm unstartup` prints the matching removal command.

## Containers

`3pm runtime` is the entrypoint: daemon in the foreground as PID 1, app
output on stdout/stderr, SIGTERM drains, and it exits `2` once nothing is
left to supervise so the orchestrator's restart policy takes over
(`--no-auto-exit` to stay up). Every `start` flag applies.

```dockerfile
CMD ["3pm", "runtime", "ecosystem.toml", "--env", "production"]
```

The apps run in a real daemon, so `docker exec c 3pm ls` / `logs` /
`events` work while it runs. See `examples/Dockerfile`.

## Ecosystem file

```toml
[defaults]                   # optional: every app starts from these
restart_policy = "on_failure"

[[apps]]
name = "api"
script = "./server.js"
args = ["--port", "3000"]
max_memory_restart = "300M"
max_cpu_percent = 80         # restart when a CPU sample exceeds this
limit_scope = "tree"         # what both limits judge: the whole process tree, or "leader"
cgroup_enforce = false       # Linux: make both limits hard (cgroup v2)
deny_gpu = false             # hide GPUs via CUDA/ROCm env vars
restart_policy = "on_failure"  # always (default) | on_failure | never
watch = true                 # or "src", or ["src", "config"]
ignore_watch = ["*.log"]     # added to the built-in dotfiles + node_modules
watch_delay = 500            # ms
stop_exit_codes = [0, 3]     # these exits park the app instead of restarting it
depends_on = ["worker"]      # start after every worker instance is online
start_delay = 1000           # ms before the first spawn
cron_restart = "0 2 * * *"   # restart nightly at 02:00 (5 or 6 fields)
log_date_format = "%Y-%m-%d %H:%M:%S"   # persisted prefix; none by default
log_rotate_size = "50M"      # per-app override of the daemon policy

[apps.env]
NODE_ENV = "development"
[apps.env_production]        # applied with --env production (pm2)
NODE_ENV = "production"
[apps.profiles.staging]      # any field, applied with --env staging
instances = 2

[[apps]]
name = "worker"
script = "./worker.py"
instances = 4
out_file = "/var/log/worker/{name}-{instance}.out"  # placeholders: {name} {instance} {namespace}
error_file = "NULL"                                # no file for stderr (the stream still flows)
merge_logs = false
```

Relative paths resolve against the config file's directory. Unknown fields
warn with a did-you-mean instead of failing (`3pm validate` makes them
errors). pm2's aliases (`out`, `error`, `err_file`, `log`, `combine_logs`,
`exec`/`cmd`/`command`, `exec_interpreter`, `node_args`, …) are accepted.
The same file can be written in JSON or YAML, or as pm2's
`ecosystem.config.js` (evaluated by your `node`, in your shell's
environment); see `docs/ECOSYSTEM.md`. `3pm start eco.toml` always
restarts (pm2); `3pm apply eco.toml` is the idempotent form — unchanged
apps keep running.

## Logs

Each instance writes `logs/<app>-out.log` and `logs/<app>-error.log`
(`<app>-<i>-…` for instances ≥ 1; `<app>.log` when merged). The file holds
the app's bytes verbatim; with `log_date_format` every persisted line is
prefixed `"<strftime>: "`. Custom destinations (`out_file`, `error_file`,
`log_file`) accept `{name}`, `{instance}`, `{namespace}`; a custom path
shared by several instances is infixed `-<instance>` automatically so two
instances never interleave in one file. `NULL` / `/dev/null` disables a
stream's file (it still streams), `disable_logs` disables everything.
An app on a terminal (`tty`) has one stream: one file (`<app>.log`),
stripped of escape sequences so it reads as text.

**Rotation is built in and hot**: the daemon counts bytes as it writes and
rotates the moment a file crosses the threshold — no restart needed, no
line lost or torn. Policy, most specific wins:

| Layer | Where |
|---|---|
| per app | `log_rotate_size` / `_files` / `_days` / `_compress` in the ecosystem file |
| environment of the daemon | `THREEPM_LOG_MAX_SIZE` (`"0"` = off), `THREEPM_LOG_MAX_FILES`, `THREEPM_LOG_MAX_DAYS`, `THREEPM_LOG_COMPRESS` |
| `~/.3pm/config.toml` | `[logs] rotate_size = "20M"`, `rotate_files = 5`, `rotate_days = 14`, `rotate_compress = false` |
| defaults | 20 MiB, 5 generations, 14 days, no gzip |

Generations are `<file>.1` (freshest) … `<file>.N`; with `compress`, `.2`
and older are gzipped (`.1` stays plain so it tails). An hourly sweep
prunes by count and age even for files that never rotate. `3pm internals`
shows the effective policy, open files and rotations; `3pm logs --size`
and `3pm show` show per-file sizes.

Reading: `3pm logs` tails then follows from the event bus (subscribe-first,
so a rotation or a chatty app never opens a gap). `--timestamp` stamps
streamed lines with their arrival time (the wire carries `ts`, epoch ms),
`--format` emits logfmt, `--json` emits the wire events — history and
stream in one schema.

## Architecture

```
3pm CLI ──rpc.sock (NDJSON)──▶ 3pm daemon ──▶ supervised processes
   ▲                              │
   └────pub.sock (event bus)◀─────┘   log lines + lifecycle events
```

One binary. The first `3pm` command spawns the daemon (detached, own
session); a `flock` guarantees exactly one daemon per home even under
concurrent cold starts. State lives under `~/.3pm` (`THREEPM_HOME`
overridable): sockets, `3pm.log`, `logs/<app>-{out,error}.log` (+ rotated
`.N`/`.N.gz`), `pids/`, `dump.json`, optional `config.toml`.

The workspace is split so the supervision logic stays pure and provable:

| Crate | Role |
|---|---|
| `threepm-core` | Sans-io domain: the lifecycle policy state machine, backoff, paths — no tokio, no I/O |
| `threepm-protocol` | Wire types, capped NDJSON codec, client (connect / auto-spawn / subscribe) |
| `threepm-daemon` | Runtime: supervisor actor, per-child tasks, pumps, RPC server, event bus, metrics |
| `threepm-cli` | The `3pm` binary: commands + pure renderers |
| `threepm-testsupport` | E2E harness (isolated homes, leak-proof daemon cleanup) |

Design notes worth knowing:

- All lifecycle decisions run through a pure `step(state, input, now) →
  actions` function — deterministically unit- and property-tested (the
  actor never mutates policy state itself).
- Signaling and reaping are serialized in one task per child, so a signal
  can never hit a reaped (possibly reused) pid.
- Log pumps are byte-oriented with a 64 KiB cap: binary output, `\r`-only
  progress bars, or invalid UTF-8 can't wedge logging or the daemon.
- Realtime `3pm logs` streams from the event bus (subscribe-first, then
  tail), not from file watching.

## Divergences from pm2 (deliberate)

- **Group-kill, not treekill**: signals go to the process group; a child
  that `setsid()`s itself escapes (pm2's ps-walk has different blind
  spots) — it still counts in the app's figures and `3pm ls --tree`
  marks it `⚠`. Wrap-scripts should `exec` the real binary.
- **Crash-loop accounting**: consecutive-quick-exit counting (reset by one
  stable run) instead of pm2's wall-clock window — the window breaks as
  soon as `restart_delay` or a backoff paces the restarts.
- Sockets are `0600` and peer-uid-checked (pm2 uses group-writable `775`).
- **Watch mode is derived from state**: one watcher per app (not per
  instance), armed while the app runs; `stop` turns watch off as in
  pm2, so the list never says "enabled" with no watcher. `ignore_watch`
  adds to the built-in ignores instead of replacing them, `watch_delay`
  is milliseconds everywhere, and a failed native watcher is reported
  (`watch_error`) instead of logged and forgotten.

## Development

Recipes live in the [`justfile`](justfile) ([just](https://github.com/casey/just),
`brew install just`):

```console
just             # list recipes
just test        # fast tests: unit, property, component (paused-time), snapshot
just e2e         # real binary against real daemons in isolated /tmp homes
just ci          # everything CI gates: fmt, clippy, snapshots, tests, e2e
just cov         # coverage report (cargo-llvm-cov; spawned daemons included)
```

Without `just`, the equivalent raw commands:

```console
cargo nextest run --workspace          # the whole suite
cargo clippy --workspace --all-targets -- -D warnings
cargo insta review                     # snapshot changes
```

The e2e suite runs the real binary against real daemons in isolated
`/tmp` homes; every test cleans up its daemon even on panic, and CI sweeps
stragglers by `--test-marker`.

## Documentation

- [Extensibility & pm2 Feature Map](docs/extensibility.md) — the extension
  seams (commands, spec fields, lifecycle triggers, control messages) and
  how the remaining pm2 surface maps onto them.
- [Metrics Reference](docs/metrics.md) — every metric `ls`/`show`/
  `internals`/`hostmetrics` report, the process tree behind an app's
  figures, per-platform sources, retrieval model.
- [Application Metrics SDKs](docs/app-metrics.md) — `io` for Node/Electron
  (`@3pm/api`), Python (`threepm-io`) and C++ (`threepm_io.hpp`): custom
  metrics, histograms, events, `notify_error`.
- [Host Metrics Agent](docs/hostagent.md) — the separate process behind
  `3pm hostmetrics`: cache, lifecycle, failure semantics, knobs, files;
  [Sampling Tiers](docs/hostagent-tiers.md) — the collector cost audit
  behind its fast / medium / slow periods.
- [Runtime Cheatsheet](docs/RUNTIME.md) — `3pm runtime` for containers:
  invocations, exit codes, Dockerfile shape, rules of thumb.
- [Ecosystem File Cheatsheet](docs/ECOSYSTEM.md) — every field with its
  default and unit, layering rules, `validate` / `apply` workflow.
- [Serve Cheatsheet](docs/SERVE.md) — `3pm serve`: invocations, flags,
  responses, attached mode, pm2 ecosystem compatibility.
- [Modules](docs/modules.md) — `3pm install` / `uninstall` / `module
  list`, `3pm <module> …`, the `module.toml` contract, writing a module.
- [Fleet link](docs/backend-agent/README.md) — `3pm agent` and
  `3pm-backend`: the wire contract, the shared crate, the agent module,
  the backend API, status and caveats.
- [Logs Reference](docs/logs.md) — capture, destinations, line format,
  hot rotation and its policy layers, `logs`/`flush`/`rotate`/`reloadLogs`.
- [Download site](docs/SITE.md) — https://3pm.edgecr4ft.net: what is
  online, the three `just publish*` commands that upload it, the landing
  page, the generated documentation and `llms.txt`, the nginx vhost.
- [Resource Limits Reference](docs/resources.md) — `max_memory_restart`,
  `max_cpu_percent`, `limit_scope`, `cgroup_enforce` (Linux cgroup v2),
  `deny_gpu`: cheatsheet, semantics, permissions, soft vs hard.

Six contributor-facing documents left the tree on 2026-09-09 and live in
OpenViking under `viking://resources/3pm-docs-archived/`: `architecture`
(internals: the policy state machine, daemon lifecycle, race hardening —
the §numbers quoted across this codebase are its), `development` (task
runner, test layers, CI map), `coverage`, `releasing`, `nix-testing`, and
`backend-agent/threepm-link`.

## License

MIT