# Logs Reference ## Cheatsheet ```console # read 3pm logs # all apps: last 30 lines, then follow 3pm logs api --lines 200 --rotated # reach into api-out.log.1, .2.gz… 3pm logs api --err --timestamp # stderr only, stamped on arrival 3pm logs api --timestamp='%H:%M:%S' # custom strftime 3pm logs api --format # logfmt: ts= app= id= instance= type= message= 3pm logs api --json --lines 0 # wire events only (NDJSON), no history 3pm logs api --highlight ERROR # bright-black background on matches 3pm logs api --raw --nostream # bare lines, tail and exit 3pm logs --size # file sizes + rotated generations 3pm show api # paths, sizes, date format, rotation policy # write-side setup 3pm start app.js --time # file lines prefixed "%Y-%m-%dT%H:%M:%S: " 3pm start app.js --log-date-format '%H:%M:%S' 3pm start app.js -l /var/log/app.log # one merged file (stdout + stderr) 3pm start app.js --merge-logs # merged, default path: logs/app.log 3pm start app.js -o /var/log/app.out -e NULL # custom stdout, no stderr file 3pm start w.js -i 4 -o '/var/log/{name}-{instance}.out' # {name} {instance} {namespace} 3pm start app.js --disable-logs # no file, no stream, no crash tail # operate 3pm flush # empty every log file + 3pm.log, in place 3pm flush api # one app 3pm rotate api # rotate now (→ .1), regardless of size 3pm reloadLogs # reopen by path after an external mv kill -USR2 $(cat ~/.3pm/3pm.pid) # same as reloadLogs (daemon log included) 3pm delete api --flush # delete the app AND its log files 3pm logrotate --print # logrotate(8) copytruncate drop-in sudo 3pm logrotate --user $USER # install it in /etc/logrotate.d 3pm internals # effective policy, open files, rotations, bytes # rotation policy (per app > env > config.toml > defaults 20M / 5 / 14d / no gzip) THREEPM_LOG_MAX_SIZE=50M THREEPM_LOG_MAX_FILES=10 THREEPM_LOG_COMPRESS=1 3pm ls # env → daemon THREEPM_LOG_MAX_SIZE=0 ... # rotation off printf '[logs]\nrotate_size = "50M"\nrotate_days = 7\n' > ~/.3pm/config.toml ``` ```toml # ecosystem.toml, per app [[apps]] name = "api" script = "./server.js" log_date_format = "%Y-%m-%d %H:%M:%S" merge_logs = false # or log_file = "/var/log/api.log" out_file = "/var/log/{name}-{instance}.out" error_file = "NULL" log_rotate_size = "50M" # "0" = off log_rotate_files = 10 log_rotate_days = 7 log_rotate_compress = true ``` Daemon restart needed for env/config.toml changes (`3pm update`). --- Everything about how 3pm captures, stores, rotates and serves application output. Mechanism rationale lives in OV `3pm-docs-archived/architecture` (§6.4 log pumps, §7 logs pipeline); this is the field-level reference. User-facing walkthrough: `../FEATURES.md` ("Logs"). ## 1. Capture Each managed process is spawned with `stdin` null and `stdout`/`stderr` piped. Two pumps per incarnation (`child_task::pump`) read the pipes: | Property | Value | Why | |---|---|---| | Line reader | `read_until(b'\n')` capped at `MAX_LOG_LINE` = 64 KiB | a child streaming GBs without a newline must not grow the daemon | | File content | the child's bytes, verbatim | pm2 behavior; no UTF-8 conversion on the persistence path | | Bus content | UTF-8 lossy, trailing `\n`/`\r\n` stripped | readers get text | | Cap hit mid-line | `\n` appended in the file | the file stays line-structured | | Invalid UTF-8 / write error | never kills the pump | a wedged pump stops logs *and* eventually blocks the child on a full pipe | | Write errors | logged once, not per line | a full disk must not flood `3pm.log` at the app's output rate | | Crash tail | stderr pump feeds a 30-line / 4 KiB ring | attached to the `exit` event and to `ProcessRecord.stderr_tail` | Under `disable_logs` the pumps still drain the pipes but write nothing, emit nothing and keep no crash tail. ### An app on a terminal (`tty = true`) A `tty` app runs on a pseudo-terminal: **one** output stream (the pty master), so **one** pump (`child_task_tty.rs`) and no stderr. Each read (≤ 16 KiB) feeds the daemon's screen of the app first (what `3pm attach` shows), then the logs: | Property | Value | Why | |---|---|---| | Destination | one file: `.log` / `-.log` (the merged name), or the `log_file`/`out_file` given; `err_log` is `""` | a terminal has no second stream; `error_file` with `tty` is refused by `check_fields` (`apply`, `validate`, the daemon) | | File, bus, crash tail content | the output **stripped of escape sequences** (`threepm-core::ansi::Stripper`, streaming: a sequence cut by a read is still recognised) | a TUI's bytes are cursor moves and colours; the log must read as text | | Stripped | CSI, OSC/DCS/APC/PM/SOS strings, short escapes, C0 controls but `\n` and `\t`, DEL; `\r\n` → `\n`, a lone `\r` dropped | | | Horizontal gaps | a move right on the line (`CSI n C`, `CSI n G`, ``CSI n ` ``) after text becomes one space | TUIs draw word gaps that way (`Accessing\e[12Gworkspace` logs `Accessing workspace`) | | Line cap | `MAX_LOG_LINE`, as for pipes | | | Crash tail | the last lines of the one stream (`3pm show`: "last output before exit") | there is no stderr | | `disable_logs` | no file, no bus line, no crash tail — the screen is still fed | the exit for a TUI that redraws too much to be worth a log | | Keystrokes | never logged (what the app echoes is output, like any app's) | | Across `3pm update` the pump stops at a read boundary: the partial stripped line is delivered first by the next pump, and what the read ended inside of (an escape sequence, a UTF-8 character) is replayed to the next screen and stripper, so neither is logged twice nor as text. ## 2. Destinations Resolved by the pure `Paths::log_targets(spec, instance)` (`threepm-core/paths.rs`) on every spawn; the daemon re-targets only when the result changed (a re-spec takes effect on its restart, the old file is left alone). | Situation | stdout | stderr | |---|---|---| | default, instance 0 | `/logs/-out.log` | `/logs/-error.log` | | default, instance i ≥ 1 | `--out.log` | `--error.log` | | `merge_logs` | `.log` / `-.log` (one file) | same file | | `log_file = P` | P (implies `merge_logs`) | P | | `out_file` / `error_file` | as given | as given | | `"NULL"` / `"/dev/null"` | no file (still on the bus) | no file | | `disable_logs` | nothing | nothing | | `tty` | `.log` / `-.log`, or `log_file`/`out_file` | none (one stream; `error_file` refused) | Custom paths: - Relative paths resolve against the app's `cwd` (ecosystem file: the config's directory); `~/` expands. The daemon creates parent directories. - Placeholders: `{name}`, `{instance}`, `{namespace}` (`default` when the app has none). - **Multi-instance guard.** A custom path on an app with `instances != 1` that does not contain `{instance}` is infixed `-` before its extension for instances ≥ 1 (`/var/log/app.log` → `/var/log/app-1.log`); instance 0 keeps the path as written. `3pm start` warns once. Property test: no two instances of one app ever resolve to the same file. - Two *different* apps pointed at one file share it: one handle, writes serialized, one rotation (the first opener's policy wins; the daemon logs a warning when policies differ). `ProcessRecord.out_log` / `err_log` carry the resolved paths (`""` = no file; equal when merged). `3pm show` and `3pm logs --size` add on-disk sizes. ### An app that runs as another account (`user`) A root daemon that hosts users (`[users]` below) writes an app's files in the **home of the account the app runs as** — files follow `spec.user`, visibility follows `spec.owner`: ``` /var/lib/3pm/users/srt/logs/[-]-out.log[.N[.gz]] root:srt 0640 /var/lib/3pm/users/srt/pids/[-].pid root:srt 0640 ``` The mode and group are set **explicitly** (not left to the umask), on the live file and on every generation a rotation produces — `.N` and `.gz` alike — and again after a `flush` truncates in place. So root keeps writing and the account (through its private primary group, and nobody else) keeps reading: `3pm logs` of that user tails its own files directly, and the home itself is `root: 0750`, so it cannot replace them. An app a hosted user starts may **not** choose its log paths: `out_file`, `error_file` and `log_file` are refused (`"NULL"` to switch a stream off still works). A path of its choosing would have root open — and `delete --flush` unlink — any file on the machine. ## 3. Line format - **File**: raw bytes. With `log_date_format`, each line is prefixed `": "` in local time (pm2 shape). The format is validated at `3pm start`; unknown specifiers fail there. `--time` on the CLI is shorthand for `%Y-%m-%dT%H:%M:%S`. There is no CLI-only default. - **Bus** (`pub.sock`, NDJSON): the bare line plus metadata. Events are additive — older peers ignore the extra fields. ```json {"event":"log","id":1,"name":"api","stream":"out","line":"hello","instance":0,"ts":1787669351920} {"event":"log","id":1,"name":"api","stream":"err","line":"boom","instance":0,"ts":1787669351933} {"event":"log_rotated","id":1,"name":"api","instance":0,"path":"/home/u/.3pm/logs/api-out.log"} {"event":"lagged","missed":12} {"event":"daemon_shutdown","reason":"kill"} ``` `ts` is the daemon's receive time, ms since the Unix epoch, and it is always there: a history line the CLI replays from a file carries the **live** file's last write, the closest thing this side knows. Every line of one tail therefore shares a stamp, and with `--rotated` the lines read from older generations share it too — a rotated file's own mtime is when it was rotated, not when its lines were written, so it would be no truer. ## 4. Rotation Built in, **hot**: the sink (`daemon/logsink.rs`) counts bytes as it writes; the write that crosses `max_size` lands, then — still under the sink's lock — the generations are shifted, the live file becomes `.1` and a fresh live file is opened. Every writer shares the same handle, so all of them follow. No restart, no lost or torn line. Layout: `` live, `.1` freshest generation … `.N` oldest, `.gz` suffix when compressed. Planning is pure (`threepm-core/logrotate.rs`): - remove generations that would exceed `max_files`, shift `.N → .N+1` oldest-first (never clobbers), move live to `.1`; - `max_files = 0`: discard everything, truncate the live file in place; - `compress`: gzip `.2` and older (`.1` stays plain so it can be tailed). Runs on a blocking thread but under the sink lock — released, a concurrent rotation could rename `.N` mid-read and the wrong plain file would be deleted. Output is written to `.gz.tmp` and renamed; - age pruning (`max_age_days`) piggybacks on each rotation and on an hourly housekeeping tick (`LOG_HOUSEKEEPING_INTERVAL`), so retention holds for files that rarely rotate. ### Policy Most specific wins, field by field: | Layer | Keys | |---|---| | per app (ecosystem / `SpecPatch`) | `log_rotate_size` (human: `"10M"`, `"0"` = off), `log_rotate_files`, `log_rotate_days`, `log_rotate_compress` | | daemon environment | `THREEPM_LOG_MAX_SIZE`, `THREEPM_LOG_MAX_FILES`, `THREEPM_LOG_MAX_DAYS`, `THREEPM_LOG_COMPRESS` (`1/true/yes/on`) | | `/config.toml` | `[logs] rotate_size`, `rotate_files`, `rotate_days`, `rotate_compress` | | defaults | `20M`, `5`, `14`, `false` | The daemon resolves its policy once at startup (`logsink::daemon_policy`) and logs it; invalid values are logged and skipped, never fatal. The effective daemon policy is in `3pm internals` (`log_rotate_*`), a per-app override in `3pm show` ("log rotation"). ### Rotation vs. external logrotate(8) Both work. `3pm logrotate --print` renders a `copytruncate` drop-in (`sudo 3pm logrotate` installs it in `/etc/logrotate.d/3pm-`). For `create`/move mode, add `postrotate kill -USR2 $(cat /3pm.pid)` or run `3pm reloadLogs`: every open log (and the daemon's own `3pm.log`) is reopened by path. ## 5. Commands | Command | Effect | |---|---| | `3pm logs [target…]` | tail then follow (see §6); several targets are a union, a name matching nothing is a notice | | `3pm logs --size [target]` | per-file sizes and rotated generations; `--json` = `[{id,name,instance,stream,path,bytes,generations:[{path,bytes,compressed}]}]` | | `3pm flush [target]` | truncate live files in place; no target = every app **and** `3pm.log`. Generations untouched | | `3pm rotate [target]` | rotate now regardless of size | | `3pm reloadLogs` | reopen every log by path (SIGUSR2 equivalent) | | `3pm delete --flush ` | remove the rows' live + rotated files, except a file a surviving row also writes to | | `3pm logrotate [--print] [--user U]` | logrotate(8) drop-in | | `3pm show ` | paths, sizes (+ generations), date format, rotation policy | | `3pm internals` | `log_sinks`, `log_bytes_written_total`, `log_rotations_total`, effective policy | `flush` / `rotate` / `reloadLogs` reply `Response::LogsOp { touched }` (the paths acted upon); `--json` prints `{"touched":[…]}`. A target is a name, id or namespace. ### `3pm start` flags `-o/--output PATH`, `-e/--error PATH`, `-l/--log PATH` (conflicts with `-o`/`-e`), `--merge-logs`, `--log-date-format FMT`, `--time`, `--disable-logs`. Rotation fields are ecosystem-only. ## 6. Reading: `3pm logs` Order: resolve targets → **subscribe** to `pub.sock` → tail files → stream. Subscribe-first trades a possible duplicate line for the guaranteed gap of tail-then-subscribe on a chatty app, and it is why a follower crosses a rotation without noticing (realtime never comes from file watching). | Flag | Default | Effect | |---|---|---| | `--lines N` | 30 | history per file (`0` = none) | | `--rotated` | off | when the live file holds fewer than N, continue into `.1`, `.2`, … (gzip decoded through a bounded deque), chronological | | `--err` / `--out` | both | one stream. On a merged file the tail is printed once (as stdout); the filter applies to the stream only | | `--raw` | off | no prefix, no color | | `--timestamp[=FMT]` | off; `%Y-%m-%d %H:%M:%S` | stamp streamed lines from `ts`; history lines get blank padding of the same width | | `--format` | off | logfmt: `ts= app= id= instance= type=out\|err message=` (quoted/escaped as needed; `ts=` omitted on history) | | `--highlight TEXT` | — | bright-black background on matches (color only) | | `--nostream` | off | tail and exit | | `-f/--follow` | — | accepted, hidden (following is the default) | | `--json` | off | wire events, tail and stream in one NDJSON schema, `process` and `lagged` events included | Text mode: stdout lines to stdout, stderr lines to stderr, in every mode. Prefix `id|name | ` green/red (pm2 look). `[3PM] log rotated: …` and `[3PM] log stream lagged, N lines missed` go to stderr as chrome. ## 7. Files under `` ``` logs/[-]-out.log[.N[.gz]] stdout (default naming) logs/[-]-error.log[.N[.gz]] stderr logs/[-].log[.N[.gz]] merged 3pm.log the daemon's own log (tracing; THREEPM_LOG=debug …) config.toml optional; [logs] rotation policy, [users] hosted accounts, [hostmetrics] Extended Monitoring, top processes (hostagent.md), [modules] off = ["lan"]: auto modules switched off (modules.md) hosted only in a hosted home: the root daemon owns this home ``` ### `[users]` in `config.toml` Read once, at daemon start (`3pm update` to apply a change), and only by a **root** daemon — anywhere else the section is ignored with a warning: ```toml [users] allow = ["srt"] # accounts this daemon hosts; each needs a private # primary group (`useradd -m -U srt`), and uid 0 is refused # dir = "/var/lib/3pm/users" # where their homes go; root-owned ancestors only # max_apps = 32 # most instances (rows) one account may run ``` Every problem — a name nobody has, a shared primary group, a `dir` anyone but root may write — is logged and skips that user (or the section). A typo must not keep the root daemon, and every app on the machine, down. Full walkthrough: `../FEATURES.md` → *Hosted users*. Log files survive `3pm delete` (use `--flush`), `3pm kill` and daemon restarts: sinks reopen in append mode and seed their byte counter from the file's size, so rotation thresholds carry over. ## 8. Invariants (tested) - Two instances of one app never share a file (proptest on `log_targets`). - No byte is lost across a rotation chain; every generation is line-structured (component + e2e). - A sink shared by several writers follows a rotation (all handles are the same `Arc`). - A rotation never renames a file another rotation is compressing (gzip under the lock). - `disable_logs` drains the pipe (200 KiB written by the mock, nothing persisted, nothing on the bus, empty crash tail). - A `tty` app logs text: no ESC nor C0 control in the file, any split of the stream into reads strips like one read, stripping is idempotent (proptests in `ansi.rs`); two `tty` instances never share a file (`log_targets`); the e2e (`e2e_tty.rs`) reads the log of a real pty. - Wire growth: `ts`, `instance`, `log_rotated`, `LogsOp`, `Delete.flush` are all `serde(default)` / lenient-decoded. Where things live: `threepm-core/{spec,paths,logrotate,ansi}.rs` (pure), `threepm-daemon/{logsink,child_task,child_task_tty}.rs` + `supervisor/` (`do_spawn`, `FlushLogs`/`ReloadLogs`/`RotateLogs`, `LogHousekeeping`), `threepm-cli/commands/{logs,logsops,logrotate,start}.rs`, `threepm-cli/render/{logs,show}.rs`.