# Resource Limits Reference `max_memory_restart`, `max_cpu_percent`, `limit_scope`, `cgroup_enforce`, `deny_gpu`. The first two are *soft* guards (observe a sample of the app's whole process tree, restart gracefully); `limit_scope` narrows what they judge to the spawned pid; `cgroup_enforce` turns the same two numbers into *hard* kernel limits on Linux; `deny_gpu` hides accelerators through environment variables. ## Cheatsheet ```console # soft guards: the process tree sampled every second, restart when a sample crosses the line 3pm start api.js --max-memory-restart 300M # RSS of the whole tree > 300 MiB → graceful restart 3pm start worker.py --max-cpu-percent 80 # CPU > 80 % of one core → restart 3pm start train.py --max-cpu-percent 250 # multi-core: 250 % = 2.5 cores 3pm start api.js --max-memory-restart 1G --max-cpu-percent 120 # both # hard limits (Linux, cgroup v2): the kernel throttles / OOM-kills 3pm start api.js --max-cpu-percent 50 --cgroup-enforce # cpu.max = 50000 100000 3pm start api.js --max-memory-restart 512M --cgroup-enforce # memory.max = 536870912 3pm start api.js --cgroup-enforce # own cgroup, no caps (accounting only) # GPUs 3pm start infer.py --deny-gpu # CUDA_VISIBLE_DEVICES="" NVIDIA_VISIBLE_DEVICES=none … # inspect 3pm ls # cpu / mem = the tree's totals, `+N` = its other processes 3pm ls --tree # who weighs what under each app; ⚠ = a stop will not reach it 3pm show api # restarts count, every process, "last error" if enforcement failed 3pm logs api --err # the app's own view (OOM kill = SIGKILL, exit signal 9) cat /sys/fs/cgroup/3pm/api-1-0-*/cpu.stat # nr_throttled, throttled_usec (Linux; leaf = ----g) cat /sys/fs/cgroup/3pm/api-1-0-*/memory.events # oom_kill counter (Linux) # change limits: restart on the new spec 3pm start api.js --max-cpu-percent 60 # same name = update spec + restart 3pm restart api --update-env # re-read the ecosystem file it came from ``` Ecosystem file (`3pm start ecosystem.toml`): ```toml [defaults] max_cpu_percent = 200 # every app starts from this [[apps]] name = "api" script = "./server.js" max_memory_restart = "300M" # "512", "100K", "200M", "1G" max_cpu_percent = 80 # float, > 0; overrides [defaults] limit_scope = "tree" # what both judge: "tree" (default) or "leader" (the spawned pid alone) cgroup_enforce = true # Linux only deny_gpu = true [apps.profiles.dev] # 3pm start ecosystem.toml --env dev cgroup_enforce = false # laptops have no delegated cgroups ``` ## 1. Soft guards: `max_memory_restart`, `max_cpu_percent` The metrics sampler reads the process table once per second (`THREEPM_METRICS_INTERVAL_MS` shrinks it for tests) and attributes to each instance its **process tree** — the spawned pid and every descendant; the rules are in `docs/metrics.md` → *Process trees*. Each sample is fed to the pure policy as `MemorySampled{gen, tree_rss, leader_rss}` and `CpuSampled{gen, tree_cpu, leader_cpu}`; the policy — not the daemon — picks the figure named by the spec's `limit_scope` (`tree`, the default, or `leader`), compares it to the limit and, when **strictly above**, stops the app gracefully — `kill_signal` (default SIGINT) to the process group, SIGKILL after `kill_timeout` — then restarts it as after a crash: `restart` event, `waiting restart` for `restart_delay` / the exponential backoff, respawn, restart counter +1. What that implies: - **The whole tree is judged.** `max_memory_restart = "300M"` fires when the members' RSS *summed* crosses 300 MiB, even if no single process does — a browser, a build or a helper that grows recycles the app. That is what `3pm ls` shows (`+N`, and `--tree` for who weighs what). An app whose children are meant to grow keeps the old meaning with `limit_scope = "leader"` (per app, in the ecosystem file or the `@3pm/api` start options — no CLI flag: the spawned pid alone is judged, the sampler still measures the tree); `THREEPM_PROCESS_TREE=off` on the daemon does it for every app. - **A detached member counts but is not restarted.** A process that left the group (`setsid`; `⚠` in `3pm ls --tree`) weighs in the sum, and the restart's signals go to the group: it survives the restart, and no later incarnation adopts it — its environment names the old incarnation, and after a cold restart of the daemon an older lineage too. Kill it by hand, or `limit_scope = "leader"`; under `cgroup_enforce` it dies with its leaf when the incarnation ends. - **One sample is enough.** A one-second CPU spike above `max_cpu_percent` recycles the app. Size the limit for the app's steady state, not its peak, or leave the CPU guard off and use `cgroup_enforce` to throttle instead of restarting. - **The first sample after a spawn reads 0 % CPU** (CPU% is a delta between refreshes). A guard therefore fires no sooner than ~2 s after start. - **Limits only fire on `online` rows.** A row that is `launching`, `stopping`, `stopped`, `errored`, or being shut down with the daemon ignores samples. A stale sample for a superseded incarnation (gen mismatch, pid reuse) is dropped. - **Restart budget applies.** A guard restart is judged like a crash, on the incarnation's uptime **at the trip** (the graceful stop does not count), counted from the moment it went `online` — its spawn, or under `wait_ready` its ready signal or `listen_timeout` — against `min_uptime` + 10 s (`GUARD_SETTLE_MS`: the sampler needs a few periods to see a limit, the CPU guard two samples), or + 3 periods when the sampler is slow enough (`THREEPM_METRICS_INTERVAL_MS` above 3.33 s) that this is longer: a trip before that is unstable and counts toward `max_restarts`, a later one starts the count over. `max_restarts` unstable trips in a row end in `errored`, without a respawn — with the defaults (16, 1 s, a 1 s sampler) an app over its limit from the moment it is `online` is `errored` after 16 trips, one that runs more than ~11 s between trips never is. Known limit: under `limit_scope = "tree"` the uptime is the leader's, so a wrapper whose busy (or growing) child appears late — past that line minus the two periods the guard needs to see it — trips as a healthy run, and such an app may cycle without ever reaching `errored`. A user `restart` / `stop` / `delete` during the guard's stop, a watch or cron restart start a new series (unstable count and backoff at 0, immediate respawn); `3pm reset` clears the counters. A guard's stop in progress, or its pending respawn, crosses an in-place `3pm update`. `max_restarts = 0` means "never respawn on its own", as for a crash: the first trip ends `errored`, stable or not. - **`restart_policy` and `stop_exit_codes` do not apply.** They judge the exits an app makes on its own; the exit after a trip is its answer to the guard's `kill_signal`. A guard restart is the daemon's own recycle, like a watch or cron restart — and like pm2, whose `max_memory_restart` reloads the app whatever `autorestart` says — so an app under `restart_policy = "never"` (or `autorestart = false`) that crosses its limit is restarted. To have it stop instead, leave the guard off; `cgroup_enforce` then caps it without a restart. - **Units.** `max_memory_restart` is RSS in bytes (`K`/`M`/`G` are 1024-based), summed over the tree — an upper bound when members share pages. `max_cpu_percent` is percent of one core, summed over the tree — `100` is one core saturated, `400` is four; it may exceed 100. Validation: `max_cpu_percent` must be a finite number > 0. `3pm start --max-cpu-percent 0` and `max_cpu_percent = -1` in a file are rejected before anything reaches the daemon; the daemon re-validates decoded specs so a hand-edited `dump.json` cannot smuggle one in. ## 2. Hard limits: `cgroup_enforce` (Linux, cgroup v2) With `cgroup_enforce = true` every incarnation runs, from before its `exec`, in a leaf of its own: ``` /sys/fs/cgroup/3pm/----g/ cpu.max " 100000" quota = max_cpu_percent % of 100 ms (only if set) memory.max "" = max_memory_restart (only if set) cgroup.procs the child (written by itself before exec), then all it forks ``` `` is the daemon's `THREEPM_BOOT` (16 hex digits, drawn at every cold start, kept by an in-place `3pm update`) and `` the incarnation's: a leaf is never shared, neither with the next incarnation nor with a daemon started cold after a crash, which numbers its rows from 0 again. `` is sanitized. The kernel then throttles the cgroup at the CPU quota and OOM-kills it above the memory cap. The leaf's `cgroup.procs` is also the exact membership of the app's process tree for the sampler: a `setsid` escapee stays in the cgroup — a member, marked `detached` since the group's signals will not reach it. The soft guards keep running on top: with both, memory usually ends in an OOM kill (exit signal 9, seen in `3pm show`) before the sampler sees the RSS cross the line, and the restart policy handles it like any crash — except for a tree that shares many pages (Electron): the kernel counts them once, the sampler's RSS sum counts them per member, so the soft guard may fire first. `limit_scope = "leader"` leaves the kernel in charge of the tree: the guard then judges the spawned pid alone while `memory.max` bounds the whole cgroup. Order of operations (`supervisor::lifecycle::spawn_child`, `cgroup.rs`, `spawner_cgroup.rs`): 0. Open the log files (every spawn, enforce or not): one that cannot be opened fails the spawn, `last error` naming it, before any process or leaf exists. 1. Resolve the hierarchy **before** spawning. Not Linux, or no `cgroup.controllers` at `/sys/fs/cgroup` → the spawn fails with `cgroup_enforce is only supported on Linux` / `needs cgroup v2 mounted at /sys/fs/cgroup`. No process was started; the row backs off and retries per its restart policy and, once the budget is spent, sits in `errored` with the reason under `last error` in `3pm show`. 2. Prepare the leaf (`CgroupFs::prepare`): create `/sys/fs/cgroup/3pm`, enable the `cpu` / `memory` controllers for its children as needed, create the leaf — one that already exists is refused: it is somebody else's — and write the limits. Any failure fails the spawn, nothing started; a leaf whose limits cannot be written is removed at once. 3. Spawn. The daemon opens the leaf's `cgroup.procs`; the child writes `0` (itself) into it between `fork` and `exec`, so the program and everything it forks run inside from their first instruction. A join that fails in the child fails the spawn with `cgroup_enforce: cannot join /cgroup.procs: …` — never blamed on the program — and counts as a spawn failure. An enforce spec never runs unenforced. 4. A spawn that failed after step 2 removes its leaf (one `rmdir`, `CgroupFs::discard`). 5. When the incarnation is over — exit, `delete`, death during the handoff window — `CgroupFs::cleanup` runs on a blocking thread: everything left in the leaf is killed (`cgroup.kill`, Linux ≥ 5.14; otherwise SIGKILL pid by pid from `cgroup.procs`, nested cgroups included, never pid 0, 1 or the daemon — with the leaf frozen by `cgroup.freeze`, Linux ≥ 5.2, from before the read to after the last kill, so that no member can exit and its pid go to another process in between; a frozen task still dies of SIGKILL; thawed after, waited for at most ≈ 45 ms to report `frozen 1`), the cgroups the app made below its leaf are removed deepest first (directories only, never through a symlink, 8 levels and 1 024 of them at most), then the leaf (up to 10 attempts 20 ms apart, ≈ 0.2 s; one still busy is left to a later cleanup), then the **empty** leaves of the same instance and lineage with a **lower** gen go too. A newer leaf (the next incarnation's, made while this runs), another instance's, another lineage's or a leaf named before lineages is never swept. Known gap: the empty leaves a crashed daemon left (another lineage) are not cleaned; `rmdir` them by hand. `cgroup_enforce` also changes what the end of an incarnation reaches: without it, the process group is SIGKILLed at the exit and a member that left the group (`setsid`) survives; with it, the whole leaf is killed, those members included. ### Permissions Writing under `/sys/fs/cgroup` needs either root or a delegated subtree. The error message says which file was refused: ``` cgroup_enforce: /sys/fs/cgroup/3pm: permission denied — run the daemon as root or delegate /sys/fs/cgroup/3pm to this user ``` Options, from simplest to cleanest: ```console # a) daemon as root (3pm startup systemd installs it as the given user — use root) sudo 3pm start ecosystem.toml # b) one-off delegation to the daemon's user (lost at reboot) sudo mkdir -p /sys/fs/cgroup/3pm sudo chown -R "$USER" /sys/fs/cgroup/3pm echo '+cpu +memory' | sudo tee /sys/fs/cgroup/cgroup.subtree_control # c) systemd: run the daemon in a delegated slice (survives reboot) # in the unit written by `3pm startup systemd`, add: # [Service] # Delegate=cpu memory ``` Under (c) the daemon's own cgroup is delegated, not `/sys/fs/cgroup`; 3pm still uses the fixed `/sys/fs/cgroup/3pm` path, so pair it with (b)'s `chown` on that directory or run as root. Containers: the container's cgroup namespace must expose a writable `/sys/fs/cgroup` (`--cgroupns=host --privileged`, or a delegated `--cgroup-parent`). ### Checking it works ```console $ cat /proc/$(3pm show api --json | jq .pid)/cgroup 0::/3pm/api-1-0-3f9c2a17e4b80d51-g1 $ cat /sys/fs/cgroup/3pm/api-1-0-3f9c2a17e4b80d51-g1/cpu.max 50000 100000 $ cat /sys/fs/cgroup/3pm/api-1-0-3f9c2a17e4b80d51-g1/cpu.stat | grep throttled nr_throttled 1432 throttled_usec 71283910 $ cat /sys/fs/cgroup/3pm/api-1-0-3f9c2a17e4b80d51-g1/memory.events oom_kill 2 ``` `3pm doctor`-style prerequisite checks do not exist yet: the spawn error is the diagnostic. ## 3. `deny_gpu` Sets, after the app's own `env` (so the deny wins over a user value): | Variable | Value | Honoured by | |---|---|---| | `CUDA_VISIBLE_DEVICES` | `""` | CUDA runtime, PyTorch, TensorFlow, JAX | | `NVIDIA_VISIBLE_DEVICES` | `none` | NVIDIA container toolkit | | `HIP_VISIBLE_DEVICES` | `""` | ROCm / HIP | | `ROCR_VISIBLE_DEVICES` | `""` | ROCm runtime | Best effort by construction: a process that clears these variables, or opens `/dev/nvidia*` / `/dev/kfd` directly, still reaches the hardware. A real wall is a device cgroup (`devices.deny` / BPF device programs), which 3pm does not manage. Works on every platform since it is only environment. ## 4. Choosing between soft and hard | Goal | Use | |---|---| | Recycle a leaking service before it swaps | `max_memory_restart` | | Recycle an app stuck in a busy loop | `max_cpu_percent` (steady-state sized) | | Keep a batch job from starving neighbours, no restart | `max_cpu_percent` + `cgroup_enforce` — set the soft guard above the cap: the kernel throttles at the cap, so the sampler never sees it crossed | | Guarantee an app can never exceed N bytes | `max_memory_restart` + `cgroup_enforce` | | Let the children grow (a build, a browser) without recycling the app | `limit_scope = "leader"` — the guards judge the spawned pid alone | | Share a GPU box with CPU-only workers | `deny_gpu` on the workers | | Same file on Linux servers and macOS laptops | `cgroup_enforce = true` in the base, `false` in a `[apps.profiles.dev]` overlay | Note the throttle case: `cpu.max` caps usage *at* `max_cpu_percent`, and the guard fires only *above* it, so with `cgroup_enforce` a CPU-bound app is throttled rather than restarted. The memory pair does not have that property — an OOM kill is a crash, and a leak that hits the cap restarts either way. ## 5. Where it lives - Spec fields, validation, TOML/CLI mapping: `threepm-core/src/spec.rs` (`max_cpu_percent`, `limit_scope`, `cgroup_enforce`, `deny_gpu`), overlay rule in `profile.rs` (scalar: the profile's value wins). - Decision: `threepm-core/src/policy/guards.rs` — `MemorySampled` / `CpuSampled` carry both figures, `scoped` picks by `limit_scope`, `on_resource_sampled` is shared by the two guards, `after_guard_stop` respawns through `respawn_or_errored` (`policy.rs`, the crash path), `settle_margin_ms` is the stability margin (`GUARD_SETTLE_MS` or `GUARD_SETTLE_PERIODS` sampler periods, whichever is longer; the period rides each sample, `sample_period_ms`). - Membership and totals: `threepm-daemon/src/metrics/tree.rs` (pure), `metrics/table.rs` (the sweep), `supervisor/trees.rs` (the record). - Enforcement: `threepm-daemon/src/cgroup.rs` (`CgroupFs::prepare` / `discard` / `cleanup`), `spawner_cgroup.rs` (the child's join before `exec`), hooked from `supervisor/lifecycle.rs::spawn_child`, released (`release_cgroup`, on a blocking thread) on `ChildExited` (`actor.rs`) and `remove_process`. - GPU env: `threepm-daemon/src/spawner.rs::GPU_DENY_ENV`. - Tests: fake-hierarchy unit tests in `cgroup_tests.rs` (a leaf per incarnation, the older-leaves sweep, a leaf whose limit cannot be written removed) and `cgroup_kill_tests.rs` (`cgroup.kill`, the frozen SIGKILL fallback, the cgroups below a leaf, symlinks and the depth bound); the join before `exec` in `spawner_tests.rs` (`a_cgroup_leaf_is_joined_before_the_program_runs`, `a_join_that_fails_in_the_child_is_named_as_the_cgroups`); policy tables in `policy/guard_tests.rs` (`limit_scope_picks_the_tree_sum_or_the_leader_alone`, `a_guard_tripping_on_every_incarnation_ends_errored_with_the_defaults`, `a_trip_is_judged_on_its_uptime_at_the_trip`, `deliberate_restarts_never_count`, `no_restart_budget_errors_on_the_first_trip_as_on_the_first_crash`, `a_guard_restarts_whatever_the_restart_policy_and_the_stop_exit_codes`); component tests `cpu_exceeded_sample_triggers_restart`, `memory_guard_judges_the_tree_sum_unless_the_scope_is_the_leader`, `cgroup_enforce_off_linux_fails_the_spawn_before_it_happens`, `a_log_file_that_cannot_be_opened_fails_the_spawn_before_anything_starts` (`supervisor/tests/spawn.rs`); e2e `max_cpu_percent_cycles_the_process`, `deny_gpu_hides_devices_via_env`, `cgroup_enforce_off_linux_errors_the_row_with_a_reason`.