# 3pm-backend route `queryMetrics`

Paths are relative to the backend's origin (https://fleet.edgecr4ft.net, or your own 3pm-backend), not to this site. Index: https://3pm.edgecr4ft.net/docs/backend/api.md

---

tokens:
  claude: 3992
  o200k: 2672
# POST /api/buckets/{clusterId}/metrics/query (queryMetrics)

Time series for charts: metrics, filters, aggregation and step, scoped to the bucket.

- auth: an account — Authorization: Bearer 3pmu_<token>, or the dashboard's session cookie
- node: also open to a node's own bearer (3pma_…) when its row is granted fleet:read; that node's bucket only, and for the calls it placed itself
- body: MetricsQuery (JSON)
- answer: MetricsResult (JSON)

## Details

Discovery first: `metricsCatalog` gives every metric id, the labels each one carries and its `default_time`; `metricLabelValues` gives what a label holds in this bucket. A metric the catalog lacks, a label the metric does not carry, `rate` on a gauge, a regex that does not compile or any unknown field is a 400 `ERR_BAD_REQUEST` naming what was wrong.

Body `{ range, queries, formulas?, options? }`:

- `range` = `{ from, to?, step? }`. `from` and `to` are epoch milliseconds or `now` / `now-<duration>` (`now-90m`, `now-7d`); `to` defaults to `now`. `step` is `30s` / `5m` / `1h` or milliseconds, and is optional: left out, it is derived from `options.max_points`. Both ends are aligned to it, and the answer says which step was used (`range.step_ms`).
- `queries` = 1 to 8 `{ ref, metric, filters?, time?, aggregate?, hidden? }`. `ref` matches `^[A-Z][A-Z0-9]{0,7}$` and is unique across queries and formulas. `metric` is a catalog id (`process.cpu.utilization`, or `app.<name>` for an application's own). `filters` are AND-ed `{ label, op, value | values }` over `eq | neq | in | nin | regex | nregex`. `time` is the time function and its window — `rate` / `increase` / `delta` on a counter, the `*_over_time` family on a gauge, `quantile` (with `q`) on a histogram, `raw` on either — the window defaulting to twice the step. `aggregate` = `{ fn: sum | avg | min | max | count | quantile, by: [labels] }` collapses the series that share `by`; without it, one series comes back per label combination the metric carries.
- `formulas` = up to 8 `{ ref, expr }` over the other refs: `A / B * 100`, with `+ - * /` and parentheses. A query marked `hidden: true` is computed for a formula and not returned.
- `options`: `max_points` (the step is raised until the range fits), `max_series` with `sort` (`avg` or `last`) deciding which are kept, `fill` (`null` | `zero` | `previous`), `mode` (`series`, or `summary` to drop the values and keep min/max/avg/last/count per series), `explain` to get the compiled expression back.

Answer `{ range, times, results, took_ms }`: one `times[]` shared by every series, then one entry per visible ref with `series[].labels`, `series[].values` aligned on `times`, `series[].summary`, and `series_total` / `truncated` when the ranking cut some off.

The bounds are `limits.queryMetrics` of `GET /api/v1/routes`, and the step may not go below the store's resolution (`store.interval_ms` of `GET /api/v1/info`). They bound the grid, not the window, and what a step is measured against is `options.max_points` — 600 by default, 2000 at most: 30 days at `5m` is 8 640 points and is refused whatever it is set to. The same 30 days draws at `2h` on the default and at `30m` (1 442 points) with `max_points: 2000`; a `5m` step fits two days on the default and just under seven at 2000. Leave `step` out and the finest step of the ladder that fits is picked for you.

Example — 24 hours per app: the peak resident memory, and the average CPU turned into a percentage by a formula (the ratio itself computed and not returned). Every metric it names is a static catalog line, so it runs against any bucket:

{"range":{"from":"now-24h"},"queries":[{"ref":"CPU","metric":"process.cpu.utilization","filters":[{"label":"app","op":"in","values":["api","worker"]}],"aggregate":{"fn":"avg","by":["app"]},"hidden":true},{"ref":"RSS","metric":"process.memory.rss","aggregate":{"fn":"max","by":["app"]}}],"formulas":[{"ref":"PCT","expr":"CPU * 100","unit":"percent"}],"options":{"max_points":300}}

## Bounds

- default_points: 600
- default_series: 200
- max_body_bytes: 65536
- max_filters: 16
- max_formulas: 8
- max_points: 2000
- max_queries: 8
- max_range_ms: 2678400000
- max_series: 2000
- max_value_len: 256
- max_values: 64
- step_floor_from: "info.store.interval_ms"

## Types

```ts
/** `POST /api/buckets/{id}/metrics/query`. */
interface MetricsQuery {
  range: TimeRange;
  /** 1–8. */
  queries: QuerySpec[];
  /** ≤ 8. */
  formulas?: Formula[];
  options?: QueryOptions;
}

interface MetricsResult {
  range: ResultRange;
  /** One timestamp per point, shared by every series. */
  times: EpochMs[];
  results: ResultSet[];
  took_ms: number;
}

interface TimeRange {
  from: TimeValue;
  /** Default `now`. */
  to?: TimeValue;
  /** `30s`, `5m`, `1h` or milliseconds (what `step_ms` answers); omitted ⇒ derived from `options.max_points`. Both ends are aligned to it. */
  step?: DurationValue;
}

interface QuerySpec {
  /** `^[A-Z][A-Z0-9]{0,7}$`, unique across queries and formulas. */
  ref: string;
  /** A catalog id, `process.cpu.utilization`. */
  metric: string;
  /** AND-ed; at most 16. */
  filters?: Filter[];
  /** Default: the catalog's `default_time` for the metric. */
  time?: TimeFn;
  aggregate?: Aggregate;
  /** Computed for a formula, not returned. */
  hidden?: boolean;
}

interface Formula {
  ref: string;
  /** Refs, numbers, `+ - * /` and parentheses: `A / B * 100`. */
  expr: string;
  unit?: string;
}

interface QueryOptions {
  /** 2–2000, default 600: the step is raised until the range fits. Every bound here is also `limits.queryMetrics` of `GET /api/v1/routes`, which is generated from the validator. */
  max_points?: number;
  /** 1–2000, default 200: the series kept are the top ones by `sort` over the whole range (the engine ranks, the backend re-ranks the same way). */
  max_series?: number;
  /** What ranks series for `max_series`, descending; default `avg`. */
  sort?: "avg" | "last";
  /** Default `null`. */
  fill?: "null" | "zero" | "previous";
  /** `series` (default) or `summary` (no values). */
  mode?: "series" | "summary";
  /** Add `results[].expr`: the expression as compiled for the engine. */
  explain?: boolean;
}

interface ResultRange {
  /** Aligned to `step_ms`: `from` rounded down, `to` rounded up. */
  from: EpochMs;
  to: EpochMs;
  step_ms: number;
}

/** Milliseconds since the Unix epoch. */
type EpochMs = number;

interface ResultSet {
  ref: string;
  /** The catalog id; absent on a formula. */
  metric?: string;
  kind: MetricKind;
  /** `ratio`, `bytes`, `seconds`, `count`, `bool`, or the formula's; absent on a formula that gave none. */
  unit?: string;
  series: Series[];
  /** The series the engine answered, at most `max_series + 1`: `truncated` is exact, this is a floor, not the bucket's count. */
  series_total: number;
  truncated: boolean;
  /** With `options.explain`: the expression as compiled for the engine. */
  expr?: string;
}

/** Epoch milliseconds, or `now` / `now-<duration>` (`now-1h`, `now-7d`). */
type TimeValue = EpochMs | string;

/** Milliseconds, or `30s` / `5m` / `1h` / `7d` / `2w`. */
type DurationValue = number | string;

interface Filter {
  label: QueryLabel;
  op: FilterOp;
  /** `eq`, `neq`, `regex`, `nregex`. ≤ 256 characters. */
  value?: string;
  /** `in`, `nin`: 1–64 values. */
  values?: string[];
}

interface TimeFn {
  fn: TimeFnName;
  /** `1m` or milliseconds, like `range.step`; default twice the step. Not for `raw`. */
  window?: DurationValue;
  /** 0..1, `quantile_over_time` and `quantile` (a histogram's) only. */
  q?: number;
}

interface Aggregate {
  fn: AggregateFnName;
  /** Labels the metric carries. */
  by?: QueryLabel[];
  /** 0..1, `quantile` only. */
  q?: number;
}

type MetricKind = "gauge" | "counter" | "histogram";

interface Series {
  /** The `by` labels, or every public label without an aggregation — never the bucket. */
  labels: Record<string, string>;
  /** Aligned on `MetricsResult.times`; absent in `summary` mode. */
  values?: (number | null)[];
  summary: Summary;
}

/** The labels a query may filter or group on; the bucket itself is never one of them. */
/** A built-in label, or an application's own label name (`^[a-z][a-z0-9_]{0,31}$`, not reserved) on a metric that carries it. */
type QueryLabel = "node" | "app" | "instance" | "ns" | "mount" | "window" | "src" | "core" | "iface" | "blockdev" | "sensor" | "gpu" | "array" | "proc" | (string & {});

type FilterOp = "eq" | "neq" | "in" | "nin" | "regex" | "nregex";

/** `rate`, `increase`, `delta` for counters; the `*_over_time` family for gauges; `raw` for either; `quantile` (with `q`), `rate` or `increase` for a histogram. */
type TimeFnName =
  | "raw"
  | "rate"
  | "increase"
  | "delta"
  | "avg_over_time"
  | "min_over_time"
  | "max_over_time"
  | "last_over_time"
  | "quantile_over_time"
  | "quantile";

type AggregateFnName = "sum" | "avg" | "min" | "max" | "count" | "quantile";

/** Over the points the engine returned on the grid — before `fill`, so `sort` ranks what the engine ranked. */
interface Summary {
  min: number | null;
  max: number | null;
  avg: number | null;
  last: number | null;
  /** Points the engine returned (not the filled ones). */
  count: number;
}
```