Metrics

Every running replica of an agent reports CPU, memory and network samples through the supervisor heartbeat, and every request the ingress relay forwards to it is counted with its status class, duration and bytes. The platform keeps those as minute buckets, rolls them up to five minute and hourly buckets, and serves them as aligned series on GET /v1/agents/{id}/metrics (and GET /v1/runs/{id}/metrics for one run), which the dashboard's Metrics section and keel service metrics draw.

What is measured

Every point stores min, avg, max and count of the samples that fell into the bucket. The API turns that into one display value per metric: the average for gauges (CPU, memory, network, latency), the count for request classes, and avg * count for byte volumes. Across replicas (replica=all) gauges are combined as a count weighted mean for CPU and latency and as a sum for memory, network, requests and bytes.

Restarts are not a metric row. QueryMetrics derives them from the agent's service runs that ended in a crash (isServiceCrash) inside the range and returns them as restarts: [{ at, replica, runId }]; the dashboard draws them as markers.

How samples get in

• Supervisor: every 10 s the supervisor reads one stats sample (metricSampleIntervalSeconds in @keel/runtime-contract). On Docker the executor's observer writes the sample into AGENT_STATS next to peakMemoryMb (the sample field of AgentStatsRecord, computed with containerStatsSample from the Docker stats delta); on Fargate the supervisor reads ECS_CONTAINER_METADATA_URI_V4/task/stats itself. KEEL_CPU_UNITS (set by both executors from the manifest, supervisorEnv.KEEL_CPU_UNITS) is the allocation the percentage is scaled by; without it cpuPercent is null. Samples are batched on the next heartbeat under the metrics field (heartbeatMetricsField), never more than six per heartbeat (maxHeartbeatMetricSamples); a failed heartbeat puts its samples back and the newest six survive. RecordMetrics (packages/application/src/metrics) authenticates the run token, drops samples older than ten minutes or more than a minute in the future, floors each to its minute and merges the points into the store on the run's replica index.
• Ingress: the relay records every forwarded request (both the public face and the peer face, so the relay that owns the tunnel counts it) into an in-memory aggregator keyed by tunnel and minute, and flushes it every 15 s (INGRESS_METRICS_FLUSH_MS) through RecordIngressMetrics, which resolves the tunnel's attempt to its run and replica index (cached per attempt) and merges the points as the platform role. Each flush carries deltas since the previous flush, so a relay restart loses at most one window and never double counts. Latency percentiles are computed over the durations of one flush window (up to 10,000 kept per bucket) and merged into the minute, the five minute and the hourly bucket as a request weighted mean of window percentiles; treat p95 as an approximation of the true bucket p95, exact only when a bucket had a single flush.

Resolution, retention and rollups

The metrics-rollup worker role (also run in process by the API when REAPER=inprocess, every METRICS_ROLLUP_INTERVAL_SECONDS, default 300) runs RollupMetrics: for each step (minute to five minutes, five minutes to hour) it takes the newest target bucket as the watermark, recomputes every complete target bucket from twenty minutes before that watermark (twice the sample age the ingest accepts, so a heartbeat that retried for a while still lands in its rolled up bucket) up to two minutes ago from the source rows in one SQL statement, never reaching back more than six hours in one tick (insert ... select ... group by ... on conflict do update, so a rerun is idempotent), then deletes rows past their retention in batches of 5,000 (at most 20 batches per tick; busy tells the loop to run again at once). Each rollup step and each delete batch is its own short transaction, so a tick never holds locks on more than one batch of rows. A tick that hits the six hour cap logs the metrics rollup reached its window cap: that happens on the first tick of a fresh database and after an outage longer than six hours, and it means the coarse buckets before the cap stay missing while their minute rows live out their retention, the newest six hours always being rolled up first. The window of a query is aligned to the bucket size and ends at the current bucket, so the newest point is always partial.

Storage

metric_points (migration 0031, tenant RLS like every tenant table, keel_platform may read and write across tenants for the ingress flush and the rollup) has one row per (agent_id, name, resolution_seconds, bucket_start, run_attempt_id), the primary key, which also serves the (agent, name, bucket) reads; (run_id, name, bucket_start) serves run queries, (resolution_seconds, bucket_start) the rollup and retention scans, and (run_attempt_id) the cascade when an attempt, its run, its agent or its organization is deleted (every foreign key cascades, so deleting a run never leaves orphaned points). Writes are batched INSERT ... ON CONFLICT DO UPDATE statements (500 rows per statement) that merge min, max, the count weighted avg and the count, so a heartbeat and a relay flush for the same minute never race each other.

Write volume for 100 services with one replica each, all behind ingress with traffic every minute: at most 13 rows per replica and minute (5 from the supervisor, 8 from ingress), so about 1,300 new minute rows a minute, 78,000 an hour, 1.9 million a day. With 25 hours of minute rows, 8 days of five minute rows and 90 days of hourly rows the table settles around 2 million + 3 million + 2.8 million rows, roughly 1.5 GB with indexes. At three replicas each that is 3,900 minute rows a minute, 5.6 million a day, and a plateau near 6 million + 9 million + 8.4 million rows, about 8 GB with its five indexes; the rollup deletes 100,000 rows per tick per resolution at most, twenty times the minute inflow, so retention keeps up. Each minute row is upserted about four times before it is complete (a heartbeat every 15 s, a relay flush every 15 s), which is why autovacuum matters more than the row count (the operations notes). Services without ingress traffic write 5 rows a minute.

Reading latency through the relay versus inside the app

The relay measures from the moment it has the request headers until the last response byte left the relay, so it includes the tunnel hop to the replica (a few milliseconds on the single host, more across availability zones on Fargate), the time the agent takes, and the time to stream the response body back through the tunnel. It does not include the client's connection to the relay or TLS. A slow client that reads a large response slowly stretches http_bytes_out requests; a request the relay times out (INGRESS_REQUEST_TIMEOUT_MS) is counted as a 5xx with the timeout as its duration. When the app measures its own handler time and the relay shows a higher p95, the difference is tunnel flow control and response streaming, not the platform being slow to route. For per endpoint latency, the app's own logs remain the tool; the relay only sees one number per request.

API and tools

• GET /v1/agents/{id}/metrics?range=1h|24h|7d&names=cpu_percent,...&replica=all|<index> (agents.read) and GET /v1/runs/{id}/metrics (runs.read, scoped to that run, no restart markers) return { agentId, runId, range, resolutionSeconds, from, to, timestamps, replicas, series: [{ name, unit, replica, points: [{ value, min, max, count } | null] }], restarts }; HTTP API.
• keel service metrics [agent] [--range 24h] [--replica 1] prints last, average and max per series with a sparkline; --json gives the document; The CLI.
• The dashboard's agent page shows the Metrics section for service agents; The dashboard.
• Ports: MetricStore (packages/application/src/metrics/ports.ts, implemented by DrizzleMetricStore and InMemoryMetricStore), use cases RecordMetrics, RecordIngressMetrics, QueryMetrics, RollupMetrics.