Logs

Container stdout and stderr of every run, searchable across the runs and replicas of an agent, filterable by time, level, replica and stream, and tailable live in the dashboard, the API and the CLI. The supervisor redacts every line before it leaves the container (packages/redaction), so nothing described here ever sees an unredacted line.

Two stores, one ingest path

POST /ingest/v1/attempts/{attemptId}/logs (Services and APIs, HTTP API) keeps writing each batch as a gzip chunk in the payload store plus a run_log_chunks row. Since migration 0032 the same request, in the same transaction, also writes one row per line into the log_lines hot table:

Indexes: (agent_id, ts desc, id desc) and (run_id, ts desc, id desc) for search pages, (agent_id, id) and (run_id, id) for the tail's max(id) watermark, ts for retention, GIN on search for word queries and a pg_trgm GIN index on line for substring queries. There is deliberately no index on organization_id alone: every statement carries an agent or run predicate next to the RLS one, and explain showed the column's index serving no query on the table. The foreign keys cascade, but nothing in the product deletes an organization, agent, run or attempt row (agents are archived), so no cascade path depends on it either; an operator deleting one of those rows by hand scans the table, as the unindexed attempt_id foreign key already implies. The migration enables pg_trgm (a trusted extension, so the database owner may create it on RDS and on the single host without superuser rights). RLS mirrors every other tenant table (keel_app sees app.org_id only); keel_platform may select and delete across tenants for retention.

The chunk path stays the source of truth: GET /v1/runs/{id}/logs without filters, the run page's Logs tab and keel logs <run-id> read chunks and keep working for any run age. The hot table is a seven day window.

Retention

Lines older than seven days (logLineRetentionDays) are deleted by the retention worker role every ten minutes in batches of 5,000 rows, at most 20 batches per tick; a tick that hits its budget reports busy and the loop runs it again with no sleep, so the 20 batch budget is a yield point and not a throughput ceiling. Each batch is its own transaction, and the delete repeats the ts < cutoff predicate next to the id in (...) subselect so Postgres finds the rows through log_lines_ts_idx instead of hash joining the batch against a sequential scan of the whole table: measured on 300,000 rows with everything expired, 7.7 ms a batch with the predicate against 77.5 ms and a 299,943 row sequential scan without it, and only the first stays flat as the table grows. Nothing holds a lock long enough to block ingest. Search answers retainedFrom, the oldest instant the hot table can cover, so clients can say when a window falls off the edge. A from earlier than retainedFrom is clamped, a to before it answers an empty page. Chunks in the payload store are not touched by this sweep.

Because the table churns its whole contents every seven days, migration 0032 sets autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor to 0.02 on it: the default 0.2 would let a 300 M row table accumulate 60 M dead rows before a vacuum, and the two GIN indexes bloat with it. Watch pg_stat_user_tables.n_dead_tup and last_autovacuum for log_lines after a volume change.

Write volume and hot storage cost

One ingest batch is at most 500 lines and 1 MB of text (packages/api-contracts/src/ingest.ts), so one batch is at most 500 row inserts plus one chunk row and one object put, inside the transaction that already existed. The per attempt cap of 50 MB of log text (defaultMaxLogBytesPerAttempt) bounds the rows per attempt: a run that logs 100 byte lines stops being indexed after roughly 500,000 lines, and the ingest answers limitReached exactly as before. Rows are inserted after the object put and the chunk row, so ids are assigned as late as possible in the transaction.

Measured on Postgres 17 with 300,000 rows of realistic service output (average line 104 bytes, 31 MB of raw text):

Indexing a full 500 line batch into that table measured 31 ms median and 71 ms at p95 on the same machine, against 0.29 ms for the run_log_chunks row alone, so the ingest transaction is roughly a hundred times more expensive than it was on the chunk path and one connection still indexes about 16,000 lines a second. Watch the POST /ingest/v1/attempts/{id}/logs latency after a volume change.

So the hot table costs about eleven times the raw log text, of which the two GIN indexes are about four times. Budget per fleet from the line rate, not from the byte volume:

The single host keeps log_lines in the same volume as the rest of the platform database, so the cheap tier is sized for the first row of that table; past roughly 50 M rows a day the next step is daily partitions dropped instead of deleted, which also removes the vacuum cost above. The chunk path in the payload store is unaffected by any of this and stays the source of truth for a whole run.

Level parsing

parseLogLevel (packages/domain/src/logs/log-level.ts) runs at ingest and in the dashboard for chunk lines:

1. A line that is a JSON object with level, severity or lvl uses that field: the strings error|err|fatal|crit|critical|alert|emerg|emergency|panic|severe map to error, warn|warning to warn, info|notice|informational to info, debug|trace|verbose|silly|fine to debug; pino and bunyan numbers map 50+ to error, 40 to warn, 30 to info, 10..29 to debug. Any other value, or a JSON object without such a field, is null.
2. Otherwise a leading level token counts: optionally preceded by a date and a time token, optionally wrapped in [] or (), followed by ], ), :, |, - or a space. ERROR boom, warn: slow, [INFO] ready, 2026-09-16 10:00:00,123 DEBUG x and 2026-09-16T10:00:00Z ERROR: x all parse.
3. Otherwise a logfmt or key value pair level=, severity=, lvl= (also with :) within the first 512 characters counts.
4. Anything else is null and matches no level filter.

Only error, warn, info and debug exist; the aliases above are folded into them.

Search semantics

The q parameter is parsed once by parseLogQuery (packages/domain/src/logs/log-query.ts) and applied identically by Postgres and by the in memory fakes:

• One plain term (no spaces, no quotes, no leading -) is a case insensitive substring match anywhere in the line, served by the trigram index. It needs at least three characters and at least one letter or digit, because pg_trgm extracts no trigram from a pattern without one and Postgres would answer it with a sequential scan of the whole table (not_indexable).
• Several terms, a "quoted phrase" or a -excluded term switch to word mode. The line is split into alphanumeric words (the same normalisation the search column uses, so erp.internal/invoices yields erp, internal, invoices). Every positive term must match: a term with one word matches lines that contain a word starting with it (retry matches retrying), a term with several words or a quoted phrase matches those words adjacent and in order, each as a prefix, and -term excludes lines that match it. Positive terms need at least two characters and at least one alphanumeric character; a query made only of exclusions is refused.
• Queries longer than 256 characters are refused. Every accepted query reaches Postgres as an index usable condition: the trigram index for substring mode, the search GIN index for each positive word term (packages/infrastructure/src/db/log-lines.test.ts asserts both against explain). Only a -term exclusion is a filter, and it only ever narrows a positive term's index scan.

Refusals are 400 log_query_too_broad with reason (empty, too_short, only_negations, too_long, not_indexable) and minLength; from >= to is 400 log_range_invalid; a cursor the endpoint did not issue is 400 log_cursor_invalid. Levels, replica, stream and (on a run) attempt are exact matches, from is inclusive and to exclusive.

Pages are newest first with a keyset cursor over (ts, id), so a page never repeats or skips a line while new ones arrive; limit is at most 500. Highlighting in the dashboard marks each positive term as a substring of the line, which is a superset of the match in word mode.

Tail

GET /v1/agents/{id}/logs/tail is a server-sent event stream of log events ({ event: "log", lines: [...] }, event id = the last line id in the batch) for new lines of the agent across all its runs and replicas, with the same q, levels, replica and stream filters. Without a cursor the stream starts at the newest existing line; Last-Event-ID: <line id> or ?afterId= resumes after that id and replays what was missed (at most 500 lines per event). The stream polls the hot table every RUN_EVENTS_POLL_INTERVAL_MS (2 s by default, the same setting as run events), writes : keepalive every 15 seconds, rechecks the caller's access every RUN_EVENTS_ACCESS_RECHECK_MS and sends access_revoked before closing when it is gone, and counts against RUN_EVENT_STREAMS_PER_ORGANIZATION like a run event stream.

One tail per user (or API key) per agent: opening a second one closes the first with a replaced event, so a reloaded browser tab never fights its predecessor. The tail holds its delivery watermark one poll behind the newest committed id: each poll observes max(id) and delivers only up to the id the poll before it observed, so a batch that took lower ids and committed later is still inside the next window and is delivered exactly once, in id order. The cost is one extra poll interval before a line appears (two polls, about 4 s at the default 2 s interval), and a Last-Event-ID resume replays after the same delay; keepalives keep their 15 second cadence, so an idle connection still shows liveness. The one line that can still be missed is one whose ingest transaction stays open longer than a poll interval between inserting its rows and committing, which is why the rows are inserted last in that transaction, after the object put and the chunk row. Search remains authoritative for anything the tail did not show.

Context and permalinks

GET /v1/logs/{lineId}/context?before=&after= (0..200 each, 20 by default) answers the line and its neighbours within the same run in chronological order, whatever filters produced the hit. The dashboard's "Context" action and Enter on a selected line use it; ?line=<id> on the agent logs page opens the line with 100 lines around it, and ?line=n<ordinal> on a run page opens the Logs tab scrolled to that line of the chunk stream.

Surfaces

• API: GET /v1/agents/{id}/logs, GET /v1/agents/{id}/logs/tail, GET /v1/logs/{lineId}/context, and GET /v1/runs/{id}/logs with the same filters (HTTP API).
• CLI: keel logs <agent|run-id> [--since] [--until] [--grep] [--level] [--replica] [--stream] [--follow] [--limit] [--json] (The CLI).
• Dashboard: the Logs tab of a run and the agent logs page (The dashboard).