Services and APIs

An agent in service mode is an always-on daemon: a Telegram or Slack bot, a long-lived gateway, an API, anything that must stay up. Keel keeps one run of it alive per replica, restarts a replica when it crashes, keeps its working directory on a persistent volume, and bills every replica as a flat replica-month pro-rated by the clock hour, at one price whatever it runs: Agent.serviceKind is agent (an AI agent or bot, the default) or app (a regular backend such as an API, a web app or a worker), and the kind now describes the workload rather than setting the price. Runs of service agents are ordinary runs with a service trigger source, no deadline, and a reconciler instead of a trigger. A service with ingress can run several replicas behind its hostname, a new revision is rolled out one replica at a time without dropping a request, and a replica that fails its health check is replaced while the others keep serving.

Model

• Logs of every replica flow through the same ingest path as run-mode logs and are searchable across the replicas of a service for seven days, with the replica index on every line: GET /v1/agents/{id}/logs, keel logs <slug> --replica 1 and the agent logs page in the dashboard (Logs).

• agents.mode is run or service. A service agent's runs are created only by the reconciler (triggerSource = service); a manual, scheduled or webhook trigger on a service agent is refused with agent_mode_mismatch, and a service trigger on a run mode agent likewise.

• agents.service_kind (migration 0028, agent or app, default agent; domain Agent.serviceKind, serviceKinds) only means something in service mode and decides the price of every replica (Billing). It is set on POST /v1/agents (serviceKind) from service.kind in keel.yaml and changed with PATCH /v1/agents/{id}; UpdateAgent refuses the change with service_kind_locked while the service is up or any service run of the agent is active (serviceIsActive), so a billing hour never straddles two prices. Every PATCH is in the audit log like any other mutating request.

• A service run has ceilingSeconds = 0 and deadlineAt = null. The supervisor runs without a deadline timer, the reaper never times it out for its deadline or its queue wait, and the gateway binding for it expires a year out instead of at the deadline (serviceBindingTtlSeconds). Dispatch, provisioning and lost handle detection still apply, from the attempt's dispatch time (run_attempts.created_at).

• service_status (migration 0017, one row per agent, RLS like every tenant table) carries the reconciler's view: state, current_run_id, restarts, last_exit_code, last_exit_at, next_restart_at, up_since, and since migration 0026 desired_replicas, ready_replicas, generation and the rollout columns (rollout_revision_id, rollout_started_at, rollout_failed_at). The domain type is ServiceStatus (service_status.readyReplicas and service_status.desiredReplicas are the public seams other workstreams read), the port ServiceStatusRepository on TenantRepositories.serviceStatus.

• service_replicas (migration 0026, one row per (agent_id, replica_index), RLS like every tenant table) carries the per replica restart accounting: run_id (the run occupying the index), claimed_at (a start in flight), restarts, last_exit_code, last_exit_at, next_restart_at, up_since. Domain type ServiceReplica, port ServiceReplicaRepository on TenantRepositories.serviceReplicas. Service runs carry runs.replica_index and runs.service_generation; readiness lives per attempt on run_attempts.ready_at and last_ready_at.

• Desired state (serviceDesiredState): a live service agent wants to be up; paused or archived wants down. The desired replica count is agents.desired_replicas when set (POST /v1/agents/{id}/scale), else manifest.service.replicas, else 1 (desiredReplicasFor).

States and the transitions the reconciler and the ingest path take:

Replicas and health checks

The manifest's service block (packages/domain/src/agents/manifest.ts, validated by runtimeManifestSchema, written from keel.yaml) declares:

service:
  replicas: 2
  healthCheck:
    path: /healthz
    port: 8080
    intervalSeconds: 10
    timeoutSeconds: 5
    unhealthyThreshold: 3
    startPeriodSeconds: 30

replicas defaults to 1 and is clamped by clampManifestToPlan to plan.limits.maxReplicasPerService (Free 1, Pro 3, Enterprise 10). healthCheck is optional; port defaults to the ingress port (and is required when the service has no ingress, the manifest schema refuses it otherwise), path is a plain request target (printable ASCII, no spaces), and the other fields default to the values shown. Each replica is billed as one service (Billing).

Readiness is decided by the supervisor and reported on the ingest heartbeat as ready: boolean (heartbeatReadyField in @keel/runtime-contract; readiness on the heartbeat contract) and, for ingress services, on the tunnel's control channel as stats.ready:

• With a healthCheck the supervisor sends GET <path> to the agent container over the shared network (KEEL_HEALTH_CHECK, JSON with the port filled in by the executor) every intervalSeconds with timeoutSeconds; any 2xx or 3xx is a pass. The replica is ready after the first pass; with ingress the pass only counts once the tunnel is connected, so the reconciler never drains the old replica before the relay can route to the new one. Misses during the startPeriodSeconds after the run started do not count; afterwards unhealthyThreshold consecutive misses make it not ready, and one pass makes it ready again.
• With ingress but no healthCheck, readiness is "the tunnel is connected and the ingress port accepts a TCP connect", probed with the default interval, timeout and threshold.
• Without either, a replica is ready as soon as its agent container runs.

RecordHeartbeat stores the verdict per attempt: ready_at is set when the replica becomes ready and cleared when it stops being ready, last_ready_at remembers that it was ready once. From those the reconciler sees each active run as launching (never ready), ready, unready (was ready, now failing) or draining, and service_status.readyReplicas counts the ready replicas of the current revision and generation.

Rollouts, scaling and replica restarts

The rollout state machine is a pure domain function, planReplicas in packages/domain/src/services/rollout.ts. Given the desired count, the active runs (index, current or stale, phase, since when, drained) and the per replica holds it returns orders, which the reconciler applies:

• A stale run is one whose revision is not the agent's current one or whose service_generation is not the status generation (a restart bumps the generation, see below).
• Rolling deploy: while stale runs exist the planner starts one replica of the current revision at a time, waits until it is ready (bounded by startPeriodSeconds + 60 s, readinessDeadlineMs), then drains the stale run at the same index, stops it once drained (service_superseded) and moves to the next index. A ready replica of the old revision keeps serving throughout; the surge is one run slot. Without a free concurrency slot a deploy never stops the last ready replica: the reconciler reports concurrency_exhausted every tick, the old revision keeps serving with rollout set and state restarting, and the rollout proceeds as soon as a slot frees (planReplicas input replaceWithoutSurge is false). A stale run that is not the last ready one (another ready replica exists, or it never became ready) is drained and stopped first (service_restart) so its slot goes to the new replica. Only an explicit restart (POST /v1/agents/{id}/service/restart, every stale run is on the current revision, so the user asked for the replacement) may drain and stop the last ready replica without a slot for the surge.
• A new replica that is running but misses its readiness deadline while an old one still serves aborts the rollout: the new run is stopped (canceled with health_check_failed, stop reason rollout_aborted), the old replicas keep serving, service_status.rollout.failedAt is set and ServiceNotifier.serviceRolloutAborted fires once. The reconciler does not retry that revision (rollout_failed_hold) until a new revision is deployed, the service is restarted or the old replicas are gone.
• Scale up starts every missing index that a free concurrency slot allows (concurrency_exhausted names the shortfall and retries on the next tick); scale down drains the indexes above the desired count and stops them once drained (service_scaled_down), never touching the ones that stay. POST /v1/agents/{id}/scale refuses a count beyond the plan or beyond the organization's free concurrentRuns before anything is stored.
• An unhealthy replica (unready) is stopped with health_check_failed, which the restart policy treats as a crash of that replica alone: its service_replicas row gets the backoff and the count, the other replicas are not touched. A replica that never became ready within the deadline and has nothing to fall back to is treated the same way.
• Draining (the ingress notes): the reconciler marks the run's tunnels draining in ingress_tunnels (ServiceTunnelDrainer, implemented by IngressTunnelDrainer over the ingress binding tables); the relay stops routing to them and records drained_at once nothing is in flight; the reconciler stops the run when every tunnel drained or after drainTimeoutMs (30 s). A run without tunnels is stopped at once.
• Restart (POST /v1/agents/{id}/service/restart, RollingRestartService) bumps service_status.generation; every run of the old generation is stale, so the same rolling machinery replaces the replicas one by one on the same revision.

service_status.currentRunId points at the lowest index replica of the current revision, and the run event stream of that run carries the service events as before.

Reconciler

ReconcileServices runs as the service-reconciler worker role every 15 seconds, and ChangeAgentLifecycle and CreateRevision invoke it for the changed agent through the optional onAgentChanged listener (AgentChangeListener), so a pause, resume, archive or deploy takes effect without waiting for the tick. It reads the platform-wide ServiceDirectory (service agents and active service runs, one query each) and then works per agent inside a tenant transaction that locks the status row.

For every service agent:

• Desired down: every active run is stopped through stopActiveRun (a canceled outcome with stop reason service_stop), the status becomes stopped, and for an archived agent the workspace is removed and the status row deleted so the agent leaves the directory.
• Desired up: the reconciler builds the replica view (buildReplicaSet in packages/application/src/services/replica-view.ts: the active runs with their attempts' readiness, the drain state of their tunnels, the replica rows and their holds), asks planReplicas for orders and applies them: stops through stopActiveRun (canceled for superseded and scaled down runs, failed with health_check_failed for unhealthy ones), drains through the tunnel drainer, and starts through CreateRun with the replica index and generation on the run. Starts pass the start gates first, in order: workspace over its allowance, maxServices (active services of other agents in the organization; 1 on Free, unlimited on Pro and Enterprise), the subscription gate (BillingAccessPolicy), and the spend cap projection with a one hour service reservation (decideAdmission with serviceReservation: a slot plus one hour at the hourly service price, no run minutes). A closed gate leaves the status untouched and is reported as the gate's name. Otherwise each started index is claimed on its service_replicas row (claimed_at), the workspace is ensured, the run is created and recorded on the row; a failed create releases the claims with a five minute hold, and a claim older than two minutes with no run is treated as stale. A running replica of the current revision whose row has no owner (the reconciler died between creating the run and recording it, or a duplicate at the same index was drained) is adopted by its row on the next tick (adoptUnownedRuns) so its crashes and stable uptime keep being accounted. Replicas whose next_restart_at is in the future are held (held); a service level hold (service_status.next_restart_at, crash loop or gated stop) holds every start but drains and stops still complete. The status is then derived from the runs: up when a current replica is running, starting while the first ones launch, restarting while a rollout runs with nothing new up yet or a crashed replica waits for its backoff, plus desiredReplicas, readyReplicas and rollout.
• Cases reported per agent (ReconcileAction.case): start, rolling_restart_start, restart_stop, superseded_stop, draining, scale_down, waiting_readiness, rollout_aborted, rollout_failed_hold, unhealthy_restart, concurrency_exhausted, held, crash_loop_hold, marked_up, noop, the gate names, stop, workspace_removed and not_a_service.

Restart accounting lives in the domain (packages/domain/src/services/restart-policy.ts) and is applied by finalizeAttempt (recordServiceExit) when a service run reaches a terminal state. It is kept per replica; the hold is per service:

• A crash is any agent exit, including exit 0 (a service is not supposed to exit) and a health_check_failed stop: the replica row that owns the run gets restarts + 1, lastExitCode, lastExitAt and nextRestartAt with exponential backoff 5 s * 2^(restarts - 1) capped at 5 minutes (recordReplicaCrash). The count resets once that replica stayed up for ten minutes (upSince). The service row mirrors the sum of the replica counts, the last exit, and goes to restarting only when no other replica is running (recordServiceCrash); while others run it stays up and only the crashed index waits.
• A replica's fifth consecutive crash enters the whole service into crash_looping with a fifteen minute hold on service_status.next_restart_at; the running replicas keep running, no replica is started, the reconciler notifies once (ServiceNotifier.serviceCrashLooping) and the looping replica starts fresh with restarts = 0 after the hold.
• Platform stops are not crashes: platform_lost, dispatch_lost, provisioning_timeout, image_pull_failed, platform_error, a user or reconciler cancel (superseded, scaled down, paused). The replica row frees its run with no hold, so a lost replica restarts on the next tick. A stop for spend_cap_exceeded or subscription_inactive is also not a crash but carries a five minute hold on the replica and the service so the gate is not hammered.
• A terminal report for a run that no replica row owns (the superseded half of a rollout, or an aborted rollout's new run whose index was handed back to the old run) does not touch the accounting.

Workspaces

manifest.workspace = { enabled, mountPath, sizeGb } mounts a per-agent volume at mountPath in the agent container and, at the same path, in the supervisor. clampManifestToPlan refuses a size above plan.limits.workspaceGbPerAgent (workspace_exceeds_plan), refuses ingress without the plan feature, and refuses service mode without plan.features.services; these are typed errors on CreateRevision, nothing is silently cleared.

• Docker: a named volume keel-ws-{agentId} (DockerWorkspaceStore), created before the first start and bound into both containers. It survives runs and restarts; archiving the agent removes the agent's finished containers and the volume.
• Fargate: one EFS file system per organization, created by the workspace provisioning step (EfsTenantWorkspaceProvisioner: encrypted with the tenant key, elastic throughput, one mount target per run subnet in the workspaces security group, a file system policy that admits access point mounts over TLS and denies plaintext), recorded as organizations.workspace_file_system_id. One access point per agent (EfsWorkspaceStore: uid and gid 1000, root /agents/{agentId} created 0750), recorded as agents.workspace_access_point_id. The task definition declares a keel-workspace volume with efsVolumeConfiguration (transit encryption on, IAM authorization off because Keel tasks have no task role) mounted read-write in the agent and the supervisor. Archiving deletes the access point; the directory stays on the file system until a retention job removes it.
• Size: EFS cannot cap a directory, so the supervisor reports workspaceBytes (a bounded walk of the mount path, sampled once a minute) on every heartbeat and RecordHeartbeat stores it on agents.workspace_bytes. The settlement job compares it with the manifest allowance, and on Fargate the file system's metered size with the sum of the organization's allowances, notifies once per process (ServiceNotifier.workspaceOverLimit), and the reconciler refuses new starts of an over-limit agent (workspace_over_limit) while a running one is left alone.

Terraform (infra/terraform/modules/workspaces) provides the security group, the API role rights, and the environment variable WORKSPACE_SECURITY_GROUP_ID; with it set and RUN_SUBNET_IDS present, createAwsProvisioners returns a workspaceProvisioner and provisioning runs the extra step.

Supervisor

• KEEL_DEADLINE_AT is optional; the registration response's deadlineAt may be null. Without a deadline the deadline phase never fires.
• KEEL_WORKSPACE_PATH makes the heartbeat carry workspaceBytes (the field name is heartbeatWorkspaceBytesField in @keel/runtime-contract).
• Every 10 s the supervisor samples the agent container's CPU, memory and network (from the executor's AGENT_STATS file on Docker, from the task metadata stats on Fargate; KEEL_CPU_UNITS is the allocation CPU is scaled by) and batches up to six samples on the next heartbeat under metrics (heartbeatMetricsField). RecordMetrics stores them as minute points on the run's replica; Metrics.
• KEEL_HEALTH_CHECK (JSON { path, port, intervalSeconds, timeoutSeconds, unhealthyThreshold, startPeriodSeconds }, set by both executors from the manifest for service launches, the port already resolved against the ingress port) starts the readiness prober (apps/supervisor/src/readiness); without it an ingress service probes its port over TCP. Every heartbeat carries ready, a change of readiness sends a heartbeat at once, and the tunnel's periodic stats carry ready too, so the relay learns about it without waiting for the platform.
• On SIGTERM the supervisor first drains its tunnel (announces ready: false on the control channel and waits up to ten seconds for its in-flight requests to finish), then writes /keel/control/STOP and waits up to KEEL_STOP_GRACE_SECONDS (default 30) for the agent to exit before it reports stopped. An agent that wants a clean shutdown polls that file. The Docker executor stops the supervisor first for service launches (label keel.service=true) and gives it the grace plus its own stop timeout; on Fargate every container receives SIGTERM at once and the agent's stopTimeout of 30 seconds is the grace.

Billing

See Billing, section "Services are billed per replica-month". In short: a service costs pricing.servicePriceCentsPerMonth (79 EUR) per replica, pro-rated by the hour it is up, and the steps its runs record count against the per-agent step allowance. Dispatch reserves one concurrency slot (serviceReservationFor(agent.serviceKind)), the service-settlement role writes one service kind usage record per completed clock hour (usage_records.window_start, usage_records.service_kind) and adds its hours to org_quota_state.period_service_hours or period_app_service_hours while the slot stays taken, terminal writes the last partial window and frees the slot, the rollup reports the hours on the service_hours or app_service_hours meter, and when the settled hours push the spend projection over the cap every running service of the organization is stopped with spend_cap_exceeded and notified.

Metrics

Each replica's CPU, memory and network and every request through its hostname are recorded as metric points and served on GET /v1/agents/{id}/metrics, in the dashboard's Metrics section and by keel service metrics. Restart markers come from the service runs that crashed. Metrics has the measurements, resolutions, retention and write volume.

Data stores

A service reaches a managed Postgres or Valkey store through an organization secret: keel db create main writes DATABASE_URL (or REDIS_URL), the service lists that name under secrets: in keel.yaml, and the next deploy or keel service restart injects it as an environment variable. The database is only reachable from the agent network at db.internal, so nothing changes in the egress policy. Rotating credentials rewrites the secret but does not restart the service. Details in Databases.

Environment variables

Beyond the manifest's environment and secrets, every agent carries its own editable set of variables (Environment and secrets): service_env_vars rows, secrets in the tenant vault under agent/<agentId>/<NAME>, edited through PUT /v1/agents/{id}/env, keel env and the agent page. DispatchRun reads the set inside the admission transaction, stores its hash on runs.env_hash and hands it to the executor as LaunchInput.serviceEnvironment; both executors merge it with mergeRuntimeEnvironment (manifest environment, then per-agent plain values, then manifest secrets, then per-agent secrets). A change never touches a running replica: GET /v1/agents/{id}/env reports restartRequired when an active service run was dispatched with a different hash, and POST /v1/agents/{id}/service/restart rolls the replicas onto the current set. Archiving the agent purges the rows and the vault entries.

Operations

• Worker roles: service-reconciler (15 s), service-settlement (5 min) and metrics-rollup (5 min, rolls minute metric points up and expires them; Metrics). Both compose the same executor as the dispatcher and reaper; the reconciler also needs a CreateRun and the dispatch queue. domain-verifier (1 min) rechecks the DNS of custom domains (Custom domains) and needs INGRESS_PUBLIC_DOMAIN but no executor. data-store-settlement (5 min) and data-store-purge (1 h) follow the same shape for managed data stores (Databases).
• Environment: WORKSPACE_SECURITY_GROUP_ID (API and worker, Fargate only). Everything else reuses the run task settings.
• Stop reasons on run_attempts.stop_reason: service_stop (pause or archive), service_superseded (replaced by a ready replica of the new revision or generation), service_restart (replaced without overlap because no slot was free and either another ready replica kept serving or the user asked for a restart), service_scaled_down, health_check_failed (unhealthy replica), rollout_aborted (new replica never became ready), spend_cap_exceeded.
• Rollout and readiness limits: readiness deadline startPeriodSeconds + 60 s, drain timeout 30 s (drainTimeoutMs), one surge slot per rollout. The relay's drain poll interval is INGRESS_DRAIN_POLL_MS (default 1000).

How the API binds to this layer

apps/api/src/composition/service-ports.ts builds one ReconcileServices and one SettleServiceUsage per process from the run ports (executor, terminal listeners, billing access, dispatch queue) and hands them out three ways: as the ServicesPort behind /v1/agents/{id}/service/*, as the onAgentChanged listener of ChangeAgentLifecycle and CreateRevision, and as the in-process loops that REAPER=inprocess starts next to the reaper (SERVICE_RECONCILE_INTERVAL_SECONDS, SERVICE_SETTLEMENT_INTERVAL_SECONDS). The worker composes the same use cases for the service-reconciler and service-settlement roles.

• Status: TenantRepositories.serviceStatus.find(agentId) returns the ServiceStatus; currentRunId points at the run whose errorClass and latest attempt's stopReason explain a stop.
• Replicas: ServicesPort.replicas (replicaViewsFor in apps/api/src/composition/replica-views.ts) joins the active runs, their attempts and the service_replicas rows into one entry per index with state, ready, runId, revisionId, upSince, restarts and the hold. ServicesPort.scale runs ScaleService (plan and concurrency checks, then agents.desired_replicas and onAgentChanged with change: "scale"); restart runs RollingRestartService (change: "restart").
• Start: a paused agent is resumed (ChangeAgentLifecycle with resume), which reconciles through onAgentChanged; a live one goes straight to ReconcileServices.reconcileAgent. A closed start gate comes back as service_start_gated with the gate name, except max_services_reached, which the API reports as service_limit_reached with the plan limit. The reconciler is the only place maxServices is counted.
• Stop: pause the agent (ChangeAgentLifecycle with pause), same hook; the desired state of a paused agent is down, so the reconciler stops the run with service_stop.
• Restart: RollingRestartService bumps the generation and reconciles; the old replicas are drained one at a time as the new ones become ready.
• Workspace usage: agents.workspaceBytes (last heartbeat) against manifest.workspace.sizeGb; on Fargate EfsWorkspaceStore.meteredBytes(organizationId) for the organization total. The heartbeat's workspaceBytes goes to RecordHeartbeat.
• Services on GET /v1/organizations/me/usage: services.agent and services.app each carry hours (the sum of billable_seconds over the period's service kind usage records of that kind, in hours), running, projectedCents (the hours at the hourly share of that kind's pricePerMonthCents) and pricePerMonthCents; services.hours, running and projectedCents are the totals and services.pricePerMonthCents the agent price. GET /v1/agents/{id}/service carries pricing: { serviceKind, pricePerReplicaMonthCents } and GET /v1/services items carry serviceKind.
• Archive: ChangeAgentLifecycle revokes the ingress binding, and the reconciler revokes it again, removes the workspace and deletes the status row when it sees the archived agent; both paths are idempotent. On Docker the workspace store stops any container that still holds the volume before removing it, so the archive request and the reconciler loop do not fight over the volume.