ADR 0002: One Postgres database with SKIP LOCKED, not a message queue

Status: accepted · 2026-09-02

Context

Herd ran on 150+ sharded Oracle databases and then DynamoDB because it executes billions of runs a day. Tin Bee’s workloads are thousands of runs a minute at most. The scheduler’s core question is “which runs are due”, which is a time-indexed query, not a queue pop, and timers, waits and retries all reduce to setting scheduled_at.

Decision

runs is the engine. A partial index on scheduled_at for active runs, and one claim query:

UPDATE runs SET lease_owner=$1, lease_until=now()+lease
WHERE id IN (SELECT id FROM runs WHERE status IN (...) AND scheduled_at <= now()
             AND (lease_until IS NULL OR lease_until < now())
             ORDER BY scheduled_at LIMIT $2 FOR UPDATE SKIP LOCKED)
RETURNING *

Any number of scheduler replicas run it. A crashed worker’s lease lapses and the run becomes due again. Every state change is one transaction that locks the run row, so operator actions and the scheduler never interleave. No Redis, no NATS in the critical path.

Consequences

• One thing to operate, back up and read with psql.
• Ceiling is a single Postgres: tens of thousands of runs a minute, a few hundred million events. The retention job compacts finished runs so tables stay bounded.
• If the ceiling is reached: partition events by month, archive compacted runs. The engine does not change.