Operating Foliot
Foliot Console (apps/console) is built for the person on call.
- Search by business key first, then run id, workflow, status, step, tag.
- Run timeline: every attempt, error text verbatim, who did what.
- Actions: redrive, rewind to a step (optionally onto a newer version), skip, edit state, send a signal, cancel.
- Pause a step across all versions; resume at a chosen rate so a recovered API is not stampeded.
- Bulk actions by filter with a mandatory dry run. Filters include “runs where step X completed between T1 and T2”, which is how you find everything a bad deploy touched, including runs that have since succeeded.
- Health per workflow: counts by status, a step heatmap, failed and succeeded in 24 hours, p50 and p95, and the age of the oldest due run.
Metrics and the two alerts that matter
Prometheus metrics are at /metrics. Two of them are worth an alert:
foliot_oldest_due_age_seconds: the scheduler is behind or down.- the per-step rate of
foliot_task_failures_total: a downstream dependency is failing.
Two histograms show where run state is heading, which every later step is sent:
foliot_step_output_bytes(by workflow and step): each step’s output size, measured before the per-step cap is applied, so the steps over it are visible too. The cap, 256 KiB by default, is a bucket edge, so the share at or under it is a single bucket.foliot_run_state_bytes(by workflow): the whole state after each write to it: a step’s output, a delivered or buffered signal, a choice, an operator’s skip, or an operator’seditRunState. The total is not capped, and arun state is largewarning is logged when a write passes the cap, naming the run, workflow and the step whose write tipped it over. An operator’s edit has no writing step, so its warning carriesedit=true, the operator (actor) and the keys it patched (keys) in place ofstep.
Choosing how to intervene
| Situation | Use |
|---|---|
| Stop it permanently; it should not have run | cancel |
| It failed or is stuck, and the cause is now fixed | redrive |
| It finished, or you must go back further than the current step | rewind |
| This step was done out of band; carry on past it | skip |
| A handler recorded a wrong value that a later step reads | state edit |
Generations, redrive and rewind
Every run carries a generation counter. Redrive and rewind increment it, and each task’s idempotency key is runId:generation:step:attempt. So ordinary retries within a generation dedupe on the app side, while an operator redrive always executes fresh code, and a late completion from an old generation is rejected.
Rewind appends a run.rewound event, removes from state the output of the target step and every step that started after it, invalidates open tasks, and runs forward again. History is never deleted. Rewind works while full history is retained: 90 days by default, after which finished runs are compacted to a summary row.
Two things rewind does not undo. It cannot take back a side effect a downstream step already had (an email sent, a card charged): steps like that need a compensation step or a handler that checks before acting. And it does not replay consumed signals: a signal’s dedupe key is spent for the life of the run, so a rewound run that waits again needs a fresh signal. A real re-shipment produces a new event; an operator can send one from Foliot Console.