Managing runs
Operators can use Foliot Console to:
- Search runs by business key, run id, workflow, status, step, or tag.
- View run details to examine each, see error messages, and other details.
- Manage actions such as redrive, rewind to a step (and another version), skip, edit state, send a signal, or cancel.
- Pause a step across all versions and then resume at a chosen rate to help an API recovering after an incident.
- Drive bulk actions using filters with a mandatory dry run. Filters include “runs where step X completed between T1 and T2”, which is how you can identify runs impacted by a bad deploy, including runs that have since succeeded.
- Track Health for each workflow including counts by status, a step heatmap, failed and succeeded in 24 hours, p50 and p95, and the age of the oldest due run.
Metrics
Foliot does not send alerts on metrics itself. Instead, your monitoring tool can read Foliot’s metrics through the API and create alerts.
To see every metric Foliot reports, with its unit, its labels and whether it keeps history, call GET /metrics/catalog.
Two useful signals include:
oldestDueAgeMsfromGET /apps/{appId}/workflows/{workflow}/health: How long the oldest run that is due has been waiting to be picked up. If it keeps increasing, runs are most likely not being processed. Foliot Console shows the same number on each workflow’s health.foliot_step_failures_totalfromGET /workspaces/{workspaceId}/metrics/query, labelled by app, workflow and step: A rising failure rate for one step usually means a service or a handler it depends on is failing.
Choosing how to intervene
| Situation | Use |
|---|---|
| Stop it permanently; it should not have run | cancel |
| It failed or is stuck, and the cause is now fixed | redrive |
| It finished, or you must go back further than the current step | rewind |
| This step was done out of band; carry on past it | skip |
| A handler recorded a wrong value that a later step reads | edit state |
Redrive and rewind
Every task Foliot sends to your handlers includes two keys (ctx.attemptKey and ctx.stepKey when using the SDK):
attemptKeyis unique to one attempt. Foliot never sends the sameattemptKeytwice. When it cannot tell whether a try finished (your reply was lost or received afterrespondByor not completed by thedeadline), it retries with a newattemptKey.stepKeyis the same for every attempt of one entry into the step. You can use it for work that must happen once per step, however many attempts it might take (see Create handlers). A newstepKeyis used if the run comes back to a step due to a loop in your workflow, a redrive, or a rewind.
An attemptKey consists of {runId}:{entry}:{step}:{attempt}, so a key r1:3:charge-card:1 represents run r1, entry 3, step charge-card, and attempt 1:
entrygoes up by one every time the run enters a step, any step, so it changes on a loop, a redrive or a rewind.attemptcounts the tries within one entry, starting at1.
The stepKey has the same structure without the attempt portion: r1:3:charge-card.
Treat both as opaque strings: you can compare them but don’t rely on their format as it can change. Say a run r1 has just entered the step charge-card. Here is what your handler receives each time:
| What happens | attemptKey |
stepKey |
What it means for your handler |
|---|---|---|---|
| Foliot sends the task | r1:3:charge-card:1 |
r1:3:charge-card |
Charge the card. You can use the stepKey as the payment provider’s idempotency key. |
Your reply was lost or came after respondBy so Foliot tries again |
r1:3:charge-card:2 |
r1:3:charge-card |
A new attempt of the same step while the first may still be running. Payment provider sees the same stepKey and returns the first charge instead of making a second. |
| A proxy delivers that same request twice | r1:3:charge-card:2 |
r1:3:charge-card |
The same request received twice and not a new attempt. Your handle should answer the copy as it answered the first. |
| The second try returned an error so Foliot tries a third time | r1:3:charge-card:3 |
r1:3:charge-card |
Still the same stepKey so the card is charged at most once across all three tries. If an earlier try already charged it, the provider returns that charge; if not, the card is charged now. |
Someone redrives or rewinds the run, or your workflow loops back to charge-card |
r1:4:charge-card:1 |
r1:4:charge-card |
A new entry into the step so a new stepKey is provided. Basically means your workflow is asking for the step to be done again, so do the work. |
A slow answer for r1:3:… arrives after the redrive |
— | — | Foliot refuses it so it can’t overwrite the new result. |
The same request can reach your service twice, for example if a proxy resends it. Both copies carry the same attemptKey and the second can arrive while the first is still being handled. So record the attemptKey when you start the work, not when you finish it:
- A copy whose key has finished: answer it as you answered the first.
- A copy whose key is still in progress: refuse it with
503and aRetry-Afterheader. Foliot tries again later as a new attempt with the samestepKey, and the refusal does not use up one of the step’s attempts.
Anything you key on stepKey, such as the payment itself, still happens once either way.
For work that must happen only once for the whole run, whether it is redriven or rewound, give your provider your own idempotency id, such as the order id.
What a rewind does. It takes the run back to a step it has already reached and runs forward from there:
- Records a
run.rewoundevent in the run’s history. - Discards the output of that step and of every step that started after it.
- Cancels any task still in progress, so a late answer to it is refused.
- Runs the step again, then carries on as usual.
Nothing is deleted from the run’s history: the earlier attempts stay visible. A run can be rewound for 90 days after it finishes, while Foliot keeps its full history. After that, it is reduced to a summary and can no longer be rewound.
A rewind only goes back. It must name a step the run has already started. A step further ahead or on a branch the run did not take is refused. To move a run forward, skip its current step instead. A bulk rewind changes only the matching runs that have started that step and its dry run tells you how many it will not act on.
What a rewind cannot undo. A rewind runs steps again, but two things do not change:
-
Work your handlers already did. Say a run charged a card at
charge-cardthen failed atcreate-shipmentand you rewind it tocharge-card. The step runs again, and unless your handler is careful, the customer is charged twice. There are two ways to stay safe:- Check before acting. Have the handler check first, for example “is order 7 already charged?”, and return the earlier result if so. Use your own data, such as the order id, rather than a key from Foliot, because a rewind gives the step new ones.
- Undo with a compensation step. A compensation step reverses what an earlier step did, such as
refund-cardforcharge-card, orrelease-stockforreserve-stock. Make it a step’sonFailureaction so it runs when that step fails and nothing is left half-done.
-
Signals the run already received. Say a run waits at
shippeduntil your warehouse sends theshippedsignal with theidempotencyKeyshipment-123, then moves on. If you rewind it to a step beforeshipped, it waits atshippedagain: Foliot does not reuse the signal it already received.Sending
shipment-123again does nothing, because each key counts only once per run. Sendshippedwith a new key instead, such asshipment-456, from your service when the order really ships again, or by hand from the run’s page in Foliot Console.
Generations in Foliot Console. Foliot Console shows how many times a run has been redriven, rewound or skipped as its generation (generation 1 on the run, g1 beside the history events that came after). Foliot uses it to ignore a late answer meant for the run (see Generations).