foliot docs

Guides / Managing runs

Managing runs

Operators can use Foliot Console to:

  • Search runs by business key, run id, workflow, status, step, or tag.
  • View run details to examine each, see error messages, and other details.
  • Manage actions such as redrive, rewind to a step (and another version), skip, edit state, send a signal, or cancel.
  • Pause a step across all versions and then resume at a chosen rate to help an API recovering after an incident.
  • Drive bulk actions using filters with a mandatory dry run. Filters include “runs where step X completed between T1 and T2”, which is how you can identify runs impacted by a bad deploy, including runs that have since succeeded.
  • Track Health for each workflow including counts by status, a step heatmap, failed and succeeded in 24 hours, p50 and p95, and the age of the oldest due run.

Metrics

Foliot does not send alerts on metrics itself. Instead, your monitoring tool can read Foliot’s metrics through the API and create alerts.

To see every metric Foliot reports, with its unit, its labels and whether it keeps history, call GET /metrics/catalog.

Two useful signals include:

  • oldestDueAgeMs from GET /apps/{appId}/workflows/{workflow}/health: How long the oldest run that is due has been waiting to be picked up. If it keeps increasing, runs are most likely not being processed. Foliot Console shows the same number on each workflow’s health.
  • foliot_step_failures_total from GET /workspaces/{workspaceId}/metrics/query, labelled by app, workflow and step: A rising failure rate for one step usually means a service or a handler it depends on is failing.

Choosing how to intervene

Situation Use
Stop it permanently; it should not have run cancel
It failed or is stuck, and the cause is now fixed redrive
It finished, or you must go back further than the current step rewind
This step was done out of band; carry on past it skip
A handler recorded a wrong value that a later step reads edit state

Redrive and rewind

Every task Foliot sends to your handlers includes two keys (ctx.attemptKey and ctx.stepKey when using the SDK):

  • attemptKey is unique to one attempt. Foliot never sends the same attemptKey twice. When it cannot tell whether a try finished (your reply was lost or received after respondBy or not completed by the deadline), it retries with a new attemptKey.
  • stepKey is the same for every attempt of one entry into the step. You can use it for work that must happen once per step, however many attempts it might take (see Create handlers). A new stepKey is used if the run comes back to a step due to a loop in your workflow, a redrive, or a rewind.

An attemptKey consists of {runId}:{entry}:{step}:{attempt}, so a key r1:3:charge-card:1 represents run r1, entry 3, step charge-card, and attempt 1:

  • entry goes up by one every time the run enters a step, any step, so it changes on a loop, a redrive or a rewind.
  • attempt counts the tries within one entry, starting at 1.

The stepKey has the same structure without the attempt portion: r1:3:charge-card.

Treat both as opaque strings: you can compare them but don’t rely on their format as it can change. Say a run r1 has just entered the step charge-card. Here is what your handler receives each time:

What happens attemptKey stepKey What it means for your handler
Foliot sends the task r1:3:charge-card:1 r1:3:charge-card Charge the card. You can use the stepKey as the payment provider’s idempotency key.
Your reply was lost or came after respondBy so Foliot tries again r1:3:charge-card:2 r1:3:charge-card A new attempt of the same step while the first may still be running. Payment provider sees the same stepKey and returns the first charge instead of making a second.
A proxy delivers that same request twice r1:3:charge-card:2 r1:3:charge-card The same request received twice and not a new attempt. Your handle should answer the copy as it answered the first.
The second try returned an error so Foliot tries a third time r1:3:charge-card:3 r1:3:charge-card Still the same stepKey so the card is charged at most once across all three tries. If an earlier try already charged it, the provider returns that charge; if not, the card is charged now.
Someone redrives or rewinds the run, or your workflow loops back to charge-card r1:4:charge-card:1 r1:4:charge-card A new entry into the step so a new stepKey is provided. Basically means your workflow is asking for the step to be done again, so do the work.
A slow answer for r1:3:… arrives after the redrive — — Foliot refuses it so it can’t overwrite the new result.

The same request can reach your service twice, for example if a proxy resends it. Both copies carry the same attemptKey and the second can arrive while the first is still being handled. So record the attemptKey when you start the work, not when you finish it:

  • A copy whose key has finished: answer it as you answered the first.
  • A copy whose key is still in progress: refuse it with 503 and a Retry-After header. Foliot tries again later as a new attempt with the same stepKey, and the refusal does not use up one of the step’s attempts.

Anything you key on stepKey, such as the payment itself, still happens once either way.

For work that must happen only once for the whole run, whether it is redriven or rewound, give your provider your own idempotency id, such as the order id.

What a rewind does. It takes the run back to a step it has already reached and runs forward from there:

  1. Records a run.rewound event in the run’s history.
  2. Discards the output of that step and of every step that started after it.
  3. Cancels any task still in progress, so a late answer to it is refused.
  4. Runs the step again, then carries on as usual.

Nothing is deleted from the run’s history: the earlier attempts stay visible. A run can be rewound for 90 days after it finishes, while Foliot keeps its full history. After that, it is reduced to a summary and can no longer be rewound.

A rewind only goes back. It must name a step the run has already started. A step further ahead or on a branch the run did not take is refused. To move a run forward, skip its current step instead. A bulk rewind changes only the matching runs that have started that step and its dry run tells you how many it will not act on.

What a rewind cannot undo. A rewind runs steps again, but two things do not change:

  • Work your handlers already did. Say a run charged a card at charge-card then failed at create-shipment and you rewind it to charge-card. The step runs again, and unless your handler is careful, the customer is charged twice. There are two ways to stay safe:

    • Check before acting. Have the handler check first, for example “is order 7 already charged?”, and return the earlier result if so. Use your own data, such as the order id, rather than a key from Foliot, because a rewind gives the step new ones.
    • Undo with a compensation step. A compensation step reverses what an earlier step did, such as refund-card for charge-card, or release-stock for reserve-stock. Make it a step’s onFailure action so it runs when that step fails and nothing is left half-done.
  • Signals the run already received. Say a run waits at shipped until your warehouse sends the shipped signal with the idempotencyKey shipment-123, then moves on. If you rewind it to a step before shipped, it waits at shipped again: Foliot does not reuse the signal it already received.

    Sending shipment-123 again does nothing, because each key counts only once per run. Send shipped with a new key instead, such as shipment-456, from your service when the order really ships again, or by hand from the run’s page in Foliot Console.

Generations in Foliot Console. Foliot Console shows how many times a run has been redriven, rewound or skipped as its generation (generation 1 on the run, g1 beside the history events that came after). Foliot uses it to ignore a late answer meant for the run (see Generations).