Making long-running workflows survive process failure

Summary

During a review of a workflow engine, I asked what would happen to a workflow that was two days into a delay when we deployed the backend. Nobody had a clean answer. That worried me.

The builder already supported triggers, conditions, loops, AI steps, API calls, multi-day delays, approval gates, and execution logs. It looked complete from the UI. Underneath it, every continuation lived inside an application process, so a deploy would kill every active delay. The engine depended on one Node process staying alive indefinitely.

I found the failure mode during implementation review and pushed the runtime toward durable execution: persisted step state, waits that outlived a process, approval checkpoints, duplicate-safe resume claims, and per-step execution records.

What I asked during review, and what it exposed

The early implementation used asynchronous JavaScript, basically setTimeout-style waits. The timers did not block the event loop or burn OS threads. Their state simply disappeared with the process.

The problem was where the continuation lived. "Wake execution X tomorrow at step B" existed only as a callback inside the application process. A normal deploy replaces that process, so every in-flight timer disappears with it. If a workflow was three days into a five-day wait, it was gone. The database knew that a workflow record existed, but it did not know that the workflow was waiting, when it should wake up, or what should happen next.

I sketched the deploy sequence during the session:

code push
   ↓
new image
   ↓
old application process replaced
   ↓
in-memory continuation disappears

The system could not survive a routine deploy without silently discarding business processes.

Why "it will resume from the same step" wasn't enough

A developer told me the workflow should resume from the step where it stopped after a restart. Fine as an intention. I asked how a new worker would know there was anything to resume, and the conversation got quiet.

Knowing that a workflow exists is not enough. The database also needs to know whether it can resume, when it becomes eligible, which step comes next, whether a worker has already claimed it, and whether the previous step's side effect already happened. Those details were missing. The team acknowledged the gap. Adding retries to the existing promise chain would not fix it. Execution progress had to become durable state instead of a position in an in-memory call stack.

Restarting the workflow from the beginning was not safe either. Say step 1 qualifies a lead, step 2 sends an email, and step 3 waits for two days. If the process crashes during step 3 and replays the whole workflow, that email goes out twice. Payments, webhooks, CRM writes, external API mutations, and expensive AI calls have the same problem. Once a workflow changes something outside itself, "start over" is no longer a recovery strategy.

From call stack to database row

The runtime I pushed for stored workflow progress in the database. The current step, wait status, resume time, approval state, and the awkward case where a step ran but did not confirm completion all became explicit fields instead of positions in a call stack.

The mental model shifted from "the worker owns the workflow" to:

durable execution state
        ↓
available worker claims next work
        ↓
execute step
        ↓
persist result / transition
        ↓
worker can disappear

A twenty-four-hour wait now meant a row with status = 'waiting' and resume_after = some_timestamp, not a function suspended for a day. Once that timestamp passed, a poller or scheduler sweep marked the execution as eligible. The next worker could then continue from the correct step. Human approvals used the same mechanism. We persisted the approval, made the execution resumable, and let any worker pick it up. In runtime terms, an approval is just a wait with no known duration and an external trigger.

Two workers, one row

Persisting execution state creates another problem: two workers can find the same resumable execution at the same time. Without coordination, both may run it. That wastes some CPU for a pure computation. For a webhook or card charge, it is a correctness bug.

The runtime needed duplicate-safe resume claims. Depending on the stack, that can mean a transactional UPDATE ... WHERE status = 'waiting' with an atomic status change, uniqueness constraints, leases, or version tokens. The important part is keeping "eligible to resume" separate from "claimed by this worker." Mixing those states is how duplicate side effects happen.

This is also why "put it on a queue" was not a complete answer. A queue can announce that work exists. It does not define the authoritative state, checkpoint behavior, duplicate handling, or replay rules. We still needed a state machine that understood side effects.

The crash-after-side-effect problem

The worst failure happens when a worker calls an external API, the API succeeds, and the worker crashes before saving "step completed." Another worker sees an unfinished step and makes the same call again. An email goes out twice. A customer gets charged twice. Retrying reliably does not help if the operation is not idempotent. It only makes the duplicate more reliable.

When the external system supports idempotency keys, the runtime can send a key built from the workflow run ID, step identifier, and logical operation. When it does not, you need an internal side-effect ledger, reconciliation, or duplicate detection in the domain itself. I've used all three. None is particularly pleasant. Persistence alone does not solve this failure mode.

Choosing the runtime

I evaluated dedicated orchestrators and persistence-oriented frameworks. They provide durable timers, distributed workers, scheduling, and recovery behavior that takes years to get right. They also add infrastructure, operational work, authoring constraints, and a migration path for every existing definition. A custom implementation gives you control, but now your team owns every failure case, including the ones where a wrong assumption charges a customer twice.

I've seen teams succeed and fail with both approaches. My requirement was simpler: a workflow could not depend on one application process staying alive. Temporal and a Postgres status column are very different engineering choices, but either can enforce that requirement.

What I check now

These are the cases I walk through with any long-running system:

• If a worker dies before starting a step, another worker should eventually claim it.
• If a worker dies after the external side effect succeeds but before marking the step complete, the system must avoid duplicating that side effect.
• If a deployment happens during a multi-day wait, the wait should remain in durable state and become eligible at the correct time, without being lost or resumed early.
• Duplicate wake signals must be idempotent.
• An approval submitted twice must not advance the workflow twice.
• If the workflow definition changes while a run is in-flight, the old run should execute against the version it started with.

Workflow products often get the builder, logs, and branching UI before anyone answers what survives a deploy. I now ask these questions during the first runtime review.

What shipped

The runtime now has persisted waits, approval checkpoints, duplicate-safe resume claims, failure-aware execution for unsafe mutations, and per-step state and logging. Its correctness no longer depends on one process staying alive. The broader workflow product was a team effort. I owned the failure analysis and the core durable-runtime semantics.

A deployment can replace the compute. It should never delete the business process that compute happened to be running.