Idempotent Recovery Loop
Recovery should reconcile authoritative intent against actual external or derived state, then apply only repeatable corrections until the system converges. If running recovery twice is dangerous, the first run was not safe enough.
Problem
Recovery often starts where the happy path stopped: a process crashed, a dependency failed, an external system kept some state, a queue lost work, or an operator restarted a service halfway through cleanup. If recovery is written as a one-time script, every interruption creates a new incident.
Context
Use this pattern when a system has external effects, derived state, enforcement state, delayed work, crash recovery, dependency reconnection, or operational cleanup. It is especially important when the external system can outlive the process that created the effect.
Forces
- External state can survive process failure.
- Operators may need to run recovery more than once.
- Retrying a non-idempotent effect can create duplicate damage.
- Logs explain what happened, but they are not always the safest source to reconcile from.
- A deployment lock or manifest may be the most honest record of intended state.
- Recovery needs enough observability to say what changed.
- The system must converge toward intended state without requiring incident-specific memory.
Solution
Make recovery a repeatable loop: read authoritative intent from the source of record, read actual external or derived state, compute the difference, apply only idempotent corrections, record what changed, and repeat safely until actual state matches intended state or a safety boundary stops progress. Where recovery might reattempt an external effect, bind that effect to an idempotency key so the second attempt can be recognized as the same intended correction. If a dependency keeps failing, open a circuit breaker and expose the degraded state rather than retrying indefinitely. The retry policy should be explicit enough that recovery can be run under pressure without inventing new rules during the incident.
Consequences
Recovery becomes a procedure instead of a ritual. Operators can run it under pressure without protecting the system from the recovery tool. The system converges toward intended state rather than accumulating side effects from every retry.
Failure Modes
- Recovery replays logs instead of reconciling against current authoritative intent.
- The cleanup step cannot run twice safely.
- Multi-step mutation has no transaction or rollback boundary.
- The external system is treated as truth because it is the only surviving state.
- A half-completed recovery run leaves no record of what changed.
- The retry loop hammers a failing dependency instead of opening a visible degraded state.
Proof Points
firewall state; recovery removes orphaned rules and reapplies missed bans idempotently.
and delayed work from truth rather than treating acceleration state as authoritative.
- CapBan — active bans can be reconciled against actual
- Ampriot — workers rebuild queues, locks, projections,
Full source pattern: idempotent-recovery-loop.md.