← all features · evolve-loop explainers
Resilience
What happens when the machine dies, the binary vanishes, or every API is walled — and how each answer became automatic.
How it works today
- Per-run state isolation — each lane keeps cycle-state inside its own run dir; a crash cannot wedge the repo-level state. Recovery after the Jul-11 power loss found no root wedge: two finished lanes had closed out, one died mid-phase, its task simply returned to the queue.
- Boot orphan sweep (cycle 721, after a 704 attempt was itself killed by the cascade) — at loop boot, orphaned tmux sockets, stranded worktrees, and half-written state are swept, each leaving a bounded tombstone so cleanup is auditable and garbage cannot accumulate.
- Self-SHA provenance + auto-repin — ship verifies the binary’s hash against a pinned expectation; boot self-heals drift. Operator heal for the known clobber:
make -C go build && evolve reset-sha -operator.
- Rebase-and-re-audit on contention — lanes racing to land on
main hit Not possible to fast-forward; the orchestrator classifies it transient, rebases the lane branch, and routes back through audit so the verdict always matches the tree that ships. Queueing, not failure.
- Checkpoint-defer on quota exhaustion (cycle 740) — all CLI families walled → checkpoint, defer, resume after reset.
The night that exercised all of it
| Event | Mechanism exercised | Outcome |
|---|
| Machine power loss mid-fleet (Jul 11, 20:02) | Per-run isolation, queue re-pick | Zero lost tasks; cycle 698’s work re-landed as 702 |
| Shared binary deleted by a failed ship’s rollback | SELF_SHA verification (caught it), operator heal recipe | Cascade stopped in minutes; two defects filed, one already landed |
| Ten concurrent retro REPLs starving (width 10) | Failure-adapter verdict routing | Non-fatal classification; defect filed to decouple retro timeouts from cycle verdicts |
| Quota walls at full burn | Benches + min_lanes floor + (now) checkpoint-defer | Width held; walls became pauses |
The meta-pattern — every resilience mechanism on this page is a scar with a test attached. The fleet’s best fuel is its own incident log: several of tonight’s fixes were filed during the incidents and landed by the fleet within hours.
Generated by /evo:explain · sources: loop logs, cycle dossiers, ship commits, campaign plans.