Prove an action is safe, then pace it within limits
Before a drain, reboot or rollout touches production, Raincurve checks it against live state and in-flight work — then schedules it so the fix does not cause the next incident.
Runbooks check the action. Raincurve checks the network.
An action that is safe in isolation becomes dangerous next to maintenance elsewhere, or after a device has already failed. Runbook checks ignore both — and even state-aware reachability checks miss the loss of backup paths.
Raincurve enforces a do-no-harm rule: after the action, every server keeps at least as much resilience as it had, up to its required level. Using biconnected components, the check covers the whole estate in one linear-time pass and explains any violation in plain terms.
How fast should the fix roll out?
Go too fast and reconvergence storms cause their own outage; go too slow and devices stay degraded. Raincurve adapts the Almgren-Chriss optimal-execution model from trading to remediation, with a hard cap on changes per minute where the control plane stops coping.
When a fix might itself be bad, the schedule starts slowly to learn, then accelerates to the churn limit once enough devices succeed.
actionrequested by change CHG-2291
statelive
checkrequired: 2
resultretry after agg-3 restored
pacingchurn cap
Guardrails that travel with every action
Works for any proposer
Humans, scripts and models all pass through the same check before execution.
Explanations, not just verdicts
Every block names the endpoint, the resilience before and after, and the requirement it violates.
Sandboxed execution
Approved actions run in isolated microVMs with snapshots for fast, clean rollback.
Progressive automation
Start with recommendations, move to approvals, then to automatic execution inside agreed limits.
Change-system aware
In-flight maintenance and freeze windows are part of the state being verified.
Full audit trail
Proposal, verification, approval, execution and outcome are recorded together.
Research
Method, results and limits for the techniques on this page.
Proving a network action is safe before it runs
Exact graph checks: zero unsafe approvals, 14,680 devices in 0.19 s.
Read paper →ResearchHow fast should a network fix roll out?
Almgren-Chriss pacing with a hard churn cap cut expected damage 43%.
Read paper →ResearchFirecracker sandboxing for secure execution
MicroVM isolation and copy-on-write snapshots for remediation workloads.
Read paper →