What trading desks can teach a NOC about rollout pacing
Optimal execution and network remediation share the same trade-off: move fast and pay in churn, move slowly and pay in exposure.
You have found the problem. A configuration change, a firmware fix or a route policy update will repair 48 degraded devices. The question nobody writes a runbook for is: how fast should you roll it out?
Push to everything at once and the control plane spends the next ten minutes reconverging, which is its own outage. Roll out one device at a time and the other 47 stay degraded all afternoon. Somewhere between those extremes is a best schedule. Traders have been computing it for 25 years.
The trader's problem
A trader who needs to sell a large block of shares faces the same tension. Sell fast and your own selling pushes the price down — market impact. Sell slowly and you are exposed to the price moving against you while you wait — volatility risk.
In 2000, Robert Almgren and Neil Chriss wrote down the optimal answer. Minimize expected cost plus a risk-aversion weight times the variance of cost, and the optimal number of shares still to sell decays along a hyperbolic sine curve: steep at first, then flattening. More risk aversion means a steeper start.
Mapping it to a network
The translation to remediation is surprisingly direct:
| Trading | Network remediation |
|---|---|
| Shares to sell | Devices to fix |
| Market impact of fast selling | Reconvergence churn from many simultaneous changes |
| Price volatility while waiting | Ongoing damage on devices not yet fixed |
| Risk aversion λ | Preference for predictable over average outcomes |
We simulated fixing 48 devices over 30 one-minute intervals and measured cost in device-minutes of degraded service, across 20,000 runs per schedule. Exposure cost one device-minute per unfixed device per minute, plus shared random shocks. Churn cost grew with the square of changes per minute.
Where networks differ from markets
Real control planes are not smooth. They absorb a certain rate of change and then fall over. We modeled that as a threshold: more than eight changes per minute triggers a reconvergence storm, costing 20 device-minutes for every change over the limit.
Plain Almgren-Chriss does not know about thresholds. With moderate risk aversion it planned 16 changes in the first minute — twice the limit. So we tried the obvious fix: keep the Almgren-Chriss shape, but cap it at eight per minute.
| Schedule | Mean cost | Worst 5% | Peak changes/min |
|---|---|---|---|
| Big bang (all at once) | 1,952 | 1,952 | 48 |
| Uniform over 30 min | 739 | 1,048 | 2 |
| Almgren-Chriss, λ = 0.05 | 549 | 634 | 16 |
| Almgren-Chriss, λ = 0.05, capped at 8 | 312 | 436 | 8 |
The capped schedule had both the lowest average cost and the lowest spread of anything we tested — 43% less expected damage than plain Almgren-Chriss. It violates the model's smoothness assumption and wins anyway, because the network's real cost curve has a cliff in it.
Risk aversion runs backwards
In trading, a nervous seller front-loads: get out before the price moves. In our network, a highly risk-averse schedule front-loaded 36 devices into the first minute, blew straight through the churn threshold and did worse. Moderate caution plus a hard cap beat maximum caution. The lesson generalizes: in a system with thresholds, "reduce exposure fast" and "don't break the control plane" pull in different directions, and the cap has to win.
What if the fix is wrong?
Everything above assumes the fix works. Real fixes sometimes do not. When we gave the remediation a 5% or 20% chance of being bad — detected gradually, with a five-minute rollback — the ranking changed.
With a 20% chance of a bad fix, the classic canary schedule (1%, 10%, 30%, 100% with validation pauses) had the best worst case: 3,461 device-minutes versus 3,640 for capped Almgren-Chriss. It paid for that with a higher average. Slow early progress is worth something when you are still learning whether the fix is safe.
Almgren-Chriss has no concept of learning during execution. Canary doctrine has no concept of optimal pacing once you are confident. The schedule operators actually want is both: go slowly while unsure, then accelerate to the churn limit once enough devices have succeeded. That combines optimal stopping with optimal execution, and as far as we can tell neither literature covers it. It is the most interesting open problem from this work.
How Raincurve uses this
In the platform, remediation pacing is a first-class step after verification. Every fleet-wide action gets a schedule with a hard changes-per-minute cap, a slow start while the fix is unproven and automatic acceleration as evidence accumulates.
Limits
The costs in this study are synthetic — chosen to be plausible, not measured — and the ranking depends on their ratios. Damage was shared across devices, weights were uniform and there was a single failure mode. Real networks should weight high-traffic devices more heavily. The research note has the full method.