Perspective

The incident is cross-layer. The tooling isn't.

Organizations are divided by layer. Failures are not. What changes when the unit of analysis is the dependency, not the component.

RaincurveSep 20265 min read

Most infrastructure organizations are drawn along layers: a network team, a compute team, a platform team, an application team, and increasingly an ML infrastructure team. Each has its own tools, its own alerting and its own on-call rotation. That structure makes sense for building things.

It makes much less sense for fixing them, because failures do not respect the org chart.

The anatomy of a cross-layer incident

A typical expensive incident starts small and low in the stack. A component degrades rather than fails, so nothing pages. Its effect travels upward through dependencies, getting larger and less recognizable at each layer, until it surfaces as a customer-facing symptom owned by a team that had nothing to do with the cause.

By then there are alerts in four tools and tickets in three queues. The first hour is spent establishing whose problem it is.

Three habits that make it worse

  1. Correlating by time. Grouping alerts that happen close together feels right, and fails exactly when the network is noisiest.
  2. Assuming redundancy. Designs are redundant on paper; after a few unrepaired failures, they often are not.
  3. Acting under pressure. The fix that is safe in a runbook can be unsafe given everything else that is broken or being changed.

Each of these is a reasonable shortcut for one layer. Across layers, each one turns a small incident into a bigger one. We have written about the research behind all three: causal alarm grouping, verifying actions against live state and pacing remediation.

Change the unit of analysis

The alternative is to make the dependency, not the component, the thing you reason about. That means maintaining one graph of how everything depends on everything else, across vendors and domains, and asking every question against it: which alarms are related, what is the origin, who is affected, is this action safe.

Teams keep their tools and their ownership. What they gain is a shared answer to the first question of every incident — what actually happened — before the war room starts.

Make infrastructure intelligence operational.

Start with a conversation about your environment.