In most software and configuration outages, the fix takes minutes and finding it takes hours. Read enough public postmortems with timestamps and the pattern is hard to miss: the rollback, the flag flip, the bad file swapped for a good one all show up near the end of the timeline, as a single line. Everything above that line is people trying to work out what's going on.
That time barely registers in how we talk about incidents. We count time to detect and time to resolve, and the long stretch between them, where the team is looking at the wrong thing, doesn't get a number of its own.
Two Kinds of Outage
Axis Intelligence keeps a cloud outage tracker that tries to measure this directly. It takes the minutes from the start of customer impact to the operator's published root-cause identification and divides that by total customer-impact minutes. They call it a Diagnostic Lag Ratio. Across the six recent major incidents they scored, the split is stark: the two power and cooling failures scored zero, because everyone knew immediately what had broken, and the software and configuration failures averaged 0.41. Their summary: "Physical failures are recovery-bound, software failures are diagnosis-bound."
The worst case in their table is a Microsoft Azure OpenAI incident on May 29, 2026, where 401 of the 446 minutes of customer impact passed before the root cause was identified. For more than six and a half hours, the constraint on recovery was knowing what to fix.
The tracker is upfront about its limits. It only works for operators transparent enough to timestamp their own confusion, and in its own words it measures "published diagnostic narrative, not ground truth." Six incidents is a small sample. But the direction matches what anyone who's run a config-driven incident already knows from the inside.
What Diagnosis Looks Like From Inside
Cloudflare's postmortem of its November 18, 2025 outage is one of the most honest accounts of this you'll find. A database permissions change went out at 11:05 UTC. Impact started at 11:20. The actual cause was a Bot Management feature file that had doubled in size and was crashing the proxy that loaded it.
It was 13:37 before the team's focus shifted to rolling that file back. Main impact was resolved at 14:30.
The two hours in between weren't carelessness. The bad file was regenerated every five minutes, and depending on which database node produced it, it was sometimes good and sometimes bad. In Cloudflare's words, "the entire system would recover and then fail again." A network that heals itself and then falls over again looks a lot like an attack. Then Cloudflare's status page, which is hosted entirely outside its own infrastructure, went down at the same time. That was a coincidence, but it "led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our status page."
Every signal the team had was real. Put together, they pointed at the wrong story. This is the same problem as the piece on four monitoring tools showing four views of one incident and the one on assuming clustered incidents are connected: most of an investigation's early time goes to building a story from partial evidence, and a wrong story costs more time than having none.
One detail in that timeline is worth noticing. At 13:05, half an hour before the work turned to the feature file, the team had already switched Workers KV and Access to fall back to an earlier version of the proxy, which cut their error rates sharply. They reduced the damage before they understood the cause.
Ignore the Instinct
Google's SRE book says it bluntly in the chapter on troubleshooting. Your first response in a major outage may be to go straight for the root cause: "Ignore that instinct!" The job is to "make the system work as well as it can under the circumstances," and to preserve logs and evidence so the root cause can be found afterward.
Nearly everyone agrees with this, and few teams actually work that way. The incident channel fills with hypotheses, and the one person asking "can we just roll back the 11:05 change?" gets told it's probably unrelated. Sometimes it is. But the costs aren't symmetric. Rolling back a change that turns out to be innocent costs you a redeploy and some pride. Holding off until you're sure costs you every minute of the diagnosis.
A useful rule for the incident commander: once the investigation passes a fixed point, say 20 minutes, with no confirmed cause, the default becomes reversing the most recent change in the blast radius, whether or not anyone can explain the connection yet. The piece on routine changes covered why the most recent change is guilty more often than it looks. A default rollback also gives the investigation a clean test. If impact stops, the search space shrinks to one change. If it doesn't, you've ruled a suspect out in minutes instead of arguing about it for an hour.
Hindsight makes this sound easier than it is. Cloudflare's triggering change was a database permissions tweak, and nothing about it said "this will crash the proxy." The rule isn't a guarantee that you'll roll back the right thing. It just makes rolling back the default instead of something the team has to argue for.
Mitigations You Can Pull Without an Explanation
The fastest recoveries in public postmortems usually come from a mitigation that was built in advance and doesn't require knowing why things broke. Google Cloud's June 12, 2025 incident is the clearest example. A policy change with blank fields hit an untested code path in Service Control, Google's API management layer, and crashed it in every region. The new code wasn't behind a feature flag, but it had shipped with a "red-button" to turn that serving path off. Within 10 minutes the team had identified the root cause and was putting the red-button in place. By 40 minutes it was fully rolled out and regions were recovering.
That's what having the switch ready before the incident buys you. If your system's only lever is "diagnose, write a fix, ship it through the normal pipeline," then diagnosis time goes straight into the outage. Feature flags, kill switches for new code paths, the ability to drain a region or pin a config to its last-known-good version: each one lets whoever is on call cut the impact while the investigation continues.
For each of these, ask a practical question: can the on-call engineer use it at 3 a.m. without a senior engineer on the call? If using it takes someone who understands the system well enough to diagnose the problem, it won't save you any diagnosis time.
When Diagnosis Isn't the Bottleneck
Not every long software outage is diagnosis-bound, and treating them all the same leads to the wrong playbook.
AWS's October 2025 us-east-1 outage started at 11:48 PM PDT, and by 12:38 AM AWS engineers had identified DynamoDB's DNS state as the source. DNS was fully restored by 2:25 AM. The last services didn't recover until 2:20 PM that afternoon. The long tail came from EC2's droplet workflow manager, which in AWS's words had "entered a state of congestive collapse and was unable to make forward progress," with work timing out and being requeued faster than it could finish. Engineers had to throttle incoming requests and restart components to clear the backlog.
Google Cloud's June incident had a smaller version of the same thing. The red button went out quickly, but restarting Service Control tasks created a herd effect on a Spanner table they depended on, because Service Control didn't have randomized exponential backoff in place. Recovery in us-central1 took up to about 2 hours and 40 minutes.
Once you know the cause and the system still isn't coming back, the playbook changes: shed load, throttle, restart in batches, and let queues drain instead of pushing them. It's the same retry-amplification pattern from Roundup #1. The first thing to settle in any long incident is which constraint you're under. Are you still looking for the cause, or have you found it and the system can't recover on its own?
Put the Gap in the Postmortem
If diagnosis is where the time goes, the postmortem timeline should show it. Add two timestamps most templates skip: when the team first had a working hypothesis that turned out to be correct, and when a mitigation was first available, as opposed to when it was used. The gap between those two and the start of impact is the part of the incident your tooling and your process actually control.
Then ask the questions that gap raises. What did we believe during that stretch, and why? Which signal pointed the wrong way? Was there a lever we could have pulled earlier, and if not, why doesn't one exist? Those questions produce action items that shorten the next incident, which a line like "improve monitoring" never does. That's the difference the postmortems without teeth issue was about.
The fix in most of these incidents was one line. The work worth doing is on what it took to find it.