An alerting pipeline can fail silently, and when it does, the outage it was built to catch shows up with no page at all. Most post-incident reviews ask "did we have an alert for this?" The question I'd add is how you'd know if the alert fired and never reached a person.

Lucidworks' postmortem for a July certificate expiry is the clearest example I've read this year. On June 27, a synthetic test raised a 30-day warning that the certificate on its SaaS platform was about to expire. The warning went to an internal alert channel, where nobody acted on it. Then, from July 1, alerts stopped being delivered from the monitoring platform to the incident management system, so none of the follow-up notifications reached the team. The check ran and fired as designed for four more weeks. On July 27 at 17:56 UTC the certificate expired. Users got certificate errors for about 88 minutes, and Beacon signals sent during the window "were not ingested and are not recoverable."

The line in that writeup I keep coming back to is this one: "no monitoring existed on the alert pipeline itself." Most teams I've worked with could write the same sentence about themselves, including some with very good monitoring.

Silence Means Two Different Things

A quiet on-call shift has two explanations. Either nothing is wrong, or something is wrong and the news can't get to you. From where the pager sits, those look exactly the same. Zero pages on a Tuesday night feels like a good night, and nobody opens the alert router at 2 AM to check that it's still sending.

It's the blind spot from the hidden cost of green dashboards, one layer further down. A green dashboard can hide a failing service. A silent pager can hide a failing alerting system, and that one gets even less scrutiny, because the whole reason you have alerting is so you don't have to keep looking.

The Alerting Path Is a Production System

Write out every hop between "a metric crosses a threshold" and "a phone buzzes," and the list runs longer than most people expect:

• the probe or collector that produces the signal

• the rule evaluator: Prometheus, a vendor monitor, a synthetic check

• the router, such as Alertmanager or a vendor's notification policy

• the integration into the incident tool, usually a webhook URL or an API key

• the service and escalation policy inside that tool

• the on-call schedule that decides who is on right now

• that person's notification settings, and the phone itself

Any of these can break without raising an error anyone sees. Someone rotates an integration key during a security cleanup, and the router starts getting 401s, logs them, and moves on. A service gets renamed in the incident tool while the routing rule still points at the old name. A schedule has a two-hour hole at a timezone handoff. An engineer gets a new phone and never re-enables push notifications. Nothing upstream notices, because everything upstream is still working.

Lucidworks' break was at the fourth hop, between the monitoring platform and incident management. Every hop before it kept doing its job, which is why nothing looked wrong. The checks ran, the dashboards were up, and the warnings kept firing into a pipe that went nowhere.

Request-path services get SLOs, runbooks and named owners. The alerting path usually gets none of those, even though it's what you're relying on at the exact moment everything else has failed.

When the Outage Takes the Pager Down With It

The other way to lose alerting is shared fate, where the failure that breaks production also breaks the path that would have told you about it.

Slack's January 4, 2021 outage is the standard example. Early in the investigation, in Slack's words, "our dashboarding and alerting service became unavailable." Those services ran in a different VPC from their own backend databases, so reaching their data meant crossing the same AWS Transit Gateways that were overloaded and causing the outage. Engineers fell back to internal consoles, command-line tools and querying the metrics backends by hand. The page that kicked off the response came from "an external monitoring service," one that didn't share the broken network.

Pico.sh's May 2024 postmortem is smaller, and almost funny. A VM ran out of memory at 11:02 AM and took four services down with it. Prometheus "did receive alerts about the service outage," and sent them to an IRC channel. Phone notifications from that channel came through an IRC bouncer, and the bouncer was one of the four services on the dead VM. So the alerts went to a channel nobody was connected to. The team found out at 12:19, 77 minutes in, and that's 77 minutes before anyone could even start the diagnosis work I wrote about in where incident time actually goes.

Both are the failure-domain mismatch from Roundup #3, applied to the alerting stack instead of the service. If whatever pages you shares a network, a VM, a region or an identity provider with what it watches, you have a fire alarm wired to the circuit most likely to burn.

Delivered to a Channel Isn't Received

Lucidworks had a second, quieter failure, and it came before the pipeline broke. The June 27 warning was delivered. It landed in an internal alert channel and sat there.

Most teams have a channel like that, where bots post into a scroll of messages that people mute after the first week. A 30-day certificate warning doesn't need to wake anyone up. It does need an owner and a deadline: a ticket assigned to a team, due well before expiry, that escalates if it's still open. That's the direction Lucidworks took. Certificate alerts now go to "a paged alerting path that requires acknowledgment," and certificate rotation is being automated so the alert should rarely need to fire.

The test I'd apply to any warning with a long lead time: if it can only land in a channel, plan on it being missed.

A Heartbeat for the Pager

The usual fix for silent pipeline failure flips the logic around. Send a signal that should always arrive, and raise the alarm when it stops.

The Prometheus ecosystem has this built in. kube-prometheus ships an alert called Watchdog whose rule is just vector(1), which means it is always firing. Its runbook states the purpose plainly: it is "meant to ensure that the entire alerting pipeline is functional." You route Watchdog to an external dead man's switch service that expects to hear from it every few minutes. When the heartbeat stops, whether because Alertmanager crashed, a credential expired or someone changed a firewall rule, that outside service pages you over a separate path.

Two details decide whether this protects you or only looks like it does. First, the receiver has to sit outside the failure domain of everything it watches: another provider, and ideally another network. Second, the heartbeat should pass through as many real hops as you can manage. If it goes from Alertmanager straight to the dead man's switch, it proves Alertmanager is running and tells you nothing about the integration into your incident tool. Lucidworks' failure sat downstream of the point where a lot of Watchdog setups stop.

Test the Last Mile With a Real Page

Heartbeats cover the machinery. They don't cover the last few hops: the schedule, the person's notification rules, the phone left on Do Not Disturb.

The cheapest check I know of is to send one real test page through the production integration at every on-call handoff, to whoever is coming on, and have them acknowledge it. It takes half a minute. It catches the expired key, the renamed service, the schedule gap and the new phone all at once, at 10 AM on a Monday rather than 3 AM in the middle of an outage.

Add at least one black-box probe that runs from outside your infrastructure and pages through its own route, the way Slack's external monitor did. When your internal stack goes down with production, that's the page that still gets through.

Questions for Your Next Alert Review

Next time a postmortem says "the alert didn't fire," or the worse version, "it fired and nobody saw it," these are worth asking:

• What tells us when the alerting pipeline itself has stopped, and where does that signal go?

• Does anything in our paging path share a network, VM, region or identity provider with the systems it monitors?

• Which alerts only ever land in a channel, and who owns each one?

• When did someone last get a real page end to end, rather than a test notification sent from inside the tool?

• Which integration keys does our alert routing depend on, and what happens to paging the day one of them is rotated?

Lucidworks' check did its job for 30 days straight. The part that failed was the part nobody checked, because it had always worked before. If your pager has been quiet for a long stretch, it's worth sending yourself a page to make sure it still works.

On-call is the part of an SRE, DevOps or platform job that the offer letter says least about, and it's the part most likely to decide whether you still want the job a year later. Salary, level and title all get negotiated in writing. The rotation usually gets one line in the job description ("participation in an on-call rotation") and a reassuring sentence from the hiring manager.

Most of what you need to know about a team's on-call can be found out before you sign, if you ask the right questions and listen carefully to how they're answered. This is the list I'd take into the last round.

Asking Without Sounding Like You're Dodging the Pager

The worry that stops most people asking is that it'll read as "I don't want to do on-call." In practice it reads the other way. Engineers who've carried a pager for a while ask specific questions about it; people who haven't tend to ask nothing, or ask whether there is one at all. When I wrote about what SRE interviews are actually testing for, the argument was that the differentiator has moved from debugging speed to judgment, and a sharp question about alert volume shows judgment as well as most of the answers you'll give in the loop.

Timing matters more than phrasing. A recruiter usually can't answer any of this. Save it for the hiring manager, and if you can, for an engineer who is on the rotation right now. Frame it as wanting to understand the work: "What does a typical on-call week look like for this team?" gets you further than "How bad is on-call?"

How Many Times Did the Last Person Get Paged?

Ask about a specific recent shift, not the average. "How many pages did whoever was on-call last week get, and how many of them were overnight?" An average smooths over the week that ruined someone's sleep, and "it varies" tells you nothing.

There's a useful benchmark to hold the answer against. Google's SRE book, in its chapter on being on-call, works out that dealing with one incident properly (root cause, remediation, the postmortem) takes about six hours, and concludes that "the maximum number of incidents per day is 2 per 12-hour on-call shift." It also says the median should be zero: most shifts should be quiet. That's a ceiling Google wrote for its own teams, not a law, but it gives you something concrete to compare against, and a team fielding five pages a night is well past it.

What you're also listening for is whether they know the number at all. A manager who can say "about three pages a week, mostly from one flaky dependency we're replacing" has a team that measures its pain. One who has to guess probably doesn't, and pain nobody measures rarely gets fixed.

How Big Is the Rotation?

This one is just arithmetic, and it's the question people most often forget to ask. With four people sharing a weekly rotation you're on-call one week in four, every month, for as long as the team stays that size. With eight it's one in eight.

The same Google chapter derives its minimums from a rule that no more than 25% of an SRE's time goes on on-call: "the minimum number of engineers needed for on-call duty from a single-site team is eight," covering a primary and a secondary, and "a reasonable minimum size of each team is six" when two sites split the day between them. Most teams you'll interview with are smaller than that, and that's fine if the service is quiet. It's less fine when a small rotation meets a noisy service.

Two follow-ups are worth asking here. Is there a secondary, so you're never the only person between an alert and an outage? And if the team spans time zones, is it genuinely follow-the-sun, or does one region cover nights because it's smaller?

What Happens to an Alert That Pages for Nothing?

This is the question that separates a rotation that's bad today and getting better from one that's bad and staying bad. Every team has noisy alerts. What matters is whether anyone is allowed to spend time fixing them.

Google's guidance is short: "All paging alerts should also be actionable." The same chapter warns that low-priority alerts firing every hour cause fatigue that "can also cause serious alerts to be treated with less attention than necessary", which is the quiet way most alerting failures happen (I went into one version of it in The Hidden Cost of Green Dashboards).

So ask for an example: "When did the team last delete or retune an alert, and who decided?" A good answer names a recent alert and a person. A worrying answer is that alerts are owned by another team, or that tuning happens "when we get time", which usually means it doesn't happen.

Whose Code Are You On-Call For?

On a platform or central SRE team, you can end up carrying the pager for services written by people who've never been woken up by them. That isn't automatically bad, but it changes everything about whether the rotation can improve.

Ask what happens when a service another team owns is paging too much. Google's answer, again from the same chapter, is that in extreme cases SRE can "give back the pager," asking the developers to be on-call for their own system "until it meets the standards of the SRE team in question." Very few companies have that lever written down. What you want to hear is that something like it exists, even informally: a way to push reliability work back onto the team that's generating the pages. If the honest answer is "we just absorb it," you'll be absorbing it too.

Is It Paid, and What Happens After a Bad Night?

Ask about pay directly, and don't read too much into the answer on its own. Gergely Orosz's overview of how more than 120 firms handle it at The Pragmatic Engineer, Oncall Compensation for Software Engineers (2022, so treat the specifics as dated), found that "Most of Big Tech don't pay for oncall with cash compensation. Google is the only exception." He also observed that "The more an employer compensates software engineers, the more likely they expect oncall to be a given." An unpaid rotation at a top-of-market salary is a different deal from an unpaid rotation at an average one. Some countries also regulate on-call pay and rest, so where you'll be employed matters as much as where the company is based.

The question I'd weight more heavily is the informal one: if you're up from 2 a.m. to 5 a.m. fixing something, are you expected online at 9? Teams that tell you to start late, or take the morning off, without anyone needing to ask, usually treat on-call as real work in other ways too. Teams where that question gets an awkward pause usually don't.

Reading the Answers

Nobody will describe their on-call as terrible in an interview, so what you're judging is how specific the answers are. Numbers, names and recent examples are good signs even when the numbers are bad, because a team that knows exactly how noisy it is can usually tell you what it's doing about it. Vague reassurance, or a manager who answers every question and then won't let you talk to someone on the rotation, is the pattern to worry about.

Bad on-call isn't automatically a reason to turn an offer down. A noisy rotation with a funded plan to fix it, and a manager who can point to the last three alerts they killed, can be a great place to learn this work fast. If you join one, the AI agents now landing in on-call tooling may take some of the triage off you, though they won't fix a rotation that's understaffed. What you want to avoid is the version where nobody knows the numbers and nobody has time to change them.

You'll put more effort into negotiating your salary than any of this. It's still worth twenty minutes in the last round, since you'll be living with the answers every few weeks for as long as you're there.

Three outages in the past week, at Microsoft Azure, Slack and GitHub, broke in the gap between two systems that each worked on their own. At Azure, two changes Microsoft itself calls unrelated hit the same regions at the same time. At Slack, a third-party app stuck in a loop tripped a sending limit, and the limit blocked free-plan workspaces from posting. At GitHub, the network link between an on-premises data center and cloud-hosted databases partly failed. None of the three started with a component that was simply broken.

Azure: Two Unrelated Changes in the Same Regions

From 20:30 UTC on September 30 to 02:15 UTC on October 1, some customers using Azure's gateway services in multiple regions saw degraded or interrupted network connectivity, gateways failing to load in the portal, and failed or delayed network management operations. The affected products are most of the ways an enterprise network gets into Azure: ExpressRoute Gateway, VPN Gateway, Azure Firewall, Application Gateway and its WAF, and Azure VMware Solution. Early coverage put the footprint at 18 regions, but Microsoft has since corrected the region list in its status entry, which now names France Central, North Europe, Southeast Asia, UK South and UK West as the last to recover.

Microsoft's preliminary review, under tracking ID 7Q30-010, says "a recent change to a regional gateway management service triggered a higher-than-expected load when an unrelated operating system servicing maintenance proceeded gradually through multiple regions." The gateway service would normally autoscale, but the extra work also raised demand on the services it depends on, and that stopped it from scaling. Engineers paused the OS servicing and reverted the gateway-manager change. A full post-incident review is due within 14 days.

Microsoft's own timeline shows how long a seam takes to see. Impact began at 20:30. Investigation started at 21:29, focused on ExpressRoute Gateway in UK South. At 22:27 the team realized several regions were involved, and at 23:05 it found "a correlation with operating system servicing activity" and paused it. That's two and a half hours from first impact to linking the two changes.

The word doing the work in Microsoft's explanation is "unrelated." A few weeks ago I wrote about the opposite trap, where on-call engineers assume incidents that show up together must share a cause. Here, two changes that had nothing to do with each other really did combine. The OS servicing was moving region by region, which is the careful way to do it. But a staged rollout tests a change against the system as it stood when the rollout began. It has no way to test against a separate change that is also still in flight, and when the damage only appears where the two overlap, going region by region just carries the overlap into more regions.

Recovery, at least, was the straightforward kind. Reverting a service change is the case where rollback does what you expect, which Monday's issue argued is less common than teams assume.

https://azure.status.microsoft/en-us/status/history/?trackingId=7Q30-010

Slack: Someone Else's App Hit the Limit

At 12:39 UTC on October 1, Slack reported that "some customers on our free plan are experiencing message-sending limit blocks in an unusual spike pattern." Seventeen minutes later it had the cause: "a third-party app is stuck in a loop that has triggered billions of messages." Later updates added that "a feature toggle ramp-up caused an app integration to repeatedly post messages in a processing loop," and that this "pushed free plan customers over the message volume threshold, triggering the sending block."

The limit did its job. It saw an enormous spike and stopped the sending, but the people it stopped were free-plan users trying to post ordinary messages. Slack's updates don't say whose feature toggle it was, or whether the blocked workspaces were only the ones with that app installed, so I won't guess at either. The timeline is what stands out. Slack named the cause 17 minutes in, yet it didn't announce a potential fix until around 18:52 UTC, and sending wasn't restored for every affected workspace until just after 19:00, six hours and 23 minutes after the first report. Knowing exactly what was wrong didn't come with a quick way to lift the block.

The seam here runs between a platform and the integrations people build on it. Rate limits and abuse thresholds are usually designed around one kind of traffic. If an integration's messages count toward the same threshold that guards your users, a bug in someone else's code becomes your users' outage. Roundup #4 looked at teams paged for outages that started at a vendor. This one runs the other way: the platform got hurt by something built on top of it.

https://slack-status.com/2026-10-01

GitHub Actions: The Link Between the Old Data Center and the Cloud

On October 5, between 18:48 and 22:49 UTC, GitHub Actions and its hosted runners degraded badly. In the worst 90 minutes, 47.0% of workflow runs on GitHub-hosted runners failed and 71.9% of hosted jobs didn't start within five minutes. Repository lists, billing and licensing pages, and some Copilot features also threw intermittent errors. Self-hosted runners were not affected.

GitHub's summary puts the cause down to "a partial networking failure that disrupted connectivity between one of our on-premises datacenter regions and a subset of cloud-hosted regional database and storage services." It mitigated by moving database and storage traffic to healthy regions and endpoints and adding compute capacity, and says it is working with its cloud provider on the network failure itself.

That sentence describes a company partway through a move. GitHub has been migrating off its own data centers onto Azure, and for as long as a migration like that runs, some of the system lives in the old place and some in the new, with a network path between them that the whole system now leans on. Both ends can be healthy while the path between them isn't. Four days earlier, GitHub had a smaller Actions incident it attributed to "throttling within an upstream Azure dependency." Neither incident means the migration is a mistake. Both show what the middle of one looks like from outside: new dependencies on the hot path, each with its own way to fail.

https://www.githubstatus.com/incidents/c11dc9nb1zdq

Who Owns the Seam?

Ask which team owned each of these failures and the answer gets awkward. At Azure, the gateway-manager change and the OS servicing were separate pieces of work that Microsoft describes as unrelated. The looping app belonged to someone outside Slack entirely, while the limit it tripped belonged to Slack. At GitHub, the failing path ran between an environment being retired and one being adopted. Every component had an owner. The connections between them are the parts nobody draws a dashboard around.

Three checks follow, and none of them is expensive:

• Before you start a rollout, find out what else is rolling out to the same regions or hosts that week. A shared change calendar is low-tech, but it's the only thing likely to show two "unrelated" changes heading for the same place.

• Look at what your protective limits actually count. If an integration, a batch job or a retrying client can push the same counter your users depend on, decide ahead of time who gets blocked when it trips and how quickly you can lift it.

• If you're mid-migration, give the links between the old and new environments an owner, an SLO and their own alerts. They're meant to be temporary, and temporary infrastructure tends to get none of those.

Microsoft's full review of the gateway incident is due within two weeks. If it says more about how the two changes met, it'll show up in a future Roundup.

A rollback puts the old code back. It doesn't put back anything the new code did while it was running: rows it wrote, flags it gave a new meaning to, machines it left unable to take another update. Most rollbacks work anyway, because most changes don't touch any of that. The ones that do are where recoveries go badly wrong, sometimes worse than the original fault.

Last week's issue on where incident time goes argued that after about 20 minutes without a confirmed cause, rolling back the most recent change should be the default. I still think that's right. But it assumes the rollback is safe to run, and this issue is about the cases where it isn't.

What the Pipeline Actually Reverts

When a deploy tool rolls back, it swaps one artifact for another: a binary, a container image, a config file. That's the entire scope of the operation. Everything the new version did to the world outside that artifact stays done.

So every rollback carries an assumption nobody writes down: that the old version can run correctly against the world the new version left behind. For a stateless change, a bug in how a page renders or a bad timeout value, that's trivially true. It stops being true the moment the change wrote something, changed what a value means, or broke the channel the rollback has to travel through. Public postmortems have clear examples of each.

When Old Code Reads New Meaning

The worst rollback on record may be Knight Capital's, on August 1, 2012. The SEC's order lays it out. Knight was deploying new code for the NYSE's Retail Liquidity Program to eight servers. That code reused a flag that used to switch on "Power Peg," an order-routing function Knight had stopped using in 2003 but never deleted. A technician didn't copy the new code to one of the eight servers, and nobody checked.

When orders carrying the reused flag hit that eighth server, the old Power Peg code woke up and started sending orders without tracking whether the parent order had been filled. While the team was trying to find the problem, they did what most of us would do and backed out the new deployment. The SEC's account of that step: Knight "uninstalled the new RLP code from the seven servers where it had been deployed correctly. This action worsened the problem."

The rollback restored the old code on seven more servers. But incoming orders still carried the flag, and the old code still read that flag as "run Power Peg." In about 45 minutes Knight's router sent millions of orders and racked up 4 million executions. The firm lost $460 million.

The lesson usually drawn is about dead code and deployment review, and both apply. The part that matters here is narrower. Once a value has been given a new meaning and other systems have started sending it, you can't roll that meaning back by redeploying. Every flag, enum value, or field whose meaning you change is a one-way door. Add a new one instead.

Rolling Back the Wrong Half

Roblox's 73-hour outage in October 2021 shows the opposite mistake: putting the state back while the trigger stays in place.

Roblox's service discovery ran on a Consul cluster, and that cluster's writes had slowed to a crawl. About twelve hours in, the team shut the whole cluster down and restored it from a snapshot. Their reasoning was sound: "We expected that restoring from a snapshot taken when the system was healthy would bring the cluster into a healthy state." It was healthy for a while. Then, once internal traffic came back, "Consul performance began to degrade again, and eventually we were back to where we started," with median write latency back at two seconds.

The snapshot had rolled the data back. What it didn't touch was the configuration. The day before the outage, Roblox had enabled a newer Consul streaming feature on a backend service responsible for traffic routing. Under heavy load that feature caused the contention, and a separate BoltDB storage bug made things worse. Streaming wasn't switched off everywhere until October 30, more than two days into the outage, and that's when write latency finally dropped.

A rollback has two halves, the code and config that changed and the state that changed, and they can come apart. Reverting only the state means the system rebuilds the same bad state as soon as load returns. Reverting only the code (Knight's case) means the old code meets state it can't handle. Before you revert anything in an incident, it helps to say out loud which half you're reverting and why you think the other half doesn't matter.

Data You Can't Un-Write

Some changes leave nothing to roll back to. In April 2022 Atlassian ran a maintenance script with the wrong list of IDs and the wrong execution mode, and deleted 883 sites belonging to 775 customers. Atlassian's backups held, and no customer lost more than five minutes of data. Even so, some customers were down for up to 14 days.

Atlassian's review explains why. Their data stores held many customers' data together, so restoring meant they had to "manually extract and restore individual pieces from our backups," site by site, into an environment other customers were still using. The site-level deletion "did not have runbooks that could be quickly automated for the scale of this event."

Restoring from backup is a separate operation from rolling back, and it's usually much slower. The day-to-day version of this risk is the schema migration that drops or renames a column. The old code expects that column, so once the migration runs, rolling the code back just gives you a second outage. Any change that deletes or rewrites data should go into the change record as "roll forward only." Writing that down is what makes people treat the change with appropriate care.

When the Rollback Can't Reach the Broken Machine

The last case is the rollback that works perfectly and arrives too late.

On July 19, 2024, CrowdStrike pushed a content update to Windows sensors at 04:09 UTC and reverted it at 05:27 UTC, 78 minutes later. For a fleet-wide push, that's a fast revert. But the update crashed the machines that received it, and a machine stuck in a crash loop can't download the fix. Microsoft estimated that 8.5 million Windows devices were affected. The revert stopped the problem spreading, but it couldn't repair the machines that were already down. Those needed manual remediation, one machine at a time. It's the same shape as Roundup #6, where the usual recovery moves were the ones that failed.

Facebook's October 2021 outage is a smaller version of the same trap. During routine maintenance, a command meant to assess backbone capacity took down every connection in Facebook's backbone network, and with it the tools engineers would have used to undo it. The total loss of DNS "broke many of the internal tools we'd normally use to investigate and resolve outages," and engineers had to get into the data centers in person, which was slow because those buildings are designed to be hard to get into.

In both cases the path the rollback had to take ran through the thing that was broken. Facebook could undo the damage once someone was physically at the hardware. CrowdStrike could revert the file but couldn't get it to the machines that needed it.

Designing Changes You Can Actually Undo

None of this argues against rolling back. It argues for knowing ahead of time which of your changes can be rolled back, and for making more of them reversible.

Split one-way changes into reversible steps. The standard pattern is parallel change, also called expand and contract: add the new column, field, or endpoint next to the old one; move readers and writers over; remove the old one only after the new one has run in production for a while. Each step can be undone on its own, and the one destructive step comes last, when you're most confident.

Make N-1 compatibility a release check. The old version has to be able to read whatever the new version writes. Test it the way you'd test anything else in staging: deploy the new version, let it write data, roll back, and run the suite against what's left. If nobody has ever rolled back in staging, you don't actually know whether your rollback works.

Never reuse a flag or enum value. That's the Knight lesson, and it's cheap to follow. A new meaning gets a new name.

Ship behind a flag so "undo" is a config flip. If new behavior is behind a feature flag, turning it off doesn't require a redeploy, and the old code path never left. This works only if the flag guards the writes as well as the reads. Otherwise you can turn the flag off and still be left with data the old path doesn't understand.

Keep the undo path outside the blast radius. Out-of-band access to your control plane, break-glass credentials that don't depend on your own identity provider, and deploy tooling that keeps working when the thing it deploys is down. If your rollback goes through the service you're rolling back, assume it won't be there when you need it.

Stage fleet-wide pushes so the revert lands first. CrowdStrike's root cause analysis committed to exactly this: content "successively promoted to wider deployment rings or rolled back if problems are detected." A rollback only helps the machines that haven't been hit yet, and staging the rollout is what keeps that group large.

One Question Before You Revert

The 20-minute default from last week still stands. It just needs one question added before anyone runs the rollback: what has the new version written, sent, or changed that the old version will have to read?

Usually the answer is "nothing," and you should roll back right away. When the answer is a migration, a reused flag, or a queue full of messages in the new format, rolling back may not be the safest move. Turning the change off behind a flag or shipping a quick forward fix can be safer, and the time to decide that is before the rollback runs, not after it makes things worse. It's the same habit behind why routine changes cause the worst outages: the riskiest step is the one everybody assumes is safe.

Know which of your changes are one-way before the incident starts. Once it's started, there's no time to work it out.

If you've caused a production outage, the week after it matters more to your career than the outage itself. Most engineers who work near production long enough will break it at some point: a config push with the wrong value, a migration run against the wrong host, a cleanup script that was a lot less selective than you thought. The incident gets resolved in an hour or two. What people remember, months later, is how you handled yourself while it was happening and in the days that followed.

This one is for the engineer who is reading it with a knot in their stomach because last night's incident channel has their deploy in it.

Say It Early, While It's Still Down

The single most useful thing you can do during the incident is put your change on the table before anyone has to go looking for it. Something like: "I deployed the rate-limit config at 14:05, that's my best guess for the cause, I'm preparing a rollback." One message, with a timestamp and what you're doing about it.

It feels terrible to type. It also saves the team more time than anything else you could do, because in most incidents the slow part is working out what changed and why, not the fix, which is the whole argument of Diagnosis Is the Outage. Every minute the room spends reverse-engineering a change you already know about is a minute of outage you could have taken off the clock.

What you shouldn't do is quietly fix it forward on your own while everyone else is still investigating. Even if it works, someone will find your commit in the timeline later, and the conversation about why you didn't mention it is a much harder one than the conversation about the mistake would have been.

The Message to Your Manager

Send it the same day, ideally before they hear about it from someone else. Keep it short and factual: what you changed, what broke, how long it lasted, what's already been done, what you think needs to happen next. Skip the long apology. One "sorry, this one was mine" is plenty; three paragraphs of it hands your manager the job of reassuring you, on a day when they've got their own fallout to deal with.

A hypothetical version:

The checkout errors this afternoon came from my config change at 14:05. Rate limit was set at 50 req/s instead of 500. Rolled back at 14:41, error rate back to normal by 14:48. I'd like to own the postmortem and look at why nothing caught a 10x drop in a limit before it shipped.

That last sentence is the important one. It tells them you're already thinking about the system and not only about yourself.

You Know More About This Failure Than Anyone

John Allspaw's 2012 essay on blameless postmortems at Etsy is still the best thing written on this, and it makes a point that's easy to miss when you're the one who's embarrassed. The person who made the mistake has information nobody else has: what they expected the command to do, what they looked at before running it, which assumption turned out to be wrong. In Allspaw's words, "They are, after all, the most expert in their own error."

So write your part of the timeline in detail. Not a defence, an account. "I checked the staging value and assumed prod matched" is exactly the sentence a postmortem needs and exactly the one people are tempted to leave out.

GitLab's postmortem of its January 2017 database outage is a good example of what that looks like in public. An engineer trying to fix replication wiped the database directory on the primary instead of the secondary, around 300 GB was gone before they stopped it, and roughly six hours of data couldn't be recovered. The postmortem notes, almost in passing, that the engineer's name in the recovery document "was added by the engineer themselves." Most of what the rest of the industry learned from that incident (untested backups, no one owning data durability) came out because the people involved were open about exactly what they did.

What a Fair Postmortem Says About You

Look at how AWS described the S3 outage of February 2017, the one that took a large chunk of the internet down with it: "an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers." One input was mistyped. The fix AWS described wasn't retraining the engineer; they changed the tool "to remove capacity more slowly" and added safeguards against taking a subsystem below its minimum capacity.

That's the shape you want your own postmortem to have, and you can steer it there. Google's SRE book puts it plainly: "You can't 'fix' people, but you can fix systems and processes to better support people making the right choices." If the draft root cause says "human error" and stops, ask the question that follows: why did a single mistyped value reach production with nothing in the way? Usually the honest answer is a missing validation, a review step that doesn't look at config, or a deploy that went to everything at once. I went through why those routine changes cause so many outages in an earlier issue.

Then volunteer to own the action item that closes the gap, and actually ship it. An owner and a deadline are what separate a postmortem that changes something from one that doesn't, and having your name on the fix changes what people associate with you when the incident comes up again, though it usually takes a quarter or two before you notice.

When the Culture Isn't Blameless

Not every team works like this, and pretending otherwise would be a disservice. Some places still look for a name to put in the root cause, and some managers say "blameless" while running the review like a deposition.

If that's where you are, the advice above still mostly holds, but tighten it up. Keep everything you write factual and timestamped. Describe what you did and what you saw rather than characterising it ("I ran the cleanup against prod" rather than "I carelessly ran..."). Have the harder conversation with your manager one to one, not in a public channel. And keep your own notes on what you flagged before and after, because in a blame culture the written record is your protection.

It's also worth noticing what the organisation does next. A team that punishes people for good-faith mistakes is training everyone to hide the next one, which is how small incidents become long ones. If you see that happen, that's useful information about whether you want to stay.

The Part Nobody Puts in the Postmortem

You'll replay it. Possibly for a while, possibly at 3 a.m., and probably the next time you're about to hit enter on anything touching production. That hesitancy is normal, and it fades faster if you deploy again soon (with a second pair of eyes if that helps) rather than avoiding the pipeline for a fortnight.

It might help to read how a company sounds when it owns one collectively. Cloudflare's writeup of its July 2019 global outage, caused by "a single WAF rule that contained a poorly written regular expression," spends much of its length on the chain of process gaps that let one rule take the network down for 27 minutes. The emotional line in it is in the first person plural: "We're ashamed it happened." Someone wrote that rule. The company still treated the outage as something the company did, and in a healthy team, that's how yours will be treated too.

A Year From Now, It's a Story

Here's the thing that's hard to believe the morning after: an outage you caused, handled well, can turn into one of the better stories in your career. "Tell me about a time you made a mistake in production" comes up constantly in SRE interviews, and as I wrote in what SRE interviews are actually testing for, the answer they're listening for is about judgment and follow-through, not a clean record. The same goes for promotion cases. A fix that removed a whole class of incident is exactly the kind of evidence I described in putting incident work in your promo case, and it counts whether or not you caused the first one.

Nobody senior in this field has a clean record. What they have is a decent account of what they did next.

Three outages in the last ten days, at AWS, Microsoft Azure and Anthropic, had the same odd shape. Whatever was already running kept working. Whatever tried to start fresh (a new EC2 instance, a restarted backend node, a user signing back in to Claude) ran straight into the broken part. In all three, the move you'd normally reach for first during an incident was the one that didn't work.

AWS EC2: Running Instances Fine, New Ones Couldn't Launch

On the afternoon of September 21, Pacific time, EC2 in us-east-1 spent about 77 minutes failing new instance launches. AWS put it down to "a recent deployment to a subsystem within the EC2 service responsible for processing instance launch requests," and rolled that deployment back. Instances that were already up were never affected.

That sounds like the gentle kind of outage until you list what depends on launching. Auto Scaling groups replacing unhealthy hosts, ECS and Fargate starting new tasks, EMR bringing up clusters: AWS named ECS, Fargate and EMR as impacted. A service that was healthy when the window opened stayed healthy only as long as nothing about it had to change. If a host died, or traffic climbed and the autoscaler went to add capacity, the replacement never arrived. A team shipping a deploy onto fresh nodes in that window would reasonably have concluded their own release was broken, when the problem sat a layer below them. The trigger itself was a deployment, which is the kind of routine change that keeps causing the worst outages, even at the companies with the most practice shipping them.

https://awsdown.com/incidents/evt-AWS_EC2_OPERATIONAL_ISSUE_79073_047629764F9

Azure OpenAI: A Health Check That Restarted the Survivors

From 10:03 to 15:58 UTC on September 29, Azure OpenAI Service, Foundry and Cognitive Services in Sweden Central saw intermittent failures, higher latency and 5xx errors. Microsoft's preliminary review traces it to a backend service that looks up resource metadata. "An internal system process generated an increase in the overall amount of requests" to that service, its instances hit their utilization thresholds, and they began restarting. Then the automation joined in: "An automated health check then triggered additional restarts, further reducing the number of healthy instances available to process requests."

Every restart took an instance out of the pool and moved its share of the load onto the rest, which pushed those over the threshold in turn. The health check was doing its job, removing sick instances, but the sickness here was load, and when the problem is load, pulling instances out only concentrates it. Engineers got out of it by scaling the service out and raising its resource limits, and Microsoft's follow-ups focus on throttling and rate-limiting internal processes. It's a close relative of the retry storm in the first Roundup, where the recovery mechanism was also the amplifier.

https://azure.status.microsoft/en-us/status/history/?trackingId=7QL5-Z50

Claude: "If You're Signed In, Please Don't Sign Out"

The same day, Anthropic's status page reported elevated errors across claude.ai, the API, Claude Code and Cowork beginning at 14:00 UTC. The main errors were mitigated by 14:36 and existing conversations mostly came back, but sign-in stayed broken for longer: single sign-on and Apple login kept failing, along with new chats, Claude Code and Cowork sessions, and file uploads. Anthropic's update told users plainly, "If you're signed in, please don't sign out."

Anthropic hasn't published a root cause, so I won't guess at one. What's worth noticing is the advice. A live session was the thing still working, and the standard fix for a flaky web app, signing out and back in, would have thrown it away. Most status pages say what's broken and stop there. Very few tell users which of their own actions are safe mid-incident, and this one did. If part of your system can still serve people who already hold a session, a token or an open connection, say so on the status page.

https://9to5google.com/2026/09/29/claude-confirmed-outage-sept-29/

What the Three Have in Common

Restart it, scale it out, log back in: most incident reflexes quietly assume that the path for creating something new is healthier than the thing that's struggling. In these three incidents it wasn't. Launch requests, health-check restarts and sign-in are all ways of starting from scratch, and starting from scratch is what broke.

Two checks follow from that. First, go through your runbooks and mark which recovery steps depend on a control plane you don't own. If step one is "replace the node," write down what you'll do when the provider's launch API is the thing failing. Second, put limits on your own automation. A health checker or autoscaler that can cycle half your fleet in a few minutes is, under overload, a way to make the overload worse, and capping how many instances it may act on at once costs very little.

Monday's issue argued that most of the time in a software outage goes into working out what broke. Recovery steps that silently fail are one of the ways that time stretches out. Picture a team spending twenty minutes wondering why new pods won't schedule before anyone thinks to check whether the cloud can launch a VM at all.

In most software and configuration outages, the fix takes minutes and finding it takes hours. Read enough public postmortems with timestamps and the pattern is hard to miss: the rollback, the flag flip, the bad file swapped for a good one all show up near the end of the timeline, as a single line. Everything above that line is people trying to work out what's going on.

That time barely registers in how we talk about incidents. We count time to detect and time to resolve, and the long stretch between them, where the team is looking at the wrong thing, doesn't get a number of its own.

Two Kinds of Outage

Axis Intelligence keeps a cloud outage tracker that tries to measure this directly. It takes the minutes from the start of customer impact to the operator's published root-cause identification and divides that by total customer-impact minutes. They call it a Diagnostic Lag Ratio. Across the six recent major incidents they scored, the split is stark: the two power and cooling failures scored zero, because everyone knew immediately what had broken, and the software and configuration failures averaged 0.41. Their summary: "Physical failures are recovery-bound, software failures are diagnosis-bound."

The worst case in their table is a Microsoft Azure OpenAI incident on May 29, 2026, where 401 of the 446 minutes of customer impact passed before the root cause was identified. For more than six and a half hours, the constraint on recovery was knowing what to fix.

The tracker is upfront about its limits. It only works for operators transparent enough to timestamp their own confusion, and in its own words it measures "published diagnostic narrative, not ground truth." Six incidents is a small sample. But the direction matches what anyone who's run a config-driven incident already knows from the inside.

What Diagnosis Looks Like From Inside

Cloudflare's postmortem of its November 18, 2025 outage is one of the most honest accounts of this you'll find. A database permissions change went out at 11:05 UTC. Impact started at 11:20. The actual cause was a Bot Management feature file that had doubled in size and was crashing the proxy that loaded it.

It was 13:37 before the team's focus shifted to rolling that file back. Main impact was resolved at 14:30.

The two hours in between weren't carelessness. The bad file was regenerated every five minutes, and depending on which database node produced it, it was sometimes good and sometimes bad. In Cloudflare's words, "the entire system would recover and then fail again." A network that heals itself and then falls over again looks a lot like an attack. Then Cloudflare's status page, which is hosted entirely outside its own infrastructure, went down at the same time. That was a coincidence, but it "led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our status page."

Every signal the team had was real. Put together, they pointed at the wrong story. This is the same problem as the piece on four monitoring tools showing four views of one incident and the one on assuming clustered incidents are connected: most of an investigation's early time goes to building a story from partial evidence, and a wrong story costs more time than having none.

One detail in that timeline is worth noticing. At 13:05, half an hour before the work turned to the feature file, the team had already switched Workers KV and Access to fall back to an earlier version of the proxy, which cut their error rates sharply. They reduced the damage before they understood the cause.

Ignore the Instinct

Google's SRE book says it bluntly in the chapter on troubleshooting. Your first response in a major outage may be to go straight for the root cause: "Ignore that instinct!" The job is to "make the system work as well as it can under the circumstances," and to preserve logs and evidence so the root cause can be found afterward.

Nearly everyone agrees with this, and few teams actually work that way. The incident channel fills with hypotheses, and the one person asking "can we just roll back the 11:05 change?" gets told it's probably unrelated. Sometimes it is. But the costs aren't symmetric. Rolling back a change that turns out to be innocent costs you a redeploy and some pride. Holding off until you're sure costs you every minute of the diagnosis.

A useful rule for the incident commander: once the investigation passes a fixed point, say 20 minutes, with no confirmed cause, the default becomes reversing the most recent change in the blast radius, whether or not anyone can explain the connection yet. The piece on routine changes covered why the most recent change is guilty more often than it looks. A default rollback also gives the investigation a clean test. If impact stops, the search space shrinks to one change. If it doesn't, you've ruled a suspect out in minutes instead of arguing about it for an hour.

Hindsight makes this sound easier than it is. Cloudflare's triggering change was a database permissions tweak, and nothing about it said "this will crash the proxy." The rule isn't a guarantee that you'll roll back the right thing. It just makes rolling back the default instead of something the team has to argue for.

Mitigations You Can Pull Without an Explanation

The fastest recoveries in public postmortems usually come from a mitigation that was built in advance and doesn't require knowing why things broke. Google Cloud's June 12, 2025 incident is the clearest example. A policy change with blank fields hit an untested code path in Service Control, Google's API management layer, and crashed it in every region. The new code wasn't behind a feature flag, but it had shipped with a "red-button" to turn that serving path off. Within 10 minutes the team had identified the root cause and was putting the red-button in place. By 40 minutes it was fully rolled out and regions were recovering.

That's what having the switch ready before the incident buys you. If your system's only lever is "diagnose, write a fix, ship it through the normal pipeline," then diagnosis time goes straight into the outage. Feature flags, kill switches for new code paths, the ability to drain a region or pin a config to its last-known-good version: each one lets whoever is on call cut the impact while the investigation continues.

For each of these, ask a practical question: can the on-call engineer use it at 3 a.m. without a senior engineer on the call? If using it takes someone who understands the system well enough to diagnose the problem, it won't save you any diagnosis time.

When Diagnosis Isn't the Bottleneck

Not every long software outage is diagnosis-bound, and treating them all the same leads to the wrong playbook.

AWS's October 2025 us-east-1 outage started at 11:48 PM PDT, and by 12:38 AM AWS engineers had identified DynamoDB's DNS state as the source. DNS was fully restored by 2:25 AM. The last services didn't recover until 2:20 PM that afternoon. The long tail came from EC2's droplet workflow manager, which in AWS's words had "entered a state of congestive collapse and was unable to make forward progress," with work timing out and being requeued faster than it could finish. Engineers had to throttle incoming requests and restart components to clear the backlog.

Google Cloud's June incident had a smaller version of the same thing. The red button went out quickly, but restarting Service Control tasks created a herd effect on a Spanner table they depended on, because Service Control didn't have randomized exponential backoff in place. Recovery in us-central1 took up to about 2 hours and 40 minutes.

Once you know the cause and the system still isn't coming back, the playbook changes: shed load, throttle, restart in batches, and let queues drain instead of pushing them. It's the same retry-amplification pattern from Roundup #1. The first thing to settle in any long incident is which constraint you're under. Are you still looking for the cause, or have you found it and the system can't recover on its own?

Put the Gap in the Postmortem

If diagnosis is where the time goes, the postmortem timeline should show it. Add two timestamps most templates skip: when the team first had a working hypothesis that turned out to be correct, and when a mitigation was first available, as opposed to when it was used. The gap between those two and the start of impact is the part of the incident your tooling and your process actually control.

Then ask the questions that gap raises. What did we believe during that stretch, and why? Which signal pointed the wrong way? Was there a lever we could have pulled earlier, and if not, why doesn't one exist? Those questions produce action items that shorten the next incident, which a line like "improve monitoring" never does. That's the difference the postmortems without teeth issue was about.

The fix in most of these incidents was one line. The work worth doing is on what it took to find it.

Incident work is some of the hardest work to get credit for in a promotion case, and on-call engineers usually find that out the first time they sit down to write their own. You remember the 2 a.m. database failover, the rollback you called while everyone else was still reading logs, the alert you rewrote so nobody got paged for it again. Then you open the self-review template and none of it seems to fit the boxes.

This one is for anyone heading into a review cycle with a year of pager duty behind them and not much on paper to show for it.

Why Incident Work Disappears by Review Time

There are a few reasons, and none of them are about whether the work mattered.

The first is plain memory. An incident is intense for three hours and then it's over, and by the time promotion season comes round, months later, the details have gone soft for everyone, including you. Julia Evans made this point years ago in her essay on writing a brag document: "if you don't remember everything important you did, your manager (no matter how great they are!) probably doesn't either."

The second is that good incident response looks like nothing happened. A well-handled outage is shorter and quieter than a badly handled one, and "it was only 20 minutes" doesn't sound like an achievement to someone who wasn't on the call.

The third is the format. Promo committees mostly read documents written about people they didn't work alongside. They can't weigh what never reached the page.

A Count Isn't Evidence

The most common incident line in a self-review is some version of "Primary on-call for Q2, resolved 30+ incidents." It's true, and it tells the reader almost nothing. It says you were available. It doesn't say what you decided, what changed because of you, or whether the next person on rotation had an easier week.

A MentorCruise career guide for SREs names the early-career version of this: junior engineers "accumulating knowledge without accumulating evidence." The same piece describes the mid-level plateau as "excellent reactive work with no proactive track record." A pager count walks you straight into that. You get very good at responding, which is valuable, and the document you hand in describes nothing beyond responding.

Three Kinds of Incident Story That Hold Up

Not every incident is worth a paragraph. The ones that are tend to fall into three groups.

A decision you made under uncertainty. You called the rollback before root cause was known, or argued against a restart that would have wiped the evidence, or escalated to a team who didn't think it was their problem. At senior level a committee is largely asking whether you exercise judgment when the information is incomplete, and an incident is one of the few places that judgment gets recorded with timestamps.

A fix that removed a class of incident, not one instance. This is where the proactive track record comes from. If a postmortem action item you owned stopped a whole category of page from firing again, it belongs in the case, with before-and-after numbers if you have them. It's also why I keep arguing that postmortems need action items with owners and deadlines. An action item that never ships is a promo story that never exists.

A change to how the team responds. You wrote the runbook other people now reach for, fixed the alert routing, took the incident commander role so others could stay on the fix, or wrote the shift notes that stopped the next person from treating three unrelated pages in one night as one incident. Tanya Reilly calls this kind of thing glue work. It makes other people faster, and it's very easy to leave out because it doesn't feel like "your" work.

Rewriting One Line

A hypothetical, to make it concrete. Say the original line reads:

Handled the checkout latency incident in March.

A version a committee can use:

During the March checkout latency incident, spotted connection-pool exhaustion while the team was still investigating the database, and called a rollback of the previous day's config change 25 minutes in, before root cause was confirmed. Owned the follow-up: a pool-saturation alert and a pre-deploy check on pool size changes. No repeat in the six months since, and the check has blocked two bad configs.

It's longer, but every clause is something the reviewer can check or weigh: a decision, when you made it, what you did afterwards, and whether it lasted. Link the postmortem so they can read the details themselves.

Write It the Week It Happens

That rewrite is easy the week after the incident and close to impossible in November. Keep a running file (a doc, a note, whatever you'll actually open) and spend ten minutes on it after each incident you worked: what you decided and when, what you owned afterwards, and a link to the postmortem. When a follow-up lands, go back and add the result.

The same notes pay off outside promotion season. "Tell me about an incident you handled" is close to guaranteed in an interview, which I covered in what SRE interviews are actually testing for, and the stories that work there are the same three kinds as above.

What to Leave Out

Heroics, mostly. "Worked 14 hours straight to restore service" reads as dedication to you and, to a lot of senior reviewers, as a sign that something about the system or the rotation is broken. If you did pull a long night, the more useful story is usually what you changed so nobody has to do it again.

Blame, too, even softened. "Recovered from another team's bad deploy" makes the reader wonder how you'll behave in the next cross-team incident. Describe what you did, not what they did.

And volume. Three incidents written up properly will do more for your case than thirty listed.

None of this replaces a manager who actually advocates for you. It gives them something to advocate with.

Three unrelated companies went down this month, and in every case the failure wasn't where you'd expect to find it. Salesforce's outage wasn't caused by Salesforce's application layer — it was an internal login service running out of room. Microsoft's two-day outage wasn't Exchange Online's mail pipeline — it was an authentication configuration a few layers underneath it. Google Cloud's four-hour blackout wasn't a bad deploy — it was one maintenance ticket carried out too literally. Different companies, different root causes, same shape: the thing that took the platform down was never the thing with its own icon on the architecture diagram.

Salesforce: An Internal Login Service Ate the Whole Platform

On September 16, Salesforce's core platform went dark for roughly 10.5 hours across hundreds of Hyperforce instances worldwide — during Dreamforce, the company's own flagship conference. Salesforce's own explanation was specific: requests were "stalling while waiting on a response from an internal login service, which is using up available server resources." That single stalled dependency was enough to limit the platform's capacity to process any incoming request, not just logins — support-case creation, the Marketing Cloud preferences page, and API traffic all degraded alongside it. Recovery took a phased fleetwide fix plus manual restarts on instances where the automated rollout didn't finish the job. One detail worth sitting with: GovCloud, Salesforce's isolated environment for US government customers, stayed up the entire time — not because it's more resilient by design, but because it doesn't share the same internal login service the rest of the fleet does.

https://www.theregister.com/saas/2026/09/16/salesforce-staggers-back-to-feet-after-global-outage/5296800

Microsoft 365: A Two-Day Outage Traced to Authentication Configuration

Starting August 31 and not fully resolved until September 3, Exchange Online and everything hung off its identity layer — Teams, SharePoint Online, OneDrive for Business, Copilot, Universal Print, Defender XDR — degraded or failed for more than 48 hours. Microsoft eventually traced the pattern to "core authentication configuration issues used by multiple internal services within the Exchange Online infrastructure," later noting the affected footprint was "less than one percent" of its infrastructure. That's the part worth underlining: a misconfiguration confined to under 1% of the fleet still produced a two-day, multi-product outage, because so much else authenticates through that same layer. Size of blast radius and size of root cause weren't related at all here — our take on performance failures that don't trip the usual alerts covers a related version of this mismatch, where the thing that breaks isn't sized the way your monitoring assumes it will be.

https://www.bleepingcomputer.com/news/microsoft/microsoft-exchange-online-outage-causes-email-failures-auth-issues/

Google Cloud: One Technician, 13 Minutes, and Every Fiber Path Unplugged

On September 1, a routine hardware maintenance procedure in the us-central1-b zone took multiple Google Cloud products offline for about four hours and eleven minutes, with traffic drop rates hitting 100% for affected resources. Google's own incident report is unusually blunt about the cause: a technician was supposed to disconnect fiber paths one router at a time, preserving redundancy throughout; instead, "a procedural error meant that the physical maintenance action sequentially unplugged 100% of fiber paths across all devices within 13 minutes." Nineteen products — Compute Engine, GKE, Cloud SQL, BigQuery, Spanner, and more — degraded as a result. What makes this one sting is that Google's infrastructure is explicitly built with "redundancy across multiple routing devices" and diverse power sources specifically so that single or multiple device failures don't reach the customer. That design assumption held right up until a human, in 13 minutes, removed all the diversity the redundancy depended on. We wrote about this exact gap between redundancy on paper and the actual failure domain a few weeks back — this is that pattern again, just with the human as the failure domain instead of a shared rack or region.

https://www.theregister.com/off-prem/2026/09/04/google-engineer-unplugged-every-fiber-they-could-see-and-surprise-took-down-a-chunk-of-the-g-cloud/5294418

The Pattern, Not the Coincidence

To be clear about what these three don't share: no common vendor, no shared timing, no upstream dependency linking them the way three separate incidents turned out to share a single point of failure a few weeks ago did. These aren't connected the way that piece — or last week's look at teams wrongly assuming simultaneous outages must be related — warns against inferring. They're just three independent companies that each learned the same thing in the same month: a login service, an auth config, and a maintenance runbook aren't the product, but they're load-bearing for it anyway.

If your team's incident-response investment is organized around the customer-facing surface — the API, the app, the dashboard — none of these three would have shown up on your radar until users were already locked out. The fix isn't "add more redundancy." All three companies already had plenty. It's auditing what's authenticating, routing, or resourcing your actual product, and naming who owns watching that layer specifically, not just the thing it happens to support. Salesforce and Microsoft can't blame an upstream vendor for this one. Neither can Google. The failure was in-house every time — which is exactly why it's worth reading closely instead of filing under "someone else's problem."

Three separate Cloudflare incidents opened and closed within about 24 hours last week, and none of them shared a cause. A network hardware problem in Ashburn on the evening of September 17. Elevated errors for Durable Objects and several downstream services in APAC a few hours later. A Workers KV issue that took down parts of the dashboard the next morning. Every one of them was minor, every one was resolved within roughly 90 minutes, and every one had a different root cause in a different system.

If you were on call for a service that sits behind Cloudflare that night, you probably didn't experience it that way. You experienced three pages in one shift, and the first thing your brain did was ask what connects them.

Three Incidents, One Night

Here's the actual timeline, pulled from Cloudflare's own incident history:

• 18:29–19:58 UTC, Sept 17 — network performance issues in Ashburn, VA (IAD), attributed to a hardware fault.

• 23:37 UTC Sept 17 – 00:54 UTC Sept 18 — elevated errors for Durable Objects and downstream services (AI Search, D1, Workflows, Containers) specifically in the APAC region.

• 06:42–08:15 UTC, Sept 18 — a Workers KV problem affecting the Cloudflare dashboard.

Different data centers, different products, different hours of the day. The only thing they have in common is the vendor's name in the status page banner. But stack three status-page entries on top of each other in a Slack channel and they stop looking like three incidents — they start looking like one bad night at Cloudflare, and "Cloudflare is having a bad night" becomes the working theory for whatever your own service is doing.

The Reflex to Connect the Dots

This isn't a discipline failure. It's how on-call attention actually works. When a second alert fires while you're still mid-investigation on the first, checking for a shared cause is the cheap, obvious move — it might save you from duplicating work, and if there really is a shared upstream failure, missing it means fixing the same symptom twice. The instinct is correct often enough that nobody thinks to question it in the moment.

The problem is that the instinct doesn't come with an expiration condition. Nothing prompts you to stop assuming correlation once you've checked and found none. So the assumption just persists by default, quietly shaping how you read every subsequent signal for the rest of the incident.

Two Ways to Get This Wrong

Both directions cost you something different.

Assuming a shared cause that isn't there sends the whole team's attention toward one hypothesis. Someone starts building a timeline that stitches Ashburn's network fault to the APAC Durable Objects errors, looking for the dependency that ties them together — hours before, region before, deploy window before. There isn't one. Meanwhile the two incidents that actually needed separate owners and separate fixes both sit half-investigated, because everyone's staring at the same shared-cause whiteboard.

Assuming independence when there is a shared cause is the quieter failure, and it's the one that recurs. If two "unrelated" incidents both trace back to the same recent config change, the same overloaded upstream dependency, or the same deploy window, and nobody checks, you close both tickets, write two separate postmortems, and ship the same latent cause back into production twice. The Roundup on redundancy and failure domains covered exactly this pattern — systems that looked independently redundant but shared one physical or logical dependency underneath, which only showed up when both "independent" paths failed close together.

The Five-Minute Check Before You Merge Incidents

Before your team's mental model of the night collapses into "everything's connected" or "nothing's connected," it's worth spending five minutes making the comparison explicit instead of implicit. Four questions, asked out loud in the incident channel, before anyone commits to a theory:

1. Same service or component? Not "same vendor" — same actual system. 2. Same region or data center? 3. Same recent change — a deploy, a config push, a scaling event — in the window before both incidents started? 4. Shared upstream dependency, even one layer removed?

If the answer to all four is no, say so explicitly and split the incidents — separate owners, separate timelines, separate postmortems. If any answer is yes, that's the thread to pull before doing anything else. Either way, the team now has a stated reason for its theory instead of an inherited one, which matters more than which answer you land on. An unstated assumption is the thing that survives a shift handoff and quietly becomes "fact" three hours later.

This is also where tooling fragmentation makes the problem worse, not better. The piece on four monitoring tools showing four views of one incident covered how Geneos, Grafana, Splunk, and Slack can each tell a slightly different story about the same outage. Multiply that by three simultaneous incidents and everyone on the call is reasoning from a different partial picture, which makes the "are these connected" question harder to answer cleanly and easier to guess at instead.

One Incident's Factors vs. Three Incidents' Factors

It's worth being precise about what kind of multiplicity you're dealing with, because "multiple factors" and "multiple incidents" get conflated constantly. The Verica Open Incident Database's analysis of real postmortems found that the median incident involves 3.5 contributing factors, and incidents with five or more contributing factors take three times longer to resolve. That's a real, well-documented failure mode — a single incident where a config change, a missing alert, and a capacity gap all had to line up together.

That is a different problem from three status-page entries with three separate causes landing in the same shift. Treating a multi-incident night like it's one incident with several contributing factors sends you looking for the wrong kind of connection — you go hunting for the config change that ties Ashburn's hardware fault to a Workers KV bug, when the actual answer is that the vendor just had three ordinary Tuesdays back to back.

Read the Vendor's Blame, Not Just the Timestamp

None of this means ignore the possibility of a shared cause — the Roundup on outages caused by someone else's outage is a reminder that vendor incidents genuinely do cascade, and dismissing every pileup as coincidence is its own version of the same mistake. The fix isn't to default to "connected" or default to "unrelated." It's to make the check itself a habit, cheap enough to run every time three pages land in one shift, so the team's read on the night comes from four explicit questions instead of from however crowded the status page happened to look at 2 a.m.

On-call trust erodes either way you get this wrong — either from chasing a connection that wasted an hour, or from missing one that came back a week later wearing a different incident number. The fix costs five minutes and a habit. The alternative costs a shift's worth of attention spent solving the wrong problem, twice.

TITLE: The AI Skills That Actually Matter for a DevOps or SRE Career Right Now SUBTITLE: Job postings want agents and LLMs on the resume. Interviewers and promo committees are testing for something narrower — and it isn't prompt engineering.

More SRE and DevOps job postings now list AI agents, LLMs, or generative AI as a required skill, and most candidates have no idea what that requirement is actually checking for. It reads like a machine-learning line item bolted onto an ops job. It isn't. Here's what it's actually testing, and what to build toward if you want the requirement to work in your favor instead of against you.

The Job Postings Say One Thing

InterviewStack's analysis of 2,772 SRE postings over a trailing 90 days found 11.4% now explicitly require "new-wave" generative AI skills — agents, LLMs, RAG, prompt engineering, or a named platform like OpenAI or Anthropic. AI agents specifically show up in 6.1% of postings, ahead of LLMs (3.8%) and generative AI as a bare term (2.2%). The same analysis found a real salary premium attached: a median $170,000 for SRE roles requiring new-wave AI skills versus $142,700 without, and the requirement rate climbs steadily with seniority — 8.3% at entry level, 11.7% at senior, 12.6% at staff.

That's a real signal, not noise. But read literally, it also looks like a demand for machine-learning credentials that most production engineers don't have and don't need. The gap between what the posting says and what the interview actually tests is where people either build the right thing or waste six months on the wrong thing.

What the Requirement Isn't Testing

It isn't testing whether you can fine-tune a model or write a RAG pipeline from scratch. Google's own writeup of how its SRE org uses agentic AI in production is explicit about where the human stays load-bearing: agents "must explain and reason about why and how they performed an action, as well as what options were considered and rejected," and higher-risk changes keep a person in the loop even as the busywork around them shrinks. The document's whole framing is about supervising and constraining agents, not building them. That's the actual job description hiding inside the buzzword.

I wrote a few weeks ago about what AI SRE agents actually change about on-call: they're a genuinely strong first responder for investigation and a genuinely weak closer for remediation. The skill gap the market is pricing in isn't "can you build the agent." It's "can you tell, fast, when the agent is wrong."

The Three Things Actually Worth Learning

Read and challenge an agent's reasoning trace, not just its conclusion. The maturity model everyone building these systems converges on — read-only, advised, approved, autonomous — puts a human decision at every tier above the first. That decision is only as good as your ability to spot a plausible-sounding root cause that's actually wrong. This is a debugging skill, not an AI skill, applied to a new kind of coworker.

Know your own observability and runbook debt, because the agent inherits it. An agent correlating signals across a service with thin telemetry and a six-month-stale runbook produces a thin, stale answer with more confidence than it's earned. The team-level argument for closing that gap is the same one I made in production support to DevOps about moving work upstream — it just pays off faster now, because a better input feeds a tool that runs on every incident, not just the ones you personally work.

Own the verification step as a named part of your job, not an afterthought. "The agent says it's fixed" is a hypothesis. Being the person who reliably confirms or rejects that hypothesis — and can explain why — is exactly the kind of judgment call that shows up well in a promo packet, because it's concrete, repeatable, and yours.

How This Actually Shows Up in an Interview

The pattern lines up with what I found researching what SRE interviews are actually testing for now: less "walk me through this log" and more "here's what an agent concluded — do you trust it, and why." The InterviewStack data backs this from the other side: AI-skill requirements are highest in finance (24.1% of postings) and at software companies (19.7%), the same places already running agents in production and therefore already needing people who can supervise them well. If you're prepping for one of those interviews, rehearsing prompt syntax is the wrong practice. Rehearsing "here's a time an automated system was confidently wrong and here's how I caught it" is the right one.

What Not to Chase

Don't go collect a prompt-engineering certificate to satisfy a line in a job posting. It answers a keyword match, not the actual gap the posting is pointing at, and it won't survive five minutes of a real interview once someone asks you to reason about a specific agent decision. The requirement is real; the credential most people reach for to solve it isn't.

The honest version of "AI skills" for this career track in 2026 is closer to what senior production engineers already do well — reading a system's evidence, weighing what a tool is confident about against what it's actually right about, and being willing to override it. The agents just made that skill worth listing on a job posting.

Three outages this month had nothing to do with the team that got paged. GitHub Copilot degraded because an AI model provider had a bad afternoon. Shopify checkouts broke because Klarna did. A production Postgres instance went dark because of an unannounced platform issue on Supabase's side. None of these teams caused their own outage — but each one had to explain it anyway, and the interesting differences are in how each one found out. One caught it through automated monitoring within minutes. One could only point customers at someone else's status page. One had a health check that said everything was fine while the database was unreachable. Same instinct as Roundup #3: read past the headline, and look at what each team actually knew, and when.

GitHub Copilot: The Vendor Broke, the Monitoring Worked

At 13:22 UTC on September 3, GitHub's automated monitoring flagged degraded performance for Copilot's AI model providers. Within three minutes GitHub had scoped it precisely: "experiencing degraded availability for the Grok 4.6 model in Copilot Chat, VS Code and other Copilot products," then two minutes later, "the Grok 4.5 model has degraded availability as well." The cause sat entirely outside GitHub's infrastructure — an issue with the upstream model provider serving Grok. There was nothing for GitHub's own engineers to fix; the incident report says only that they were "working with them to resolve the issue." At 17:11 UTC, three hours and forty-nine minutes after it started, the provider's own mitigation cleared it: "issues with our upstream model provider have been resolved, and Grok models are once again available."

What's worth noticing here is what didn't go wrong. GitHub's monitoring caught the degradation immediately, scoped it to the exact affected models within minutes, and kept publishing honest updates through a nearly four-hour window where the honest answer was "we're waiting on someone else." That's a harder discipline than it sounds — the instinct during a vendor-caused outage is either to go quiet (nothing to report) or to over-explain (make it sound like you're doing more than waiting). GitHub did neither.

https://www.githubstatus.com/incidents/ktdr5t0xwnhp

Shopify Checkout: When the Fix Isn't Yours to Ship

On September 2, Shopify's status page opened an incident at 3:52pm: "an issue with Klarna, which may be affecting checkouts using this gateway." Twenty minutes later, with the problem still live, Shopify's update told merchants to go look somewhere else entirely: "refer to Klarna's status page for their latest updates." By 4:52pm Shopify had narrowed it down — "the problem with Klarna payment gateway as well as related integrations" — and was coordinating with Klarna directly, but the checkout flow for any merchant using Klarna stayed degraded through the afternoon while the fix sat with a company Shopify doesn't control.

This is the least comfortable version of a vendor outage: it's customer-facing, it's revenue-facing, and the affected company's own status page can only narrate someone else's incident. Shopify did the right things available to it — fast identification, honest scoping, a working relationship with the vendor — but "refer to Klarna's status page" is a sentence that only works if merchants already know Klarna sits underneath their checkout at all. A payment method integrated three product layers up doesn't usually get flagged as a dependency worth watching until the afternoon it takes checkout down with it.

https://pulsetic.com/status/shopify/incidents/6916/

Supabase: The Health Check That Lied

On September 15, a production Postgres instance on Supabase went unreachable at 14:16:06Z. Nobody paged on it — not because nothing was wrong, but because the uptime monitor was checking the wrong thing. The team's own postmortem is blunt about it: `/health returns HTTP 200 status: ok while supabase: error`. The health endpoint kept reporting success because it was checking that the application process was alive, not that it could actually reach the database. The outage ran undetected until someone noticed the app failing manually; an operator-approved restart through Supabase's Management API finally restored the connection at 15:45:16Z, one hour and twenty-nine minutes after it started. The platform-side root cause was never fully determined.

The fix that came out of it is the real lesson: the team widened an existing monitoring ticket to P1, specifically to make the health check "verify actual database connectivity, not just HTTP 200 responses," and added a deploy-time gate that now fails outright unless the app can confirm `supabase: "connected"` before shipping. That's the same trap Monday's issue described from the alerting side — a check that answers "is the process running" gets treated as though it answered "is the system working," and the gap between those two questions is exactly where an outage hides until a human finds it by accident.

https://github.com/jikig-ai/soleur/pull/8215

The pattern across all three

None of these three teams broke their own system. GitHub's Grok models failed because of xAI's infrastructure. Shopify's checkout failed because of Klarna's. Supabase's platform had a problem nobody outside Supabase could see coming. What actually varied wasn't the failure — it was each team's visibility into it. GitHub's monitoring told them exactly what was down within minutes. Shopify's monitoring told them the truth but the truth depended on a vendor's own status page. Supabase's customer had monitoring that told them a comforting lie.

A dependency you don't control is not optional to monitor just because you can't fix it. You can't patch Klarna's gateway or restart xAI's model servers, but you can decide whether your own health check verifies the thing that's actually load-bearing, or just the thing that's easiest to check. The GitHub and Supabase incidents ran for comparable lengths of time — under two hours each — but one team knew what was happening from minute one, and the other found out by accident. That gap is entirely a monitoring design choice, made long before either outage started.

Performance degradation — the slow, creeping kind that never crosses a hard failure threshold — is now rated by most reliability teams as just as damaging as an outright outage. The problem is that almost nothing in a typical alerting stack is built to notice it happening. Your status page can stay green for hours while a checkout button quietly goes from four hundred milliseconds to four seconds, no pages fire, no one's phone buzzes, and your users start rage-clicking anyway.

The Alert That Never Escalates

On July 16, 2026, Inngest's state store proxy started failing to come back up cleanly after a routine restart. Customers began seeing increased execution latency and elevated execution errors at 09:32 UTC. Checkpointing broke, and SDKs silently fell back to a slower async response path to compensate — which is exactly the kind of self-healing behavior that makes a bad situation look survivable on a dashboard.

An alert did fire, at 09:35 UTC — three minutes after impact began. It just wasn't the alert that mattered. Because the signal was classified as medium priority, it didn't trigger immediate on-call escalation. The degradation ran for 168 minutes before it was fully resolved at 12:20 UTC, and Inngest's own writeup names the real gap plainly: the alerting had a severity level for "something is wrong" but not one calibrated for "something is wrong and getting worse, continuously, for the next two and a half hours."

That's not a monitoring gap. Inngest had a metric, a threshold, and an alert. It's a taxonomy gap — the alerting hierarchy had no tier between "fine" and "down" that matched what was actually happening.

Why Binary Alerting Can't See This

Most paging systems are built around a small set of binary questions: is the process up, is the error rate above a threshold, did a health check fail. Those questions have clean yes/no answers, which is exactly why they're easy to alert on. Latency creep doesn't have a clean answer. A P99 that drifts from 300ms to 3 seconds over ninety minutes never technically crosses "down," and most static thresholds are set wide enough to avoid noisy false pages, which means they're also wide enough to absorb a slow bleed without ever tripping.

We've written before about how four different monitoring tools can each show a technically true, mutually contradictory picture of the same incident — one tool showing current state as healthy while another's time-series view already shows the trend breaking. The degradation problem is a sharper version of the same disease: it's not that the tools disagree, it's that the alerting layer sitting on top of all of them was designed to answer "is it broken" when the real question, most of the time, is "is it getting worse."

The Industry Already Knows This, On Paper

This isn't a fringe complaint. LogicMonitor's SRE Report 2026, based on 418 responses from reliability engineering professionals surveyed in mid-2025, found that nearly two-thirds of respondents — 67% — now say performance degradations are as serious as outages. That's a real, fairly recent consensus shift; "slow is the new down" has become close to a stock phrase in the field this year.

But the same report surfaces the gap underneath the agreement: only 26% of organizations consistently measure whether performance improvements actually affect business outcomes like revenue or NPS. Teams believe degradation matters. Far fewer have built the instrumentation, or the alerting tiers, that would let them act on that belief before a customer notices first. Believing something is serious and having a system that pages you when it happens are two different investments, and most organizations have only made the first one.

When the Outage Ends but the Degradation Doesn't

Railway's July 2, 2026 incident is a useful case because it shows both failure modes stacked on top of each other, with a clean handoff between them. An upstream ISP degradation caused packet loss across US regions starting at 07:44 UTC. At 08:39, engineers disconnected from a secondary carrier without first verifying routing stability, which briefly left the zone with no stable route to the internet — a real, unambiguous outage, about twenty minutes long.

Routing was reconnected and stabilized by 08:59. If your dashboard only tracked "is there a route to the internet," the incident looked over. It wasn't. Storage connections had quietly gotten stuck on a slow management network during the instability and held onto that bad path silently even after the primary failure cleared. I/O wait stayed pinned at 58% — down from nowhere, since routing was "healthy" — for another two hours, until the root cause was identified at 10:45 UTC and I/O wait finally dropped back under 5% by 11:04. Private networking, affecting roughly 20,000 host-to-host links, didn't fully recover until 11:49.

The hard outage was twenty minutes. The part that actually cost the most engineering time to find and fix was the two hours after it, where nothing was "down" by any binary definition and everything was still broken. This is the same shape as the untested-safeguard pattern we covered in a recent roundup — a system quietly holding a bad state that looks fine from the outside because no one built a check for exactly that condition.

What Actually Needs to Change

None of this requires exotic tooling. It requires treating degradation as its own alerting category instead of a lesser version of an outage:

• Add a real severity tier for sustained degradation, distinct from both "informational" and "critical outage," with its own escalation path — not a medium-priority bucket that quietly doesn't page anyone.

• Alert on duration and trend, not just threshold. A P99 that's elevated for two minutes and one that's elevated for two hours are different incidents; most alerting configs treat them identically.

• Put SLOs on the tail latency percentiles customers actually feel (P95/P99), not just uptime and error rate — uptime can look perfect while the experience underneath it quietly erodes.

• Check whether a "healthy" state is actually load-bearing, the way Railway's routing check was healthy while storage was still stuck. A recovered top-level signal doesn't guarantee every dependency underneath it recovered too.

This is the same structural lesson as why routine changes keep causing the worst outages: a process built to review intent — "is this threshold set," "is this change approved" — misses the failures that live in execution detail. An alerting hierarchy built to answer "up or down" misses the failures that live in the slope of a graph.

Reliability teams have already agreed, on paper, that slow is the new down. The next step is making sure something on your team's phone agrees too.

If you've spent years keeping someone else's system alive and you're now trying to move into a DevOps title, the rejection you're hitting isn't about your skills. It's about what your résumé is shaped like.

You know which service degrades first when the queue backs up. You know which 3am alert means "go back to sleep" and which one means "get up now," and you've fixed things under pressure that the people who built them couldn't fix calmly at their desks. None of that shows up as a DevOps qualification, because it isn't written down as one — it's written down as a list of fires, and the job description in front of you is written as a list of pipelines.

That gets talked about as a skills gap. Mostly it isn't. It's a portfolio gap — and it closes faster than the career blogs make it sound (they love the "skills gap" framing, unsurprisingly; it's what sells the course).

The title change is smaller than it looks, and more specific than it looks

Production support and DevOps overlap enough that some companies just use the two names for the same job posting. Where they actually differ is direction of ownership: "DevOps engineers own the pipelines, infrastructure, and automation that get that code to production reliably and quickly," as one plain-language breakdown of the split puts it — the thing that produces the system, not the system itself once it exists. Educative's developer-vs-DevOps rundown draws the line right there.

You've been doing the second half's homework without getting credit for the first half's output. You diagnosed exactly why deploys kept failing on Fridays. Someone with a DevOps title on their badge would have gone and changed the deploy process so Fridays stopped mattering — the same instinct behind the progressive-rollout and blast-radius arguments I made a couple weeks back about routine changes, just aimed upstream at the process instead of downstream at the fire.

The gap is three tools deep, not a philosophy

Nearly 37% of IT leaders now name DevOps as their single largest technical hiring gap — not because the discipline is exotic, but because most candidates can talk about CI/CD in the abstract and can't point to a pipeline they actually own, according to a recent rundown of in-demand DevOps skills. The concrete list under that number is short: an infrastructure-as-code tool (Terraform is still the default), a CI/CD system you've owned end to end rather than "we used Jenkins," and enough Kubernetes to explain what actually happens between `kubectl apply` and a pod taking traffic.

None of that is a career change. It's roughly one system, rebuilt properly.

What you already have that the job posting never mentions

Here's the part nobody puts in the listing: the hardest thing to teach a DevOps hire is judgment under pressure, and you already have it. "Sysadmins who embrace automation and cloud become some of the most effective DevOps engineers," notes one 2026 roadmap for the transition, which also puts the realistic timeline at 6-12 months depending on cloud exposure. That tracks with what I've seen. It's shorter than people assume, because you're not learning a domain from zero — you're learning to describe a domain you already understand in code instead of memory.

Ever watched someone who's never carried a pager automate the wrong thing with total confidence? People who've kept production alive know which failure modes are worth automating around and which ones are one-offs, not worth the engineering cost. A DevOps hire with strong Terraform and no scar tissue doesn't have that filter yet. You do — the same way the agents I wrote about handling on-call toil are decent at the mechanics of a fix and bad at knowing which fix actually matters.

Close the gap with one system, not a certificate

Pick a system you already support — ideally one whose deploy process you've personally cursed at — and rebuild its path end to end: write the Terraform for its infrastructure, put its deploy under a real pipeline instead of a runbook, and if it touches Kubernetes, get its manifests under GitOps instead of hand-applied. Do the one system properly rather than three systems halfway; a single real artifact beats a shelf of finished courses. That's your portfolio, and it's the honest answer the next time someone asks "tell me about something you automated" — a much better question to walk in prepared for than "what's the difference between DevOps and SRE," which nobody agrees on anyway.

The reframe that actually gets you hired

Every incident story you're currently telling is shaped like "I found the problem and I fixed it." For a DevOps interview, tell the version that ends at "and I made it structurally impossible to happen the same way twice" instead — same incident, just a different last sentence. If your Friday-deploy story stops at "I rolled it back," that reads as production support. If it keeps going to "and then I changed the pipeline so a Friday deploy can't take the whole thing down," that reads as DevOps, and you can say it honestly, because you're going to go build it before anyone asks. It's the same shift I noticed on the interview side of this same market: panels increasingly care less about how fast you found the fire and more about what you did so it couldn't restart.

You're not starting a new career. You're finishing the sentence on the one you're already in.

Three outages in the last two weeks share one pattern: the redundancy was real and well-designed, and the outage came through anyway — because it hit a layer the redundancy didn't cover. Google Cloud lost most of a zone to a fiber-maintenance procedure. Claude and Grok went down together because they share a building. A dozen Microsoft 365 products failed on one authentication config. Same instinct as Roundup #1 and #2: read each postmortem past the headline, and look for the shared thing nobody drew on the architecture diagram.

Google Cloud: Redundant Fiber Paths, One Maintenance Procedure

On September 1, part of the us-central1-b zone in Google's Iowa region lost external connectivity from 07:41 to 11:52 Pacific — just over four hours. Google's incident summary names the cause plainly: "the inadvertent physical disconnection of network fiber-optic cables during a routine hardware maintenance procedure." A procedural error during a scheduled capacity upgrade led an engineer to sequentially unplug every fiber path across all the devices involved, inside 13 minutes. Traffic drop rates for resources in the affected area reached 100% at the peak; virtual machines were simply unreachable. Nineteen products were affected, from Compute Engine and GKE to BigQuery, Spanner, and Cloud SQL. The fix was as physical as the cause: technicians reseated the optical fibers, and traffic shifted back to healthy capacity on its own.

Google builds redundancy across multiple routing devices and fiber paths, designed to ride out any single device or path failing. What that redundancy doesn't account for is a single procedure with reach over every path at once — the kind of low-drama routine change that causes a surprising share of serious outages. The paths were diverse; the maintenance operation running across them wasn't. Google has paused maintenance in the region pending audits and promised a full incident report — the thing to watch for in it is whether the tooling could have stopped itself once link loss started climbing across supposedly independent paths.

https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev

Claude, Grok, and ChatGPT: One Window, Two Different Causes

On September 3, Claude, Grok, and ChatGPT all degraded inside the same 90-minute window. The reflex when that happens is to reach for one shared cause — a common cloud, a CDN, an attack. This time two of the three plausibly did share one, and the third didn't.

xAI's explanation for Grok: "We are sorry for the issues you may have experienced with Grok following an outage at our Memphis compute center this morning" — and, notably, "we'd also like to apologize to our impacted compute partners." Claude had a three-hour, six-minute window of elevated errors on Sonnet 5 and other models, cleared at 16:16 UTC. Those two facts connect through a deal signed in May: Anthropic contracted the entire output of xAI's Colossus 1 datacenter near Memphis — 300 megawatts of compute — paying $1.25 billion a month through 2029. The Memphis site runs both xAI's own compute and the Colossus 1 capacity it now leases to Anthropic, so a site-level problem there reaches both tenants at once. That's why xAI was apologizing to "compute partners," and why The Register concluded the facility may have been a shared resource affecting Anthropic's service alongside Grok's.

ChatGPT is the counter-example. OpenAI's account: "A routing error starting around 7:43 am PT on Thursday, September 3 made ChatGPT and Codex unavailable for some users across platforms," resolved inside about half an hour. That was a fault in OpenAI's own routing — no Memphis, no shared datacenter, just timing that lined up with a much longer incident next door. During a multi-party outage, the correlation on your status board is not evidence of a common root; treating it as one sends you inspecting a dependency that's fine while your real fault sits unexamined. It's the same discipline as reading one incident across four monitoring tools — the views lining up doesn't make them right. "We use more than one AI provider" is also worth a second look here: it reads like diversification on a slide, but not if those providers sit in the same building, and compute is now scarce enough that they routinely sublease each other's capacity. The failure domain is the datacenter, not the logo on the contract.

https://www.theregister.com/ai-and-ml/2026/09/03/chatgpt-claude-and-grok-all-had-outages-at-the-same-time/5294322

Microsoft 365: A Dozen Services on One Authentication Config

On August 31, Microsoft acknowledged a Microsoft 365 incident — tracked as EX1464935 — that started around 17:30 UTC and, per reporting, took the better part of three days to fully clear. Exchange Online was hit hardest: failed sends and receives, mailbox-search failures, authentication errors. But the impact ran across Teams, Graph, Purview, OneDrive for Business, SharePoint Online, the 365 Admin Center, Copilot, Universal Print, and Defender XDR. Microsoft's stated cause was "core authentication configuration issues used by multiple internal services within the Exchange Online infrastructure" — a misconfiguration that stopped authentication components from deploying as expected to part of the fleet. Recovery meant re-applying those components across a sample of affected infrastructure, then expanding outward.

Each of those products has its own status page, its own team, its own on-call rotation. On paper they fail independently. They don't, because they sit on a shared authentication substrate, and when that substrate can't deploy cleanly, everything above it starts returning auth errors at the same time. It's the Memphis story one layer up the stack: the thing all these services have in common isn't on the diagram anyone pulls up during a design review.

https://www.bleepingcomputer.com/news/microsoft/microsoft-exchange-online-outage-causes-email-failures-auth-issues/

The pattern across all three

Three different layers — physical fiber, a leased datacenter, an auth config — and the same gap in reasoning each time. The redundancy was built against one failure mode (a device dies, a provider has a bad day, a service crashes), and the outage arrived through a shared dependency one level down that the redundancy didn't span. Google's diverse fiber paths shared a maintenance procedure. Claude and Grok shared a building. A dozen Microsoft products shared an auth component. None of these are exotic — they're the ordinary result of drawing the dependency graph only as deep as the design conversation needed it to go.

The question each of these postmortems leaves you with isn't "do we have redundancy." It's "redundancy against what, exactly — and does it still hold one level below where we stopped drawing." That's usually where the shared failure domain is hiding: just past the edge of the diagram, in the thing everyone assumed was someone else's problem. A dependency you've never traced to the bottom is a single point of failure you haven't found yet.

Open your laptop during a Sev-1 and count the tabs. Geneos on one screen, a Grafana dashboard on another, a Splunk search spinning in a third, and the incident Slack channel filling up faster than you can read it. Each one is telling you something true about the same outage. None of them is telling you the whole thing — and none of them agrees with the others on what time it is, what the service is called, or whether the problem is still happening.

The first twenty minutes of a lot of incidents isn't diagnosis. It's reconciliation: stitching four partial accounts into one story. That work is invisible in every postmortem, and it's a large part of why mean time to resolution stays stubbornly flat even as the tooling gets better.

Each Tool Encodes a Theory of What Monitoring Is

Every monitoring tool was built by people trying to answer one question well. That founding question never fully goes away. It shapes what the tool makes easy, what it makes hard, and what it quietly can't do at all. Run four of them and you're running four theories of what "monitoring" means — and in the middle of an incident, you're the one holding them together.

Geneos: The Present Tense

ITRS Geneos has been the default real-time monitoring layer on a lot of trading floors and banking operations teams for years, and it's built around one question: what is the state of the estate right now? Netprobes on each host sample metrics, a Gateway runs rules against those samples and assigns a severity, and the console shows you a wall of cells that are green, amber, or red this second. ITRS's own docs put the design goal plainly — "Gateways provide configuration for and gather data asynchronously from one or more Netprobes. These collected data are analyzed in real-time" (ITRS).

That focus is a strength and a blind spot. Geneos is excellent at "this specific process has a queue backing up, now." It's weak at "when did this start" and "what did it look like an hour ago," because deep history isn't the point — the dataviews are a live readout, not an archive. A cell that's red now tells you nothing about whether it was red, flapping, or fine ten minutes ago, unless someone happened to be watching it then.

Grafana: The Shape of Time

Grafana starts from the opposite instinct. Its native question is "how has this number moved?" — it renders time-series data from Prometheus, Graphite, InfluxDB and the rest into dashboards built to show change over a window. It's a serious platform now, not just a charting layer: "Grafana Labs is a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms" (Grafana).

On a Grafana panel you can see that latency started climbing at 09:14, plateaued, then stepped up again at 09:30 — the exact shape that tells you whether you're chasing one cause or two. What Grafana won't tell you is why. The panel shows p99 tripled; it doesn't show the stack trace, the failing downstream, or the one customer sending malformed requests. And a dashboard is only as good as the question its author anticipated six months ago — a green dashboard doesn't mean the system is correct.

Splunk: The Evidence Room

Splunk is where you go when you need to know what actually happened to a specific request. It indexes the logs and lets you interrogate them; SPL, its query language, is closer to forensics than to monitoring. Ask it "every 5xx from this service between 14:00 and 14:20, grouped by upstream" and it will tell you — if you can write the query, and if the fields you need were extracted the way you expect.

The cost is speed, in two senses. Splunk's "billing is primarily based on data ingest volume, which can lead to high and unpredictable costs as telemetry grows" (Grafana), so teams under budget pressure log less than they'd like, and the line you need may simply not be there. And a broad search over a bad twenty minutes can take minutes to return while the bridge call waits on you. Splunk has the answer more often than the other three — it's just the slowest to give it up, exactly when you have the least patience for waiting.

Slack: Where the Humans Are

The fourth tool isn't a monitoring tool at all, but it's where the incident actually lives. Slack is where someone says "I think it's the payments deploy from 13:50," where the incident commander assigns roles, and where the real timeline gets written in fragments between "can someone check the DB" and "wait, who owns this service now?"

Slack's blind spot is that it's lossy and ephemeral. The channel holds the richest account of the incident — live hypotheses, ruled-out theories, the exact moment someone spotted the cause — and it's also the least structured, the hardest to search a month later, and gone from everyone's attention the second the channel goes quiet. Half the work of a good postmortem is just transcribing what the channel already knew. Postmortems without teeth often begin as incidents whose Slack history was never turned into anything durable.

One Incident, Four Tools

Here's the same twenty minutes seen four ways. Geneos goes amber on a payment-gateway process: the outbound-queue-depth rule has tripped, and the number is climbing. That's all Geneos will give you — it's bad now, it's getting worse. You open Grafana for the shape: the queue was flat until 15:12, when it began a clean linear climb, no step, no plateau. One cause, started at 15:12. Now you need the why, so you search the gateway logs in Splunk around that minute — connection timeouts to a downstream fraud-scoring service, the first at 15:11:44. Meanwhile the Slack channel has already moved on: someone from the fraud team posted at 15:16 that they'd pushed a config change at 15:10 and were rolling it back.

Four tools, and the whole story only exists across all of them: what (Geneos), when and what shape (Grafana), why (Splunk), who and what next (Slack). Miss one and you're guessing. The skill isn't mastering a single tool — it's knowing which one to reach for at each step, and carrying a timestamp from one to the next without fumbling the timezone.

The Gap Between the Tools Is Where Incidents Hide

Four tools, four honest partial views. The damage is in the seams.

• No shared clock. Geneos shows one timezone, Grafana defaults to UTC, Splunk shows whatever the indexer stamped, and Slack renders timestamps in each reader's own zone. Reconstructing "what happened first" turns into timezone arithmetic under pressure — a reliable way to pin an outage on the wrong change.

• No shared vocabulary. The service is `pmt-gw` in Geneos, `payments-gateway` in Grafana, and `PaymentGateway` in the logs. Nothing joins those automatically; you do it in your head.

• No shared link. You can't hand a colleague "the thing I'm looking at right now." You send a screenshot, and they go hunting for it again in their own copy of the tool.

Each seam is a small tax. Together they're why a five-person incident call spends its opening stretch just agreeing on what's true. The "the safeguard was the problem" pattern from the last Roundup is easier to fall into when your four sources of truth are quietly disagreeing with each other.

What Actually Helps

"Single pane of glass" is the vendor answer, and it mostly doesn't arrive — you'll have these four tools, or their direct replacements, for years. The realistic fixes are cheaper than a migration.

Standardize the clock. Every dashboard, every log timestamp, every incident channel: UTC, everywhere, no exceptions. It's a one-time configuration pass that deletes an entire category of mistake.

Standardize the name. Pick one canonical service identifier and make every tool carry it — a Geneos managed-entity attribute, a Grafana template variable, a Splunk indexed field. Then a search in one tool is a search in all of them.

Make the incident channel the system of record. Not Geneos, not Grafana — the one place that captures human reasoning. Paste links, values, and screenshots into it as you go, so the timeline assembles itself and the postmortem becomes transcription instead of archaeology.

Learn each tool's blind spot before 2 a.m. Geneos has no memory. Grafana has no "why." Splunk is slow and possibly incomplete. Slack is lossy. If you know which question a tool can't answer, you stop burning minutes asking it.

None of this is a tooling project. It's a handful of conventions, agreed once and enforced in config. As the AI SRE agents piece noted, the automated investigators arriving now are good at correlating across exactly these sources — but they need the same things a human does: a shared clock and a shared name. Fixing the seams helps you either way.

The teams that debug fastest aren't the ones with the best single tool. They're the ones who've accepted they have four, learned what each is for, and spent the small effort to make the four line up — so the first twenty minutes of the next incident goes to solving it, not translating it.

For a decade, the SRE interview loop has tested one thing above everything else: how fast you get from a broken system to a root cause. That's the skill AI agents are now best at — and interview loops are quietly retooling around something else instead.

The Loop Built for a World Without Agents

The classic SRE loop is easy to describe because it hasn't changed much in years: a recruiter screen, then rounds built around live debugging under observation, SLO and error-budget math, and a system-design problem with a failure injected partway through. A cottage industry of interview-prep guides has grown up around exactly this shape — walking candidates through error-budget calculations, failure-mode system design, and how to narrate a live debugging session out loud.

That loop made sense when the scarce skill was a human converging on a root cause faster than the outage got worse. It's less obviously the right test now that a growing share of that investigative work — log correlation, timeline reconstruction, ruling out the obvious suspects — is something an agent does before a human even opens a terminal, a shift I wrote about in what AI SRE agents actually change about on-call: they're genuinely good at investigation and paperwork, and genuinely bad at the judgment call that actually matters: deciding whether a fix is safe to ship, the same gap I wrote about in why routine changes still cause the worst outages.

What's Replacing Debugging Speed

A recent piece from Packt's Deep Engineering put a name to this, interviewing engineering leaders at Honeycomb, Traversal AI, and MongoDB about how their hiring loops have actually changed. The consistent theme: the skills classic SRE interviews measure are depreciating, because agentic tooling now absorbs the time-consuming part of troubleshooting. What these leaders say they screen for instead:

• How a candidate responds to ambiguity. Honeycomb's Reid Savage calls this the strongest predictor he's found: "The largest determinant I see is what they do when faced with an ambiguous problem" — do they ask what's actually being asked, or start solving the first interpretation that comes to mind.

• Whether their motivation actually fits the work. Savage distinguishes this from surface enthusiasm in an interview — someone can perform excitement and still be a bad fit for what the job is actually like day to day.

• Judgment recovery. MongoDB's George Hantzaras looks for candidates who can reconstruct the reasoning behind an undocumented decision — reverse-engineering the tribal knowledge nobody wrote down, which is most of what a real incident actually requires.

• Forward-looking systems thinking. Traversal AI's Anish Agarwal frames the shift as moving past troubleshooting entirely, toward candidates who are "thinking about what the system will need a year from now."

None of these show up on a whiteboard. They show up in how someone talks through a real incident.

Honeycomb Cut the Coding Round

The most concrete change in that piece is Honeycomb dropping its live coding exercise for a pull-request review of real code instead. Savage's reasoning: a PR review surfaces "reading between the lines, social skills, eagerness to help, mindset, and integrity" — the things that predict whether someone is good to work with during an incident, not just good at solving one alone under a timer.

That's a bigger shift than it sounds like. A coding round tests whether you can produce correct output under pressure. A PR review tests whether you'd leave a comment that helps a teammate, or one that just proves you found the bug first. On-call is a team sport; the interview format is finally starting to test for that.

What This Means If You're Prepping

If you're studying for an SRE interview right now, the LeetCode-for-SRE material still has some value — you should be able to define an SLO and reason about an error budget, because it's real vocabulary you'll use. But it's not where the interview is decided anymore, and treating it as the whole preparation is a bad bet.

What's worth more prep time:

• A real incident you can walk through in detail — not the cleaned-up postmortem summary, the actual sequence of "here's what I thought was happening, here's what I was wrong about, here's how I found out." Interviewers asking "walk me through your last significant incident" are listening for the correction, not the happy path.

• A story about undocumented tribal knowledge you had to reconstruct. Every system has decisions nobody wrote down. If you've had to figure out why a service behaves a certain way with no comment, no runbook, and no one left to ask, that's exactly the judgment Hantzaras describes screening for.

• An honest answer about what you'd do with an ambiguous ask. If a hiring manager hands you a vague problem statement, the wrong instinct is to immediately start proposing a solution. The stronger move — and the one Savage says he's actually looking for — is asking what problem they're really trying to solve first.

If you've written a postmortem where the action items never actually shipped — a failure mode I covered in postmortems without teeth — that's also worth having ready. Being able to say plainly why a fix didn't happen, and what you'd do differently, reads as more senior than pretending every incident you've touched closed cleanly.

The Same Read Applies to Whoever's Hiring

If you're on the other side of this, running an SRE loop that still leans entirely on live debugging-under-observation and error-budget trivia, it's worth asking what that round is actually predicting anymore. It's not that those skills stopped mattering — it's that they stopped being the differentiator, because the investigative grunt work is increasingly done before a human is even in the loop. The differentiator moved to judgment, and judgment doesn't show up on a whiteboard. It shows up in how someone talks about the incident that didn't go the way they expected.

Three outages hit three different companies in the last two weeks, and each one traces back to a safeguard that didn't do its job — one that never existed, one that actively lied about the state of the system, and one that let a bad change through without catching it. None of these are stories about hardware breaking or engineers making an obvious mistake. They're stories about the layer that's supposed to catch failure before a human has to — a different angle than the last Roundup's look at retry logic amplifying failures, but the same instinct: read the postmortem past the headline.

Proton: A Cooling Failure With No Alarm

On the night of August 26 into 27, temperatures in Proton's Frankfurt datacenter went from 21.8°C to 51.9°C in under half an hour — some probes read 60°C. The trigger was mundane: air filters were being replaced on redundant compressors, and the datacenter operator didn't flag the work in advance or communicate the failure promptly once it started. Network cards built to run at 45°C hit 105°C and shut themselves down to survive. Some servers didn't; Proton's own report calls it "heat death," with an unknown effect on the lifespan of the equipment that did survive.

The part worth sitting with isn't the compressor work — maintenance goes wrong sometimes. It's that Proton found out about a cooling failure in its own datacenter the same way its users did: when things had already gone critical. There was no independent, facility-level environmental alerting sitting between "cooling stopped" and "servers are overheating." That gap is now one of Proton's own stated remediation items, alongside accelerating new datacenter capacity already underway.

https://proton.me/blog/august-27-outage-incident-report

Nebius: The Health Check That Lied About Recovery

Nebius's August 19 incident starts the same way Proton's does — a storm knocked out both the cooling system and the building-management system that would normally have alerted on it, so nobody got paged until equipment-level thermal alerts started accumulating almost an hour later. About a third of running virtual machines in the region were interrupted before it was over.

But the more interesting failure happens mid-recovery. At 11:01 UTC, while engineers were still working the incident, an automated health check reported the region as recovered and began routing console traffic back into it — into a data hall that was still overheating. It took seven minutes and a manual intervention at 11:08 to stop it. The mechanism built specifically to protect users from a degraded region instead pointed more of them at it.

Nebius's own published remediation list runs to eleven distinct action items, and one of them reads almost like an apology: "make regional health checks used for global traffic management reflect real service availability, so traffic fails over from an unhealthy region and does not return prematurely." That's exactly the kind of action item we've argued needs a named owner and a date, not just one line in an eleven-item list — a health check that lies once will lie again under the same conditions if nothing forces the fix.

https://nebius.com/blog/posts/incident-post-mortem-analysis-us-central1-service-disruption-on-august-19

Snowflake: A Config Change That Slipped Past Review

Snowflake's incident is smaller in scope but ran for nearly twelve hours — 00:00 to 11:55 UTC on September 1 — and it's the cleanest example of the third failure mode. An update to the database objects that manage sign-on for Snowflake's CoWork product carried a configuration error. Customers in affected regions across AWS, Azure, and GCP simply couldn't sign in. No other Snowflake feature was touched, and there was no workaround while the fix was rolled out.

This is a routine-change story, not a hardware story — the same pattern we wrote about after August's run of GitHub and Cloudflare outages: a config change that should have been low-risk reached production without whatever validation was supposed to catch a bad value in an auth-critical path. The blast radius stayed contained to one product, which suggests the isolation worked even where the pre-deploy check didn't.

https://status.snowflake.com/incidents/847dlygl25yb

The pattern across all three

Line them up and you get a spectrum, not three unrelated stories: Proton had no safeguard in the loop at all, Nebius had one that actively misfired at the worst possible moment, and Snowflake had one that should have caught a bad change and didn't. Different companies, different systems, same underlying question — not "do we have monitoring/health checks/deploy gates," but "have we actually tested what they do when they're wrong." A safeguard nobody has watched fail is a safeguard you're trusting on faith.

The most damaging outages of 2026 don't start with an exotic failure. They start with a routine change — a credential rotation, a config push, a one-line flag flip that ten people had done before without incident. Uptime Institute's 2026 Annual Outage Analysis puts it plainly: failing to follow established procedures is still the single largest driver of human-error outages, and human error factors into the large majority of outages it tracks. The uncomfortable part isn't that changes break things. It's that we've spent a decade adding process to stop exactly this, and it keeps happening.

The Change That Was Supposed to Be Boring

On March 21, 2025, Cloudflare rotated a set of credentials for R2, its object storage service. Standard maintenance. An engineer ran two Wrangler CLI commands to deploy the new credentials and, in doing so, left off the `--env production` flag. The new credentials landed on the default development environment instead of production. Then the old credentials were deleted from the storage backend — and the production R2 Gateway was left holding keys that no longer worked. For 1 hour and 7 minutes, 100% of writes and roughly 35% of reads to R2 failed globally. Cloudflare's own postmortem is blunt: the incident "happened because of human error and lasted longer than it should have because we didn't have proper visibility" (Cloudflare).

Read that scenario and the instinct is to say the engineer should have been more careful. But "be more careful" is not a control. The command looked right. It ran without error. Nothing in the tool told them the change had gone to the wrong place. The gap wasn't attention — it was that a destructive, irreversible step, deleting the old credentials, was allowed to happen before anything confirmed the new ones were live where they needed to be.

Why the Approval Gate Doesn't Catch It

Most change process is built around a review step: a ticket, a second set of eyes, a change-advisory board for anything deemed risky. That catches changes that look risky. It is nearly useless against changes that look boring, which is where the real damage lives.

A credential rotation, a feature-flag toggle, a Terraform apply with a small diff — these sail through review because there's nothing alarming to see. The reviewer approves the intent ("rotate the R2 keys"), not the execution ("...to the dev environment, followed by an irreversible delete"). The failure is almost never in the plan. It's in the space between the plan and the running system, and an approval gate doesn't sit there.

This is the same structural problem Postmortems Without Teeth described from the other end: a remediation item that reads "add more review to credential changes" produces a heavier process that still doesn't touch the actual mechanism of failure.

"Follow the Procedure" Assumes the Procedure Fits Reality

When the leading cause of human-error outages is failure to follow established procedures, it's tempting to hear that as a discipline problem. Often it's a design problem.

Procedures rot. The runbook was written when the deploy was three steps; it's now seven, and step four is a manual CLI command that "usually works." The procedure says "verify in the dashboard" without saying what specifically confirms success — and a green dashboard doesn't mean the system is correct. It assumes the person running it at 2 a.m. has the same context as the person who wrote it at 2 p.m. six months ago. People deviate from procedures that don't match the system in front of them because following the stale version would also cause an incident.

The fix isn't a sternly worded reminder to follow the runbook. It's to move the checks that matter out of the runbook and into the tooling, where they can't be skipped, misremembered, or done out of order.

The Levers That Actually Move the Needle

Four things reliably reduce change-triggered outages, and none of them is "more approvals."

Progressive rollout. A change reaches 1% of traffic, then 10%, then everything, with real bake time between stages. Cloudflare's remediation moved credential rotation onto hotfix release tooling that "explicitly enforces the environment configuration" instead of a hand-typed command — same principle: the safe path is the default path, and it advances in stages you can watch.

Guardrails in the tool, not the wiki. The `--env production` mistake is impossible if the deploy tool refuses to run without an explicit environment. Every check that matters should be enforced by the thing executing the change, not documented next to it.

Blast-radius limits. Ask before a change: if this is wrong, how much breaks, and for how long? A change that can take down one region is a different risk class than one that hits every region at once. Cell-based architectures, per-region deploys, and staged config propagation exist to make "how much breaks" a number you choose rather than a number you discover.

Reversibility. The Cloudflare incident ran long because a step was irreversible at the moment it mattered — the old credentials were already gone. A change you can undo in seconds is a minor event. A change you can only undo by rolling forward is an incident waiting for a trigger.

Make It Reversible Before You Make It Fast

If you take one thing from this, make it the ordering question: in any change with a destructive step, does anything confirm the new state is good before the old state is destroyed?

Cloudflare's most useful remediation wasn't the extra logging — it was requiring explicit confirmation that the new token's ID suffix appeared in the storage logs before the old token could be deleted. That's a tiny check. It converts an irreversible sequence into a reversible one. Rotate, verify the new path is serving, then revoke. Deploy behind a flag, verify, then remove the old code path. Migrate reads, verify, then cut over writes.

Most routine changes can be reordered this way for almost no cost. The ones that can't are exactly the ones that deserve a full change-review meeting — and now you have a real filter for which changes those are, instead of guessing from how alarming the ticket title sounds.

What Production Support Owns Here

If you're the one holding the pager, you don't own the deploy pipeline, but you own more of this than it feels like.

You see every change-triggered incident as it lands, which means you have the data on which change types actually hurt. Feed that back — not "be careful with config changes" but "the last four Sev-2s were single-command changes with an irreversible step and no staged rollout, here are the four." That's a specific, defensible ask.

You also own the verification half. When a postmortem produces a remediation like "improve the rotation procedure," push for the version that's enforced by tooling and has a confirm-before-destroy step, not the version that adds a paragraph to a document nobody reads under pressure. As the AI SRE agents piece noted, autonomous tooling is getting good at investigating what a bad change did — it's still on us to make the bad change hard to execute in the first place.

Routine changes will always be where outages come from, because routine is most of what production does. The teams that get hurt least aren't the ones with the most process. They're the ones where the routine path is the safe path by construction — staged, guardrailed, and reversible, right up until the moment it isn't allowed to be.