A rollback puts the old code back. It doesn't put back anything the new code did while it was running: rows it wrote, flags it gave a new meaning to, machines it left unable to take another update. Most rollbacks work anyway, because most changes don't touch any of that. The ones that do are where recoveries go badly wrong, sometimes worse than the original fault.
Last week's issue on where incident time goes argued that after about 20 minutes without a confirmed cause, rolling back the most recent change should be the default. I still think that's right. But it assumes the rollback is safe to run, and this issue is about the cases where it isn't.
What the Pipeline Actually Reverts
When a deploy tool rolls back, it swaps one artifact for another: a binary, a container image, a config file. That's the entire scope of the operation. Everything the new version did to the world outside that artifact stays done.
So every rollback carries an assumption nobody writes down: that the old version can run correctly against the world the new version left behind. For a stateless change, a bug in how a page renders or a bad timeout value, that's trivially true. It stops being true the moment the change wrote something, changed what a value means, or broke the channel the rollback has to travel through. Public postmortems have clear examples of each.
When Old Code Reads New Meaning
The worst rollback on record may be Knight Capital's, on August 1, 2012. The SEC's order lays it out. Knight was deploying new code for the NYSE's Retail Liquidity Program to eight servers. That code reused a flag that used to switch on "Power Peg," an order-routing function Knight had stopped using in 2003 but never deleted. A technician didn't copy the new code to one of the eight servers, and nobody checked.
When orders carrying the reused flag hit that eighth server, the old Power Peg code woke up and started sending orders without tracking whether the parent order had been filled. While the team was trying to find the problem, they did what most of us would do and backed out the new deployment. The SEC's account of that step: Knight "uninstalled the new RLP code from the seven servers where it had been deployed correctly. This action worsened the problem."
The rollback restored the old code on seven more servers. But incoming orders still carried the flag, and the old code still read that flag as "run Power Peg." In about 45 minutes Knight's router sent millions of orders and racked up 4 million executions. The firm lost $460 million.
The lesson usually drawn is about dead code and deployment review, and both apply. The part that matters here is narrower. Once a value has been given a new meaning and other systems have started sending it, you can't roll that meaning back by redeploying. Every flag, enum value, or field whose meaning you change is a one-way door. Add a new one instead.
Rolling Back the Wrong Half
Roblox's 73-hour outage in October 2021 shows the opposite mistake: putting the state back while the trigger stays in place.
Roblox's service discovery ran on a Consul cluster, and that cluster's writes had slowed to a crawl. About twelve hours in, the team shut the whole cluster down and restored it from a snapshot. Their reasoning was sound: "We expected that restoring from a snapshot taken when the system was healthy would bring the cluster into a healthy state." It was healthy for a while. Then, once internal traffic came back, "Consul performance began to degrade again, and eventually we were back to where we started," with median write latency back at two seconds.
The snapshot had rolled the data back. What it didn't touch was the configuration. The day before the outage, Roblox had enabled a newer Consul streaming feature on a backend service responsible for traffic routing. Under heavy load that feature caused the contention, and a separate BoltDB storage bug made things worse. Streaming wasn't switched off everywhere until October 30, more than two days into the outage, and that's when write latency finally dropped.
A rollback has two halves, the code and config that changed and the state that changed, and they can come apart. Reverting only the state means the system rebuilds the same bad state as soon as load returns. Reverting only the code (Knight's case) means the old code meets state it can't handle. Before you revert anything in an incident, it helps to say out loud which half you're reverting and why you think the other half doesn't matter.
Data You Can't Un-Write
Some changes leave nothing to roll back to. In April 2022 Atlassian ran a maintenance script with the wrong list of IDs and the wrong execution mode, and deleted 883 sites belonging to 775 customers. Atlassian's backups held, and no customer lost more than five minutes of data. Even so, some customers were down for up to 14 days.
Atlassian's review explains why. Their data stores held many customers' data together, so restoring meant they had to "manually extract and restore individual pieces from our backups," site by site, into an environment other customers were still using. The site-level deletion "did not have runbooks that could be quickly automated for the scale of this event."
Restoring from backup is a separate operation from rolling back, and it's usually much slower. The day-to-day version of this risk is the schema migration that drops or renames a column. The old code expects that column, so once the migration runs, rolling the code back just gives you a second outage. Any change that deletes or rewrites data should go into the change record as "roll forward only." Writing that down is what makes people treat the change with appropriate care.
When the Rollback Can't Reach the Broken Machine
The last case is the rollback that works perfectly and arrives too late.
On July 19, 2024, CrowdStrike pushed a content update to Windows sensors at 04:09 UTC and reverted it at 05:27 UTC, 78 minutes later. For a fleet-wide push, that's a fast revert. But the update crashed the machines that received it, and a machine stuck in a crash loop can't download the fix. Microsoft estimated that 8.5 million Windows devices were affected. The revert stopped the problem spreading, but it couldn't repair the machines that were already down. Those needed manual remediation, one machine at a time. It's the same shape as Roundup #6, where the usual recovery moves were the ones that failed.
Facebook's October 2021 outage is a smaller version of the same trap. During routine maintenance, a command meant to assess backbone capacity took down every connection in Facebook's backbone network, and with it the tools engineers would have used to undo it. The total loss of DNS "broke many of the internal tools we'd normally use to investigate and resolve outages," and engineers had to get into the data centers in person, which was slow because those buildings are designed to be hard to get into.
In both cases the path the rollback had to take ran through the thing that was broken. Facebook could undo the damage once someone was physically at the hardware. CrowdStrike could revert the file but couldn't get it to the machines that needed it.
Designing Changes You Can Actually Undo
None of this argues against rolling back. It argues for knowing ahead of time which of your changes can be rolled back, and for making more of them reversible.
Split one-way changes into reversible steps. The standard pattern is parallel change, also called expand and contract: add the new column, field, or endpoint next to the old one; move readers and writers over; remove the old one only after the new one has run in production for a while. Each step can be undone on its own, and the one destructive step comes last, when you're most confident.
Make N-1 compatibility a release check. The old version has to be able to read whatever the new version writes. Test it the way you'd test anything else in staging: deploy the new version, let it write data, roll back, and run the suite against what's left. If nobody has ever rolled back in staging, you don't actually know whether your rollback works.
Never reuse a flag or enum value. That's the Knight lesson, and it's cheap to follow. A new meaning gets a new name.
Ship behind a flag so "undo" is a config flip. If new behavior is behind a feature flag, turning it off doesn't require a redeploy, and the old code path never left. This works only if the flag guards the writes as well as the reads. Otherwise you can turn the flag off and still be left with data the old path doesn't understand.
Keep the undo path outside the blast radius. Out-of-band access to your control plane, break-glass credentials that don't depend on your own identity provider, and deploy tooling that keeps working when the thing it deploys is down. If your rollback goes through the service you're rolling back, assume it won't be there when you need it.
Stage fleet-wide pushes so the revert lands first. CrowdStrike's root cause analysis committed to exactly this: content "successively promoted to wider deployment rings or rolled back if problems are detected." A rollback only helps the machines that haven't been hit yet, and staging the rollout is what keeps that group large.
One Question Before You Revert
The 20-minute default from last week still stands. It just needs one question added before anyone runs the rollback: what has the new version written, sent, or changed that the old version will have to read?
Usually the answer is "nothing," and you should roll back right away. When the answer is a migration, a reused flag, or a queue full of messages in the new format, rolling back may not be the safest move. Turning the change off behind a flag or shipping a quick forward fix can be safer, and the time to decide that is before the rollback runs, not after it makes things worse. It's the same habit behind why routine changes cause the worst outages: the riskiest step is the one everybody assumes is safe.
Know which of your changes are one-way before the incident starts. Once it's started, there's no time to work it out.