If you've caused a production outage, the week after it matters more to your career than the outage itself. Most engineers who work near production long enough will break it at some point: a config push with the wrong value, a migration run against the wrong host, a cleanup script that was a lot less selective than you thought. The incident gets resolved in an hour or two. What people remember, months later, is how you handled yourself while it was happening and in the days that followed.
This one is for the engineer who is reading it with a knot in their stomach because last night's incident channel has their deploy in it.
Say It Early, While It's Still Down
The single most useful thing you can do during the incident is put your change on the table before anyone has to go looking for it. Something like: "I deployed the rate-limit config at 14:05, that's my best guess for the cause, I'm preparing a rollback." One message, with a timestamp and what you're doing about it.
It feels terrible to type. It also saves the team more time than anything else you could do, because in most incidents the slow part is working out what changed and why, not the fix, which is the whole argument of Diagnosis Is the Outage. Every minute the room spends reverse-engineering a change you already know about is a minute of outage you could have taken off the clock.
What you shouldn't do is quietly fix it forward on your own while everyone else is still investigating. Even if it works, someone will find your commit in the timeline later, and the conversation about why you didn't mention it is a much harder one than the conversation about the mistake would have been.
The Message to Your Manager
Send it the same day, ideally before they hear about it from someone else. Keep it short and factual: what you changed, what broke, how long it lasted, what's already been done, what you think needs to happen next. Skip the long apology. One "sorry, this one was mine" is plenty; three paragraphs of it hands your manager the job of reassuring you, on a day when they've got their own fallout to deal with.
A hypothetical version:
The checkout errors this afternoon came from my config change at 14:05. Rate limit was set at 50 req/s instead of 500. Rolled back at 14:41, error rate back to normal by 14:48. I'd like to own the postmortem and look at why nothing caught a 10x drop in a limit before it shipped.
That last sentence is the important one. It tells them you're already thinking about the system and not only about yourself.
You Know More About This Failure Than Anyone
John Allspaw's 2012 essay on blameless postmortems at Etsy is still the best thing written on this, and it makes a point that's easy to miss when you're the one who's embarrassed. The person who made the mistake has information nobody else has: what they expected the command to do, what they looked at before running it, which assumption turned out to be wrong. In Allspaw's words, "They are, after all, the most expert in their own error."
So write your part of the timeline in detail. Not a defence, an account. "I checked the staging value and assumed prod matched" is exactly the sentence a postmortem needs and exactly the one people are tempted to leave out.
GitLab's postmortem of its January 2017 database outage is a good example of what that looks like in public. An engineer trying to fix replication wiped the database directory on the primary instead of the secondary, around 300 GB was gone before they stopped it, and roughly six hours of data couldn't be recovered. The postmortem notes, almost in passing, that the engineer's name in the recovery document "was added by the engineer themselves." Most of what the rest of the industry learned from that incident (untested backups, no one owning data durability) came out because the people involved were open about exactly what they did.
What a Fair Postmortem Says About You
Look at how AWS described the S3 outage of February 2017, the one that took a large chunk of the internet down with it: "an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers." One input was mistyped. The fix AWS described wasn't retraining the engineer; they changed the tool "to remove capacity more slowly" and added safeguards against taking a subsystem below its minimum capacity.
That's the shape you want your own postmortem to have, and you can steer it there. Google's SRE book puts it plainly: "You can't 'fix' people, but you can fix systems and processes to better support people making the right choices." If the draft root cause says "human error" and stops, ask the question that follows: why did a single mistyped value reach production with nothing in the way? Usually the honest answer is a missing validation, a review step that doesn't look at config, or a deploy that went to everything at once. I went through why those routine changes cause so many outages in an earlier issue.
Then volunteer to own the action item that closes the gap, and actually ship it. An owner and a deadline are what separate a postmortem that changes something from one that doesn't, and having your name on the fix changes what people associate with you when the incident comes up again, though it usually takes a quarter or two before you notice.
When the Culture Isn't Blameless
Not every team works like this, and pretending otherwise would be a disservice. Some places still look for a name to put in the root cause, and some managers say "blameless" while running the review like a deposition.
If that's where you are, the advice above still mostly holds, but tighten it up. Keep everything you write factual and timestamped. Describe what you did and what you saw rather than characterising it ("I ran the cleanup against prod" rather than "I carelessly ran..."). Have the harder conversation with your manager one to one, not in a public channel. And keep your own notes on what you flagged before and after, because in a blame culture the written record is your protection.
It's also worth noticing what the organisation does next. A team that punishes people for good-faith mistakes is training everyone to hide the next one, which is how small incidents become long ones. If you see that happen, that's useful information about whether you want to stay.
The Part Nobody Puts in the Postmortem
You'll replay it. Possibly for a while, possibly at 3 a.m., and probably the next time you're about to hit enter on anything touching production. That hesitancy is normal, and it fades faster if you deploy again soon (with a second pair of eyes if that helps) rather than avoiding the pipeline for a fortnight.
It might help to read how a company sounds when it owns one collectively. Cloudflare's writeup of its July 2019 global outage, caused by "a single WAF rule that contained a poorly written regular expression," spends much of its length on the chain of process gaps that let one rule take the network down for 27 minutes. The emotional line in it is in the first person plural: "We're ashamed it happened." Someone wrote that rule. The company still treated the outage as something the company did, and in a healthy team, that's how yours will be treated too.
A Year From Now, It's a Story
Here's the thing that's hard to believe the morning after: an outage you caused, handled well, can turn into one of the better stories in your career. "Tell me about a time you made a mistake in production" comes up constantly in SRE interviews, and as I wrote in what SRE interviews are actually testing for, the answer they're listening for is about judgment and follow-through, not a clean record. The same goes for promotion cases. A fix that removed a whole class of incident is exactly the kind of evidence I described in putting incident work in your promo case, and it counts whether or not you caused the first one.
Nobody senior in this field has a clean record. What they have is a decent account of what they did next.