Six months ago, a batch job silently dropped a handful of end-of-day trade confirmations. The postmortem was thorough. It had a timeline, a root cause, and four action items with owners and due dates. Everyone signed off. Two weeks ago, the same batch job dropped confirmations again — different day, same silent failure mode, same root cause the document had already named.

Nobody was surprised. That's the part worth sitting with.

The Document Everyone Signs, Nobody Owns

A postmortem is easy to produce and hard to enforce. Writing it is a few hours of structured reflection. Acting on it is a commitment against next sprint's roadmap, made by someone who usually isn't in the room when the roadmap gets prioritized. The document gets approved. The action items get filed. And then the postmortem's authority quietly expires the moment the meeting ends, because nothing in most engineering organizations actually tracks whether "fix the batch job's failure handling" happened, only whether it got a ticket number.

That's not a discipline problem. It's a design problem — the postmortem process has no mechanism that outlives the meeting.

Why Action Items Die

Three patterns show up over and over in the incidents I've supported:

No owner survives a reorg. The engineer who owned the action item changes teams, and the item becomes an orphan in someone else's backlog, deprioritized by a person who wasn't at the incident and doesn't feel its weight.

Remediation competes with a roadmap that doesn't know it exists. Feature work has a stakeholder pushing for it every week. A postmortem action item has a stakeholder who shows up once, at the retro, and then goes quiet until the incident recurs.

"Low severity" is doing a lot of work it shouldn't. An incident gets labeled low-impact because this time the blast radius was small — a few dozen records, an off-peak window, no client called. The severity label follows the impact of the last occurrence, not the risk of the next one, so the same failure mode keeps getting re-triaged as low priority right up until the day it isn't.

The Recurrence Tax

The cost of a recurring incident isn't just the second outage. It's what happens to the team that watched it happen twice. The on-call engineer who flags "this is the same thing as last time" and gets told the fix is still in the backlog stops flagging it with urgency the third time. Production support engineers are usually the first to notice a pattern repeating, because they're the ones paged for it — and when that observation doesn't change anything, the rational response is to stop treating the postmortem as a real commitment and start treating it as paperwork. That's a more expensive loss than the recurring incident itself: it's the erosion of the one feedback loop that was supposed to make the system safer over time.

What "Done" Actually Requires

Google's SRE postmortem culture guidance is explicit that a postmortem's action items need assigned owners and due dates as part of the artifact itself, not as an afterthought bolted on in a follow-up meeting (sre.google/workbook/postmortem-culture). That's a necessary condition, but in practice it's not sufficient — an owner and a due date only matter if something outside the document enforces them once the due date passes quietly. The organizations that actually close the loop tend to add one more thing: a review cadence that isn't optional, where open action items get reported on whether or not anyone asks.

Treat Remediation Like Debt You Can See

The pattern that's worked best on my team: postmortem action items don't go into the general backlog. They go into a separate, small, visibly-aging list — the same instinct as a technical debt register, but scoped to promises made under incident pressure. Every item carries the date it was created, and once it crosses a threshold (30 days for anything tied to a Sev-1 or Sev-2), it escalates automatically to whoever owns the roadmap, not the engineer who happens to still remember the incident. The point isn't process for its own sake — it's making the cost of not fixing something visible to the person who has the power to reprioritize, instead of leaving that cost invisible until the pager goes off again.

The Role Production Support Actually Plays Here

Production support engineers are usually the ones who notice recurrence first, because they're the ones who feel it first. That makes them the natural owners of the aging list above — not because they should be doing the remediation work themselves, but because they're the people best positioned to say, credibly, "we already wrote this down once." Turning that observation into an escalation with teeth, instead of a comment in a Slack thread that scrolls away, is most of what separates a postmortem process that improves the system from one that just documents its failures.

A postmortem that doesn't outlive its meeting isn't a safety net. It's a record of what you knew before it happened again.

Dashboards are green.
Jobs are marked successful.
No alert has fired.

And yet, the call comes in.

Trades are missing.
Numbers don’t reconcile.
Downstream teams noticed it first.

If you’ve worked in banking production support long enough, this sequence isn’t surprising. It’s expected.

This isn’t a failure of monitoring tools.
It’s a failure of what we’ve decided to monitor.

Green Does Not Mean Correct

Most dashboards in banks answer one question very well:

“Is the system up?”

But banking incidents rarely start with systems going down.
They start with systems doing the wrong thing quietly.

Messages flow — but partially.
Jobs complete — but produce incorrect output.
APIs return success — with bad payloads.
Reconciliation breaks hours later, after market impact has already occurred.

Nothing breached a threshold, because nothing technical was broken.

What Dashboards Are Actually Optimized For

This part is uncomfortable, but important.

Most production dashboards in banks are optimized for:

• SLA reporting

• audit comfort

• noise suppression

• management visibility

They are excellent at answering:

• “Did anything crash?”

• “Are we compliant?”

• “Can we explain this in a review?”

They are far worse at answering:

• “Is the business outcome correct?”

• “Did every expected event actually happen?”

• “What could be silently wrong right now?”

This isn’t accidental. It’s structural.

The Incidents Green Dashboards Don’t See

Some of the most serious banking incidents don’t look like incidents at first.

They look like:

• trades booked upstream but not propagated downstream

• queues draining normally while edge cases are dropped

• vendor feeds delayed just enough to miss cutoffs

• end-of-day jobs marked successful with incomplete data

From a dashboard perspective: all green.
From a business perspective: latent risk.

By the time alerts fire — if they ever do — the damage is already done.

The Hidden Cost Paid by Production Support

When dashboards don’t capture risk, people absorb it.

Senior production support engineers become:

• anomaly detectors

• pattern recognizers

• living incident histories

During market hours, real monitoring looks like:

• manual sanity checks

• expected vs actual counts

• quiet questions like “does this feel low?”

• experience built from past failures

None of this appears on dashboards.
None of this is visible in KPIs.

Yet this human layer is what prevents many silent failures from becoming public ones.

Why This Keeps Repeating

Post-incident reviews usually focus on:

• tighter thresholds

• faster alerts

• more dashboards

Rarely do they ask:

• Which business invariant went unmonitored?

• What signal did a human catch before the system did?

• Why were we blind to this class of failure?

So the same incidents return — slightly modified, slightly renamed, but fundamentally unchanged.

What Actually Improves Signal (Without New Tools)

This isn’t about buying better monitoring.

What helps is changing what you consider observable:

• Expected vs actual flows
  Not just “did the job run,” but “did it produce what the business expects?”

• Absence-of-event alerts
  Silence can be more dangerous than noise.

• Business invariants
  Counts, balances, reconciliation deltas — not just CPU and latency.

• Market-hour thinking
  Risk behaves differently when money is moving.

These ideas aren’t new. They’re just harder to operationalize — and harder to summarize in status decks.

The Real Role of Production Support in Banks

Production support is often described as reactive.

In reality, it’s the last line of defense against:

• financial risk

• regulatory exposure

• reputational damage

When dashboards are green, production support is often the only function asking:

“Are we sure this is correct?”

That question doesn’t show up in SLAs.
But it’s what keeps systems honest.

Final Thought

If your dashboards are always green, it doesn’t mean your systems are healthy.

It usually means:

• risk has been pushed downstream

• detection has been delegated to humans

• experience is compensating for blind spots

The question isn’t whether dashboards work.

It’s who carries the risk when they don’t.

In the world of fintech applications, a delay on a single MT202 will cost banks millions. Here comes production support to resolve this reactively. But what if you could prevent it altogether? Welcome to Production Engineering Digest, where we dissect the big shift from firefighting to architecting unbreakable systems.

What Is Production Support?

“The role of operations is to help the business succeed.”

— John Allspaw (Former VP of Operations, Etsy)

Most people believe production support is a role that just monitors alerts, extracts reports, and attends incident calls. This is misleading. If it were only that, automation would have wiped out the role years ago.

In reality, Production Support looks different depending on who’s watching:

• For the business: It protects revenue and systems when they fail.

• For developers: It is the middleware between them and real users.

• For Support Engineers: It is decision-making under uncertainty with real consequences.

Too often, organizations use support as a buffer for rushed releases, weak observability, or poor CTB (Change the Business) documentation. But in truth, it is a lens into how systems and organizations actually behave under stress.

What Is Production Engineering?

“Everything fails, all the time.”

— Werner Vogels (CTO, Amazon)

Many organizations have started using the term “Production Engineering” in place of “Production Support.” By the looks of it, you might think it is closer to a development role—but it is not.

It is a way of thinking. If your first thought upon receiving an alert is simply to follow a KB (Knowledge Base) and close the incident, you are not yet a production engineer. As a production engineer, you think: Why should I prevent this altogether? How do I make the system resolve it even if it reoccurs? Production engineering starts by accepting failure as a baseline and then designing around it through automation, redundancy, observability, and recovery speed.

The Head-to-Head: Firefighting vs. Architecture

Comparing Production Support and Production Engineering isn’t about comparing “good” vs. “bad”—it’s about comparing reaction vs. prevention. One keeps the lights on today; the other ensures they don’t flicker tomorrow.

The Goal: Restoration vs. Resilience Production Support lives in the “now.” The mission is simple: restore service immediately. Success is a met SLA and a closed ticket. Production Engineering lives in the future. The mission is to make the system unbreakable. Success is an incident that never happens because the failure mode was engineered out of existence.

The Metric: Speed vs. Frequency In support, we optimize for MTTR (Mean Time to Restore). It’s a race against the clock to triage and mitigate. In engineering, we optimize for MER (Mean Error Rate). We don’t care how fast you fixed the ticket; we care how many tickets you eliminated through better design.

The Workflow: Interrupts vs. Projects Support is reactive and interrupt-driven. Your day is dictated by the latest alert, the loudest user, or the newest incident call. Engineering is project-driven and planned. You aren’t chasing individual alerts; you are looking at patterns, analyzing trends, and building automation to handle the “toil.”

The Toolset: Detection vs. Construction The support toolkit is built for investigation: SQL, logs, and dashboards. You are a detective searching for a root cause. The engineering toolkit is built for infrastructure: Terraform, Helm, and CI/CD. You are an architect building a system that can heal itself.

The Outcome: Heroics vs. Systems Production Support is the “hero” culture—saving the day under pressure. Production Engineering is “leverage” culture—reducing complexity so that heroics aren’t required. Support protects the business from the current failure; Engineering protects the business from the next one.

Why the Shift is a Survival Requirement

Production support is under pressure not because teams are failing, but because systems are more interconnected than ever. One failure can cascade across multiple systems; the damage is often done before a human can even intervene.

The economic reality has changed. Downtime is no longer just an internal disruption; it has a measurable and immediate reputational impact. Users no longer compare your performance to yesterday’s; they compare it to the best experience they’ve had on any other platform.

Organizations can no longer simply recruit more people to monitor alerts. The focus has shifted to reducing alert frequency through auto-healing and observability. This shift isn’t a career trend—it is a requirement for survival.

The Fintech Reality: Turning Signals into Systems

In high-stakes fintech, production support is the first line of design—often the only thing standing between a quick restart and a total outage.

Take SWIFT payments: what is usually dismissed as a “data quality issue” handled manually by support can be transformed. By analyzing patterns, teams can create pre-quality checks and auto-retries, moving from reactive to proactive.

Consider an adapter failure:

• The Support way: Create an incident, check Linux logs, and restart the adapter manually (MTTR: 15 mins).

• The Engineering way: A GENEOS alert triggers a Netcool event, which automatically opens an incident and triggers an auto-restart script (MTTR: <1 min).

Support teams see the truth first. When organizations treat those signals as noise, failures repeat. When they treat them as data, Production Support becomes the seed for Production Engineering.

The Transition Roadmap: Moving Your Work Upstream

Transitioning isn’t about swapping job titles; it’s about shifting how you optimize your time. In support, excellence is fast restoration. In engineering, excellence is prevention.

From where I stand, the shift requires changing your priorities:

• Automate before you escalate. Python scripting is your leverage. Small scripts that validate inputs or replay requests reveal patterns that manual work hides.

• Understand the system beneath the UI. Don’t just know the dashboards; know what’s behind them—how Linux handles processes and how distributed components interact under load.

• Eliminate patterns, not just tickets. The real engineering question is: “What changes ensure this never happens again?”

• Communicate in structural terms. Frame your work as class-level improvements, not individual resolutions.

This isn’t a checklist for a weekend. It’s a progression that requires curiosity and the discipline to push your work upstream.

Conclusion: From Heroics to Systems

The shift from Support to Engineering isn’t about escaping the work—it’s about expanding your impact. Support teaches you how systems fail. Engineering gives you the leverage to make those failures rarer and easier to recover from.

But this shift doesn’t happen automatically. Organizations optimize for keeping things running, not for evolving roles. If you wait for permission, you’ll stay reactive. Owning the shift means deliberately moving your work upstream: automating what repeats and turning incidents into structural improvements.

The engineers who succeed aren’t the ones who know the most tools. They are the ones who change what they optimize for: from speed today to stability tomorrow.

Production Isn’t What the Docs Describe

Production never behaves the way the diagrams say it will.
On paper, everything is clean. Predictable. Controlled.

Then you’re on-call, it’s late, alerts are piling up, and you realize quickly: none of the docs prepared you for this version of reality.
If you’ve been there, you know exactly what I mean.

Quick note before going further:
This is my first post here, and I want to be transparent. I used AI to help me shape this article—not to invent experiences, but to help me put words around real ones. The lessons and frustrations come from working in production. The tool just helped me organize the thoughts and get started. Over time, this space will sound more and more like me alone. I care more about sharing real production lessons than pretending everything comes perfectly formed on day one.

What Working in Production Actually Feels Like

It’s not heroic. It’s messy.
You’re juggling partial information, unclear ownership, and systems that evolved faster than anyone documented.

Most days, it feels less like engineering and more like careful observation. Watching behavior. Spotting patterns. Making small, safe moves because the blast radius is unclear.

That tension never shows up in design docs.

Fixing Issues Wasn’t the Hard Part

The fix itself? Usually doable.
A restart. A config change. A rollback.

The harder part was everything around it—figuring out why it happened, explaining it clearly, and knowing whether this was a one-off or the start of a recurring problem.

That context was always the missing piece.

The Same Problems Kept Coming Back

What really wore me down was déjà vu.
The same alerts. The same root causes. Slightly different symptoms.

We’d fix things, close tickets, move on.
And a few weeks later, we were back in the exact same place, asking the same questions, repeating the same work.

It wasn’t lack of effort. It was lack of shared learning.

There Was a Gap No One Was Writing About

There were plenty of blogs about tools and architectures.
Plenty of postmortem templates. Plenty of “best practices.”

But almost nothing about the in-between moments—the judgment calls, the tradeoffs, the quiet lessons you only learn after the third or fourth incident.

Those were the lessons I needed most.

I Wanted a Place for Real Production Lessons

I didn’t want another polished guide.
I wanted a place to capture what actually helped in the moment, while things were still fresh and honest.

This newsletter is that place.
A notebook, really—shared, so none of us have to relearn the same things alone.

What I Plan to Write About Here

I’ll write about incidents and patterns I’ve seen repeat.
About things that looked small but weren’t.
About decisions that made life easier—or harder—months later.

Sometimes it’ll be technical.
Sometimes it’ll be about process, communication, or pressure.
Always grounded in real production work.

Who This Is Really For

This is for you if you’ve ever thought,
“Why does this keep happening?” or
“I wish someone had warned me about this part.”

If you work close to production—support, SRE, backend, ops—you’ll recognize these moments instantly.

If not, you’re still welcome. Just know this comes from the front lines.

Not a Guide. Not a Playbook. Just Reality.

I’m not trying to tell you the right way to do things.
Production doesn’t work like that.

This is about sharing what I’ve seen, what surprised me, and what stuck.
You can take what’s useful and leave the rest.

Why I’ll Keep This Practical

Because fluffy advice doesn’t help at 2 a.m.
Clear thinking does.

I’ll focus on things you can actually apply: questions to ask, small habits to build, signals to watch for before things spiral.

No theory unless it earns its place.

A Quiet Promise to the Reader

I won’t pretend production is glamorous.
But I will be honest about it.

If something here saves you time, reduces stress, or helps you make sense of a messy incident, then this space is doing its job.

If you’ve got a production lesson you wish someone had written down earlier, I’d love to hear it.

Subscribe now