In the world of fintech applications, a delay on a single MT202 will cost banks millions. Here comes production support to resolve this reactively. But what if you could prevent it altogether? Welcome to Production Engineering Digest, where we dissect the big shift from firefighting to architecting unbreakable systems.
What Is Production Support?
“The role of operations is to help the business succeed.”
— John Allspaw (Former VP of Operations, Etsy)
Most people believe production support is a role that just monitors alerts, extracts reports, and attends incident calls. This is misleading. If it were only that, automation would have wiped out the role years ago.
In reality, Production Support looks different depending on who’s watching:
For the business: It protects revenue and systems when they fail.
For developers: It is the middleware between them and real users.
For Support Engineers: It is decision-making under uncertainty with real consequences.
Too often, organizations use support as a buffer for rushed releases, weak observability, or poor CTB (Change the Business) documentation. But in truth, it is a lens into how systems and organizations actually behave under stress.
What Is Production Engineering?
“Everything fails, all the time.”
— Werner Vogels (CTO, Amazon)
Many organizations have started using the term “Production Engineering” in place of “Production Support.” By the looks of it, you might think it is closer to a development role—but it is not.
It is a way of thinking. If your first thought upon receiving an alert is simply to follow a KB (Knowledge Base) and close the incident, you are not yet a production engineer. As a production engineer, you think: Why should I prevent this altogether? How do I make the system resolve it even if it reoccurs? Production engineering starts by accepting failure as a baseline and then designing around it through automation, redundancy, observability, and recovery speed.
The Head-to-Head: Firefighting vs. Architecture
Comparing Production Support and Production Engineering isn’t about comparing “good” vs. “bad”—it’s about comparing reaction vs. prevention. One keeps the lights on today; the other ensures they don’t flicker tomorrow.
The Goal: Restoration vs. Resilience Production Support lives in the “now.” The mission is simple: restore service immediately. Success is a met SLA and a closed ticket. Production Engineering lives in the future. The mission is to make the system unbreakable. Success is an incident that never happens because the failure mode was engineered out of existence.
The Metric: Speed vs. Frequency In support, we optimize for MTTR (Mean Time to Restore). It’s a race against the clock to triage and mitigate. In engineering, we optimize for MER (Mean Error Rate). We don’t care how fast you fixed the ticket; we care how many tickets you eliminated through better design.
The Workflow: Interrupts vs. Projects Support is reactive and interrupt-driven. Your day is dictated by the latest alert, the loudest user, or the newest incident call. Engineering is project-driven and planned. You aren’t chasing individual alerts; you are looking at patterns, analyzing trends, and building automation to handle the “toil.”
The Toolset: Detection vs. Construction The support toolkit is built for investigation: SQL, logs, and dashboards. You are a detective searching for a root cause. The engineering toolkit is built for infrastructure: Terraform, Helm, and CI/CD. You are an architect building a system that can heal itself.
The Outcome: Heroics vs. Systems Production Support is the “hero” culture—saving the day under pressure. Production Engineering is “leverage” culture—reducing complexity so that heroics aren’t required. Support protects the business from the current failure; Engineering protects the business from the next one.
Why the Shift is a Survival Requirement
Production support is under pressure not because teams are failing, but because systems are more interconnected than ever. One failure can cascade across multiple systems; the damage is often done before a human can even intervene.
The economic reality has changed. Downtime is no longer just an internal disruption; it has a measurable and immediate reputational impact. Users no longer compare your performance to yesterday’s; they compare it to the best experience they’ve had on any other platform.
Organizations can no longer simply recruit more people to monitor alerts. The focus has shifted to reducing alert frequency through auto-healing and observability. This shift isn’t a career trend—it is a requirement for survival.
The Fintech Reality: Turning Signals into Systems
In high-stakes fintech, production support is the first line of design—often the only thing standing between a quick restart and a total outage.
Take SWIFT payments: what is usually dismissed as a “data quality issue” handled manually by support can be transformed. By analyzing patterns, teams can create pre-quality checks and auto-retries, moving from reactive to proactive.
Consider an adapter failure:
The Support way: Create an incident, check Linux logs, and restart the adapter manually (MTTR: 15 mins).
The Engineering way: A GENEOS alert triggers a Netcool event, which automatically opens an incident and triggers an auto-restart script (MTTR: <1 min).
Support teams see the truth first. When organizations treat those signals as noise, failures repeat. When they treat them as data, Production Support becomes the seed for Production Engineering.
The Transition Roadmap: Moving Your Work Upstream
Transitioning isn’t about swapping job titles; it’s about shifting how you optimize your time. In support, excellence is fast restoration. In engineering, excellence is prevention.
From where I stand, the shift requires changing your priorities:
Automate before you escalate. Python scripting is your leverage. Small scripts that validate inputs or replay requests reveal patterns that manual work hides.
Understand the system beneath the UI. Don’t just know the dashboards; know what’s behind them—how Linux handles processes and how distributed components interact under load.
Eliminate patterns, not just tickets. The real engineering question is: “What changes ensure this never happens again?”
Communicate in structural terms. Frame your work as class-level improvements, not individual resolutions.
This isn’t a checklist for a weekend. It’s a progression that requires curiosity and the discipline to push your work upstream.
Conclusion: From Heroics to Systems
The shift from Support to Engineering isn’t about escaping the work—it’s about expanding your impact. Support teaches you how systems fail. Engineering gives you the leverage to make those failures rarer and easier to recover from.
But this shift doesn’t happen automatically. Organizations optimize for keeping things running, not for evolving roles. If you wait for permission, you’ll stay reactive. Owning the shift means deliberately moving your work upstream: automating what repeats and turning incidents into structural improvements.
The engineers who succeed aren’t the ones who know the most tools. They are the ones who change what they optimize for: from speed today to stability tomorrow.