My Journey from Engineer to Executive (A Retrospective)

Note: I gave a talk on this topic at SRECon earlier in 2025, reflecting on what happened when Uptime Labs suffered an outage ('When Uptime Met Downtime'). Since the outage - and indeed, since the talk - I’ve been able to dwell on the nuances of the incident. The roles and responsibilities during an incident are very delicate; the potential for a gap between the expected outcome of our actions and what actually happens is vast. This is exactly what I was thinking and feeling when Uptime met downtime - and what I’ve learned from it now. So, who wants executives on the incident bridge?At best, they can't be helpful. It's astonishing how, with a few words, they can crush the resolver teams' confidence by saying, "Do we have our best people on a call?” or “Do you have a plan?"My perspective has recently shifted from being an engineer to an executive, and I'm going to share a real-life story with you that happened not too long ago.So, in one day, it was a critical week for us. In just one hour on a random Monday morning, life turned upside down. The day started with an alarm going off, and I suddenly discovered that nothing was working. As an executive, I was thinking, “Oh my god, I have two critical sales demos in a couple of hours.” We’re a startup - every sale matters, and the dollar value is high.Sales aside, there was also customer onboarding and shareholder updates. It just threw me into a state of panic. How am I going to deal with all of this? The business is falling apart.So, the first thing I decided is that I'm going to be decisive. I decided to set a deadline for the team to resolve the incident by 11:00, but, of course, incidents don’t always work that way. At a higher level, it's like setting an objective that all Sev1s need to be fixed in 15 minutes. As we discussed a couple of weeks ago, making arbitrary calls about the incident's parameters, such as imposing a resolution deadline or setting a KPI between sev 1s, doesn’t actually help in the moment.Therefore, setting the incident deadline reflected a moment of self-doubt and incompetence. And many thanks to my co-founder who (figuratively) slapped me in the face: "Listen, this is really, really bad, but you're not helping at all. Go and find some way to be useful."From that moment onward, everything useful that I did was 100% unintuitive to me. And the key lesson for me was: executives - they're human too. Their bodies are also full of adrenaline. However, responding to incidents is really difficult: you don't have control. You watch your business, your dream, your success falling apart right before your eyes. It makes you feel exposed and insecure.After all, you can't complain to anyone. It's all under your watch; the buck stops with you. And it really does feel like your loved one is in the operating theatre, and you can't do anything about it. And I'm not exaggerating. It's that kind of feeling (even though no one dies in Uptime Labs' incidents, thankfully).But is any of this exaggeration? No. It's a normal human and animal behaviour. It's instinct: fight, flight or freeze. You might ‘fight’ - for example, by trying to troubleshoot and fix yourself- or ‘flight’ - i.e. put huge pressure on the next person in line to fix it (the ‘your problem, not mine’ mentality). Or you might freeze - not taking any action at all, when it’s most needed.So, how can executives help - i.e. do something genuinely useful - during incidents? Two ways:The first way is by creating space for teams to work - by stepping back, clearing distractions and shielding the responders from outside pressure. That means cancelling non-essential meetings - including sales calls - and letting the team focus.The second way is by nurturing a culture in an organisation that looks at incidents as an opportunity, not a disruption to business. Obviously, this is easier said than done, and requires a dedicated approach to post-incident review that not all businesses will be willing to commit to.Post-incident reviews take real effort - they’re not trivial, but they offer a high return. The question is: how many people across the business actually read them?Executives who can model effective incident response practices will provide a foundation for better incident response (and not just improved vanity metrics).And executives can't do these two things - i.e. create space and learnings - as my good friend John Allspaw said, they can buy a snack for people on call.