An Outsider’s Perspective on Incident Response

Introduction: Outside the Instincts of Incident Responders

In one of the greatest observer-participant narratives of all time, The Great Gatsby, the narrator notes, “I was within and without’: a strange paradox that encapsulates the distinction between seeing and being.

This article is written from an outsider’s vantage point - as someone who has worked alongside incident responders, but is not one. It’s based on observations from organising Incident Fest: a week-long event that threw responders from many orgs and roles into shared scenarios - i.e. structured chaos - together. Here, I’ll dissect some of the key themes surfaced from the event, from an outside perspective.

1. There are different types of incident pressure

Any sort of incident induces an injection of panic-induced cortisol. But what kind of pressure do incident responders feel, specifically?

When we pitched the poll to incident responders, we provided a variety of potential options for them to self-identify:

• A. Time pressure – needing to act quickly with incomplete information
• B. Cognitive pressure – juggling many threads at the same time
• C. Social pressure – feeling watched or judged by others (peers, managers, etc.)
• D. Consequence pressure – the fear of making choices that make the situation worse E. Coordination pressure – the challenge of aligning people, teams, or actions
• F. Emotional pressure – feeling overwhelmed, anxious, or isolated
• G. Other

There was a clear winner, and that was time pressure, which clocked in at 33.3% of the vote.

Watching responders narrate their thoughts in real-time during Incident Fest made the poll results visceral. Some active reflections on time were obvious, like Clint from [one of the oldest and largest IT companies] noting ‘They expect it to be fixed in 15 minutes’ before later noting about a task, ‘Like that might take me 15 minutes just to execute’. Some of it was more insidious, like when Sarah from [the largest American cloud-based sales platform] noted, ‘Oh man, why do I type so fricking slow?’.

Measures like time to resolution or time to mitigation, therefore, make sense when every second counts. But do these targets create the pressure responders feel - or simply expose pressure that’s already there? From an outsider’s seat, it’s hard to untangle causality.

Like its sister roles of emergency and aviation, time during incident response is a managed resource. Seeing seasoned teams react almost reflexively to shifting time indicators is a masterclass in practised coordination under pressure.

2. Time, invisible work, and other philosophies

On Tuesday at the festival, we published a short video by John Allspaw entitled ‘What is time, anyway?’. As point #1 suggests, the perception of time changes drastically. Telemetry maps time to the millisecond; human time under stress stretches and blurs.

That mismatch hides effort: the gaps between logged events are where people do the glue work and shave whole minutes off impact. In other words, invisible work is the time the responder never had to spend suffering under the pressure of an incident.

‘Invisible’ Work

Therefore, one of the most interesting philosophies around incident response is that of ‘invisible’ work – i.e. work that exists but is not perceived. Recall the old philosophical question, “If a tree falls in the woods and nobody’s there to hear it, does it make a sound?. Likewise, if an incident responder prevents an incident but no one notices, did it really happen?

When we polled incident responders on invisible work, from the (limited and statistically insignificant) responses, there was a striking contrast between organisational approaches to invisible work.

Executives occupy a liminal spot - close enough to influence the technical response, far enough to miss critical context when work stays invisible.

3. The complex role of executives in incidents

The two webinars focused on the role of executives within an incident. Executives have an interesting perspective, as they exist both inside and outside the incident. Because of the precarious nature of this position, they can have a helpful or harmful impact on the incident response.

And a significant part of the distinction between being helpful and harmful is how executives manage the pressure that incident responders feel. Note this exchange from our bonus webinar:

Hamed (CEO, Uptime Labs): How do you work as a team effectively and not panic when you're facing a maximum uncertainty situation?

Guido Smit, (Head of IT (Engineering and Reliability), ING): You mentioned the word panic. As a senior lead, you should always prevent panic from kicking in.

Another interesting part of the engineer-executive relationship is the cultural aspect: ‘ People in the Netherlands are really well trained to always speak out regardless of your role. But we also have different cultures where if your manager tells you something, you just do it,’ noted Guido.

In other words, the relationship between responders and executives is dependent on how safe people feel when acting under pressure.

Closing thoughts

What emerged throughout the week was the concept of pizza. Often, pizza was baked into the concept of nourishing incident responders: the need to tend physical needs while their minds wrestle with cognitive, coordinative and time pressures (although physical and mental performance seem aligned - ‘ Have people had a chance to get water in the past few hours? Dehydrated brains don't think well’, noted Sarah).

Pizza also seems to be the emblem of supportive executive response. According to Alex Hibbitt (Engineering Director, Storio Group), 'We are all responsible for making sure that our organisations fundamentally operate and make money, but the way that we act on that responsibility is by enabling others. Walking into a room and saying, 'We've got to fix this quickly,' is not enabling others. Walking into the room and asking them what topping they want on their pizza, that's enabling others.'

(We also mentioned in our post ‘My Journey from Engineer to Executive: A Retrospective’ that the least executives can do is provide pizza during an incident!).

‘Pizza’ is therefore a shorthand for leadership that helps, not pressures. It felt good to ‘crack this code’ within the incident response culture and community, of which there are many.

Ultimately, incident response is a deeply technical, philosophical and social field. It’s difficult for anyone on the outside to scratch the surface of the expertise and instincts of experienced incident responders. Nonetheless, working alongside incident responders at Incident Fest was a fantastically dynamic experience.