
Ready to make incident response your competitive advantage?
See how Uptime Labs builds provable, scalable incident response capability across your organisation.
2 weeks on and I’m still overwhelmed with learnings of the Incident Fest. My mind map is expanding like an explosion!
I personally watched every single incident drill played during incident fest and admired the level of skill demonstrated by expert incident commanders. But, what really blew my mind was watching another expert, Eric Dobbs, comment on the performance of the expert participants in the drills. He saw things that I missed in plain sight! It was a huge learning experience.
Let’s do an experiment. I’ll share two different exchanges that happened in the heat of an incident. Make a note of your own observations, and later in the post, I’ll share what Eric noticed when he reviewed the same incident. I selfishly would like the validation that it wasn’t just me who missed examples of expertise.
Incident context:
Sarah is the primary IC (incident commander), Alex is deputy IC, Bez is the CEO of the business, Tanya is a platform engineer, and Bob is customer service.
The incident happens in the context of an e-commerce website and every minute of disruption is a big hit on revenue and reputation. The site has been down for a long time already, and senior management have lost patience.
Exchange 1:
Immediate context: There are already multiple streams of activity in progress. The pace of new information and conversations in the incident Slack channel is very high.
The following conversation happens on audio incident bridge. Here is the transcription:
Sarah: “Are you saturated or do you have bandwidth?”
Alex: “Ive been with… I just updated the business comms channel.“
Sarah: “Awesome, thank you. Can you look at that list of deployments and see if anything looks suspicious?”
5 minutes later - the need for executing BCP (business continuity plan) is gaining more support, but IC does not know details of the BCP.
Sarah: “ Do you have bandwidth to read it [BCP]? They sent it over to me. Its this one. “
Alex: “Yeah, once I get this CAN [Circumstances, Actions, Needs report] out, Ill read it. Im almost done.”
What are your observations of the exchange?
Exchange 2:
Immediate context:The CEO (Bez) is running out of patience and, unconsciously, is interfering with the progress of the incident.
What are your observations of the exchange?
What the expert, Eric Dobbs [Principal Incident Analyst], noted:
Exchange 1:
Sarah: “Are you saturated or do you have bandwidth?”
Alex: “Ive been with… I just updated the business comms channel.“
Sarah: “Awesome, thank you. Can you look at that list of deployments and see if anything looks suspicious?”
5 minutes later - the need for executing BCP (business continuity plan) is gaining more support, but IC does not know details of the BCP.
Sarah: “ Do you have bandwidth to read it [BCP]? They sent it over to me. Its this one. “
Alex: “Yeah, once I get this CAN [Circumstances, Actions, Needs report] out, Ill read it. Im almost done.”
Sarah is clearly aware of risk of saturation both of her own and that of other people involved in the incident. This level of awareness takes experience and expertise. It’s very easy and common to completely ignore self saturation levels and unconsciously let it affect your performance (dropping tasks or doing them less thoroughly). However, once you are aware that you are at risk of overload, you can consciously manage it in a controlled way by shedding load or sharing load.
In this example Sarah chooses to share load with Alex but at the same time she is aware that Alex also could be at risk of saturation, so she check s with Alex if he has bandwidth. Eric also notes that Alex demonstrates the same level of expertise by communicating clearly his own load levels setting expectation that he will read the BCP in future once he is done with his task at hand.
The beauty of the skills here is in their simplicity. Once you notice it, it’s so obvious, hard to miss it again. However, I’ve observed so many incidents where everyone involved suffered from lack of managing saturation.
Exchange 2:
Sarah is acutely aware of disruption the CEO (Bez) causing. She also understands that Bez is restless because the business entrusted in him is suffering and he personally can’t help. She has a range of options to limit the damage Bez is creating, but each option has its own consequences in this specific context:
1- Ask Bez to leave the running of the incident to her.
Bez may feel ignored or disrespected. His worries will certainly remain un-answered.
2- Use her deputy to take the conversation with Bez to a separate channel.
The tactic may work for a period of time, but clearly all Bez’s attention is on this incident that’s threatening his business.
3- Engage Bez in productive work that will help the incident.
Sarah goes for this option. She skilfully recognises that Bez might be feeling helpless since he can’t contribute to the resolution. She also recognises that Bez is extremely worried. Taking those points into account, she plays a very smart move: she quickly finds something useful that Bez can do and asks him to help.
The trick was successful in giving Bez a sense of control and engaging him in an activity; which means he will leave the others on the bridge alone until he completes the task. At that point, he will likely come back to report on his task.
I learnt a new way of managing senior authorities that (unconsciously and with good intention) interfere with incident resolution. It’ll take me few more drills to try the technique until it becomes muscle memory - another tool in my toolbox of handling senior execs on incident bridge - that I can leverage during real-life incidents in the right context.
It’s crucial to bear in mind that what we observed in the examples are tactics that were deployed skilfully and worked in the context of these specific incidents. I’m not seeing the tactics as the right answer. Theyre just more tools in your incident management toolbox. There will be occasions when these tactics won’t necessarily help, and youll need to use different tricks.
When an expert observes expertise, they see things the non-expert eye misses. This involuntary ignorance limits the learning experience. And frankly, the vast majority of us suffer from this limitation! Imagine there was a way to catch the opportunity.
Unlock all of the incident fest materials here: Incident Fest 2025



