How Good Is Your Incident Response, Really?

‘When the pressure is on, we don't rise to the occasion. We fall back to our highest level of preparation.’

It's one of our favourite lines at Uptime Labs (originally by hostage negotiator Chris Voss), and it sits at the heart of a question we put to a room full of incident responders in our latest workshop: how good is your incident response, really? Not how good it felt during the last SEV 1, or how neat your time to recovery is. How good is it today, if/when something breaks in the next hour?

Before we started, we polled the audience. 57% said they were directly accountable for their organisation's incident response capability. Another 14% lead an incident response or security team, and 20% support incident response without owning it.

Then we asked how confident they were that they could objectively measure their team's readiness. Not a single person picked ‘very confident, we have clear data’. Answers ranged from ‘somewhat, we track a few indicators’ to ‘not at all, we don't have a reliable measure’.

And when we asked what made demonstrating readiness hardest, the answers were split almost evenly: knowing what to measure, getting objective evidence, showing improvement over time and, a little further behind, justifying further investment. They're all challenges. Let’s talk about that.

1. Meet Carla

To make this concrete, let's introduce a character. Carla is accountable for incident response in her organisation. Her job title could be anything from SRE lead to CTO, depending on how the company is set up; what matters is that it's her job to know where the team is, and to make sure that capability grows with the business.

Carla sees the symptoms on the ground:

Inconsistent responders (let’s use Stu & Joe as examples)

When Stu is on the call (calm, experienced, handled a good SEV 1 only last week) Carla can relax and go to the strategic meeting she's been planning for weeks. When Joe is running the incident, with little formal training and no practice handling an impatient stakeholder who wants an update before there's anything to say, Carla gets cold sweats. Does she give him the room to learn, or step in and take over? A SEV 1 (also known as a P1), where every second costs the business money, isn't really the place to learn.

Unknown newcomers

As the company grows, new people arrive and eventually get their rite of passage into on-call. Will they run towards the fire with all the skills they mentioned at interview, or will they freeze? It's hard to know in advance. Expertise is built through experience, and so is trust. You can build both through real incidents, but that's an expensive way to learn.

Incident rust

It's been 6 months since the last big incident (touch wood!). That sounds like good news, but readiness is a lot like fitness. You can't say "I went to the gym ten months ago, so I'm fit now." Expertise needs maintenance; without practice it fades.

Then leadership comes asking. There's a peak trading period, a product launch or a big marketing event, and they want to know, simply: are we ready? Carla's sense that the team could improve isn't enough. She needs a way to measure it and a way to communicate it, ideally without waiting for a major incident to put resilience back at the top of the priority list.

2. What benchmarking actually answers

Benchmarking is a way of answering one question: where is our incident response capability right now? More specifically, it helps with three things.

Where are we strong, and where are the gaps?

Incident response has many sides: process, coordination, communication, leadership and mindset. Most teams are good at some of these and need to focus on others.

How consistent is the team?

Your readiness doesn't only depend on how well-prepared you are. It depends on who is on call at the moment things go wrong. That’s exposure: you’re only as strong as your least confident responder. Benchmarking shows where people would benefit from development, and who's excelling and could help others.

How do we compare to others?

There's only one organisation your incident response needs to be perfectly optimised for: your own. Every company's processes, domain, risk appetite and impact profile are different. But it's still useful to see how your responders compare with experts from other organisations in a like-for-like setting.

Crucially, a first benchmark gives you a baseline. Since expertise isn't static, you can benchmark again later and see whether you're improving.

3. No two incidents are ever the same (except here)

This is the hard part. You never experience the same incident twice. Even when the failure mode repeats (yes, another expired SSL certificate) the day is different, the market conditions are different, the third-party dependencies are different, the software has changed underneath you, and the people involved bring different mental models. With AI speeding up how quickly software changes, that's more true than ever.

In a post-incident review you might think of things you could have done differently, but you can't go back in time and try them.

Simulation changes that. In an Uptime Labs drill, you can recreate exactly the same incident many times, and have hundreds of people experience the same failure mode. That does two things:

1. It shows how different approaches influence the incident outcome.
2. It shows that there is no single magic incident response strategy. When hundreds of people tackle the same incident, you see several different strategies that work equally well, and seeing that comparison is very valuable in its own right. In other words, it’s a great way to learn different approaches that you can keep in the locker for future situations/incidents.

The benchmarking drill itself is deliberately non-trivial: think of a tough SEV 1 with layers of complexity built in. That includes things like red herrings, which test whether responders can avoid fixation: holding onto a hypothesis even when the evidence points elsewhere. Other complexity patterns test sense-making, decision-making, carrying out decisions, and the interpersonal side of incidents, including power dynamics.

It's also a sociotechnical simulation, not just a debugging exercise. Responders join a virtual company in their browser with senior executives, customer-facing colleagues and technical troubleshooters, and have to coordinate and communicate as they would in a real organisation.

4. What we measure, and why it isn't a single number

Our scenarios are built using a method from cognitive systems engineering called [staged worlds](https://www.adaptivecapacitylabs.com/2025/11/11/uptime-labs-building-expertise-in-incident-response/#:~:text=Staged world scenarios come from the field of Cognitive Systems Engineering%2C developed in the early 1980s as a way to elicit expertise from nuclear control room operators.). It lets us design incidents that bring out specific expertise, which we group into four categories:

• Leadership: driving the incident home. Taking ownership, declaring the incident, setting the right severity. During a SEV 1, the incident commander effectively has the authority of the CEO.
• Coordination: managing multiple lines of investigation and working theories, and engaging with customers and stakeholders, often with incomplete information (by design).
• Communication: building and maintaining common ground, and translating between technical detail and commercial language.
• Mindset: making judgement calls and trade-offs, adapting when things don't go as expected, forming and testing hypotheses, and knowing when to prioritise restoration over root cause.

Because so many people with different levels of experience have played the same drills, from graduates handling their first incident to veterans who respond on instinct, we can grade the behaviours we observe, and when they happen.

Two things matter to us about how this is presented.

First, the language isn't judgemental. We describe skills as developing, practising, strengthening or fluent. We're not here to criticise anyone; we're here to help people see the shape of their expertise and where they could grow, in a safe space where they can learn together.

Second, we deliberately avoid reducing readiness to a single score. A single number is a very lossy summary of something as complex as incident response, and it's easily misinterpreted. "Our incident response is 35" would be convenient, but it's essentially meaningless. Most of what we evaluate is qualitative, supported by quantitative measures like time to resolve. Teams also get the full transcripts of their simulations to go back through together, plus each participant's reflections: what surprised them, what they learned, and what they'd take back to their day job.

The four categories are, honestly, a work in progress. They come from the resilience engineering and cognitive systems engineering literature, and they're the best framing we have today. We'll keep refining them.

5. What Carla gets back

In a real (anonymised) benchmark we ran recently, the moment that resonated most with the person in Carla's role was simple: she recognised her team in it. The responder she trusted to be on call showed up as fluent. The newcomer showed up as developing. For the first time, she had something tangible to discuss with each person, with the team as a whole, and with her leadership.

Three views tend to matter most.

The team view

Individual results show the range across the group, but the aggregated team view is what really matters to someone like Carla, because that's her real exposure. When the big event arrives, you can't guarantee your most experienced person will be the one on call. In this case, the team's aggregate sat below the cross-organisation average on several categories, even though some individuals were well above it. With a peak trading period coming up, that was a risk she could now name and address.

The time view

Time to resolve isn't the whole story, but it does translate skills into business terms. In one benchmark, a veteran responder moved quickly through the complexity built into the scenario. Another responder also resolved the incident, but about twenty minutes later. A third, newer responder stalled entirely; they had never been in that situation before. Ask yourself: what are twenty minutes of a SEV 1 worth to your organisation?

The evidence-based next step

Benchmarks show where a group needs practice. In one cohort, a scenario was deliberately designed to test taking ownership under high uncertainty and pressure. Most of the group struggled to get past it, compared with more experienced responders across the baseline. That led to a clear recommendation: specific drills to practise that skill, which is otherwise very hard to build without real incident scars. Recommendations based on evidence, not a finger in the air.

None of this is a final judgement. It's a conversation starter for a discipline where it's often hard to know where to start.

6. A few questions from the room

Can any size of team do a benchmark?

Yes. Benchmarks draw on sessions from every organisation that has taken part, which can be hundreds of people. If you're benchmarking dozens of people at once, you might want to split them into cohorts, but we haven't hit an upper limit yet.

Do senior executives ever take part?

The simulations include executives as characters, so responders practise handling a stressed senior leader who needs information now. We also regularly present benchmark results to executive teams as part of building the case for investment. At one company, the founders, who were the original incident commanders, ran the drills themselves and compared how they handled incidents with the way newer generations of responders do it. It was quite an education.

What is ‘mindset’, and why does it matter?

It's the stance you bring to an incident: being curious, taking it seriously, and coordinating with fellow responders in a way that gets the best out of them rather than scaring them. Customers often tell us they want people with ‘the right mindset’. Benchmarking turns that vague wish into observable behaviour. (And it isn't always the best developers who have it.)

Your people are your true source of resilience

Technology, process, AI tooling, telemetry and good architecture all matter to your incident response capability, and we're fully behind them. But at Uptime Labs, our focus is on the humans in the loop. Incidents are inevitable; the chaos isn't.

If you're a Carla (or you work with one), benchmarking gives you three things: 1) a baseline for where your team is today, 2) evidence to make the case for improvement, and 3) a way to show progress when you re-run it in 3-6 months.

Take a look at our sample benchmark report to see the full detail, and if it would help, share it with your peers.