Ready to make incident response your competitive advantage?
See how Uptime Labs builds provable, scalable incident response capability across your organisation.
Virtually every incident is surprising. Surprise, however, comes in two distinct flavours: ‘surprise’ and ‘surprise surprise’ (or surprise² for short). An awareness of both flavours is useful, both in preparing for and responding to incidents.
The relationship between surprise and surprise² is sometimes described as the distinction between surprise and astonishment. Resilience engineering folks, on the other hand, talk about ‘situational surprise’ vs ‘fundamental surprise.' I’m partial to surprise², however, because it occupies just a single word in my memory, and (at least for me) seems to encapsulate the distinction nicely.
Simply, a surprise in the context of incident response occurs when events unfold in an unexpected way but within a known frame or model. For example: You're monitoring a web service, and you expect a CPU spike to be caused by a known batch job, but it turns out to be caused by a user mistakenly running a data export. The situation is surprising, but within the bounds of your existing mental model of system behaviour.
Other examples include outages due to capacity constraints, hardware failure or even some types of cyberattacks. Such scenarios can be unexpected, serious and damaging. But they don’t fundamentally lead to a reevaluation of how the system works.
The archetypal non-IT example is buying a lottery ticket and winning. Surprising, unexpected, but plausible.
Surprise² occurs when your underlying model of the world is proven wrong or incomplete. Such events challenge the foundational assumptions or beliefs about how the system works. Case in point: 2024’s CrowdStrike configuration update, which led to the blue-screened death of Windows platforms across the world. Nobody saw that coming! These surprises are not just different due to their size or impact; they're distinct because they force a fundamental re-evaluation of incumbent mental models.
In effect, they demonstrate an extra dimension on top of standard situational surprise. They’re not just surprising: the surprise itself is surprising. Whereas standard situational surprises are unexpected, a surprise² is unimaginable, a fact that has serious implications for one’s ability to prepare.
The archetypal non-IT example is winning the lottery, but you didn’t buy a ticket.
Many (if not most) resilience efforts focus on preventing and resolving standard situational surprises. Typical strategies include (but are not limited to):
- failover
- redundancy
- auto-scaling
- fault tolerance
- playbooks
- runbooks
- backups
- rollback
- code review
…which can all help to stabilise scenarios that have been deemed possible, plausible, or likely.
Surprise², however, is more difficult to design for prior to the fact, because it is, by definition, unimagined. Which isn’t to say that the previously mentioned strategies couldn’t help if surprise² strikes - but this would be through luck, not design.
Surprise² is inevitable. You will be astonished from time to time, as the complexity of modern distributed socio-technical systems is such that unimagined scenarios will emerge. Our mental models are never perfect representations, and they will occasionally require updating or re-evaluating entirely.
So, how does one prepare for the inevitable but unimaginable? How does one prepare to be unprepared?
Surprise² requires a different approach from normal, garden-variety surprise. While many surprises can be partially prevented through good planning and monitoring, Surprise² isn’t as easily defused. Rather, the game of surprise² is less about prevention and more about preparation.
Of course, any scenario that actually plays out must have been possible. It’s the dissonance between incumbent mental models and observed reality that qualifies a surprise as surprise². Therefore, activities that surface, explore and challenge mental models can help train for the possibility of astonishment. Such activities may include:
- Encourage multiple perspectives when designing, reviewing, and operating systems
- Support red teaming or 'pre-mortems' to challenge dominant assumptions
- Encouraging skill diversity, to nurture a diversity of educated opinions rather than (just) siloed specialisms
- Learning from previous incidents that were astonishing at the time.
Perhaps the most valuable approach to acclimating to surprise² is to practice scenarios as a team. Few experiences foster effective coping strategies like the pit-of-the-stomach feeling of being adrift in a baffling incident. That same disorientation makes response work as much about managing stress as about diagnosing and mitigating the problem itself. Practising in a safe environment builds the confidence needed to do both.
And what about during the incident?
Of course, it isn’t always clear at the start of an incident whether one is dealing with a surprise or a multi-dimensional surprise². Something surprising may turn out to be astonishing, and something initially astonishing may turn out to be merely surprising. But if you do find yourself in a scenario where your mental models are under serious strain, you may experience the following.
- Playbooks and runbooks will be less effective
- Improvisation will become more important than following a plan
- A diversity of perspectives may be valuable, especially those that are unafraid to ask ‘stupid questions’
- Multi-teaming may be useful to explore alternate options and develop varied mitigating strategies
- Psychological safety will be more important than ever
One thing’s for sure: you’ve probably been astonished before, and you’ll likely be astonished again. What are you doing to prepare to be unprepared?
Stuart Rimell
