Being on the Same Page During an Incident: Not Actually Telepathy

Last week, we discussed “Effective Communication” as a common answer to the question “, What are you actually noticing when you’re noticing great incident response?”.

This week, we touch on an equally common answer to this question: effective collaboration.

Like communication, collaboration is a phenomenon that is easily recognised, both in its presence and from its absence, but it can be frustratingly elusive to describe. Everyone recognises it, but few can define it.

Collaboration is commonly described through metaphor or idiom: -

• “Being on the same page”
• “Singing from the same hymn sheet”
• “In-sync”
• “Speaking the same language”
• “Sharing common ground”

etc…

Regardless, collaboration can feel like magic. When a team is collaborating really well, one could be forgiven for suspecting that some kind of telepathy is involved - but there’s no voodoo or sleight of hand involved.

Last week, we described Clark’s Joint Activity Ladder as a recipe for effective communication. Here we’ll share an essential concept for collaboration from the same source (Common Ground and Coordination in Joint Activity - Klein, Feltovich & Woods 2004), known as ‘Grounding’.

Like communication, collaboration happens in the in-between. It’s something people do together, and it only works if everyone agrees that they’re working toward a shared goal. That shared understanding is the first part of Grounding - it’s called the ‘basic compact’.

The basic compact within an incident is a tacit or explicitly stated agreement between responders that they are setting aside other pre-existing priorities to work together for a period of time necessary to bring the incident to a resolution.

This may sound obvious, but it’s not always crystal clear, especially in the early, 'is this an incident or not?' part of an incident’s evolution. Typically, the trigger for the establishment of the basic compact amongst responders is the formal declaration of an incident and the recruitment of the initial response team.

The basic compact differentiates the response team from ‘interested stakeholders’, ‘rubberneckers’, or consulted subject matter experts. The basic compact is often implicit, especially for experienced teams, though it may benefit from the support of an incident commander or team member to clarify what is expected.

Once established, the basic compact imposes a commitment on team members to maintain alignment between responders. This alignment is termed ‘Common Ground’, and its maintenance is referred to as ‘Grounding’: a process that starts well before the incident, progresses throughout and continues afterwards.

Grounding consists of three parts, which together highlight the expansive nature of collaboration - something that’s developed through experience over time.

1 . Initial Common Ground

Within an incident response team, Initial Common Ground is all the prior knowledge that the team has historically gained about the technical system and the responding individuals involved. This includes the mental models held by individuals about the system, how it works, where it’s robust and less so (the 'explody bits').

It also includes a shared understanding of who does what, who knows the most about different domains, (when referencing Uptime Labs' drills, Tanya’s the platform expert, Mia’s the cybersecurity guru). Initial common ground even encompasses knowledge about how different people respond under pressure (Shay gets really nervous and Hamed’s a picture of calm) and how they are most effectively communicated with (for example, Dan’s diligent and quiet, but Bob’s inquisitive and talkative).

Clearly, initial common ground is built over time, through experience of working together in a variety of situations. It’s also gained in less formal settings - at meal time, over coffee, down at the gym or in the pub!

2 . Public Events So Far

Within an incident, this is everything that’s occurred so far, up to the present moment. When was the incident first detected? What are the observed symptoms? Who’s affected? Which systems are affected? How severe do we think it is? What signals have been observed? What hypotheses have been investigated, and what interventions have been implemented so far?

This information is especially important for responders who join the effort later - but it’s also valuable for those involved from the start. Multiple investigation threads may have emerged, and sharing them helps ensure the whole team stays aligned on what’s happened so far.

Within an incident bridge or a messaging platform, a quick scan of conversation history may be difficult for responders to fully grok - especially as they’re likely to be under cognitive and time pressure. That’s why it’s important to develop strategies that help keep everyone informed about what’s happened so far.

Some teams employ a scribe whose job is specifically to help maintain common ground on public events so far. Other teams are employing AI to summarise events so far.

3 . Current State of the Activity

The final aspect of grounding brings us to the present moment, which is the culmination of collaboration via initial common ground and public events so far.

‘Observability’ is crucial for maintaining common ground about the current state. While the word observability immediately brings to mind important tools such as Grafana, Splunk, New Relic, Datadog, etc (other O11y tools are available), the current state comprises more than the telemetry of the technology under investigation.

The current state also includes the hypotheses under investigation ('we currently suspect a DNS issue'), the interventions in play ('we’re refreshing the cache after rolling back the change'), and the current activities (who’s doing what?) and saturation levels (who’s overloaded?) of the responders: ‘Sarah’s overloaded trying to deflect the CTO’s questions while she’s on the phone to the 3rd party’, ‘Mia’s been working for 6 hours without a break; we need to line up a handover’.

As you’ve probably gathered, grounding is less a ‘thing’ and more a ‘process’ that needs to be continually nurtured before, during and after an incident to enable effective collaboration. The more incident teams experience situations of uncertainty and ambiguity together, the more common ground they will collect and the more effectively they’ll be able to collaborate. Practising incident scenarios together is a fantastic way to build this common ground without the consequences of real incidents.

Beyond simulated scenarios, there are a variety of other ways in which teams can facilitate grounding, reducing the likelihood of a loss of common ground, and restoring it if it indeed has been lost. We’ll cover this in a future post, but for now, I’ll leave you with a question: “What strategies do you and your teams use to nurture common ground at the three levels described? - Initial Common Ground, Public Events So Far, and Current State of the Activity?. We’d love to hear from you.

References / Further Reading

The concepts in this post all come from “Common Ground and Coordination in Joint Activity (2004)” by Klein, Feltovich and Woods. This is a wonderful paper that is well worth a read for any incident responder or budding resilience engineer.