5 Field Notes for First-Time Incident Responders

We speak to new incident responders a lot. One serious source of anxiety for them is the idea of their first critical incident. How could it not be?

As every level-headed engineer knows, incidents are inevitable. But that doesn’t mean your first serious incident has to be a chaotic nightmare that you relive for the rest of your life.

In this article, we’ve compiled a selection of tips we wish we had known the first time we picked up the pager or bore the BlackBerry.

If you think we’ve missed one, send us an email at info@uptimelabs[dot]io and we’ll add it in.

‍

1. Knowing When to Ask for Help (from Rishi)

‘If something is beyond your abilities, never hesitate to ask a lead or senior for help. This is especially common for junior or first-time responders — the urge to prove yourself in front of everyone is often stronger than the willingness to admit you're stuck.

Thoughts start creeping in: what will my lead, my manager, my teammates think of me if I ask for help, admit I can't fix it, or hand over to someone else? The antidote is keeping your focus on three things: recovering the business, understanding why this is happening, and knowing when to ask for help.

‍

2. Remember to Think out Loud (from Stuart)

‘Think out loud during incidents - it’s good when people see an error in your thinking: they can just call it out immediately. This helps avoid ‘creating layers of problems’.

Otherwise, it’s easy to go into yourself and just get nervous if you stop communicating.’

‍

3. 3 Tips in 1 (from Hamed)

• Get hard evidence on symptoms reported. Understand exactly what is not working as expected.
• Note patterns related to users affected: location, flow that is broken, error rates...
• Try to reproduce the issue in production and lower environments
• ‍

4. Keep Asking Questions (Especially Stupid Ones) (from Beth Adele Long)

‘It’s easy to feel pressure to have the right answer in an incident, but asking questions is the best way to keep everyone moving forward. Questions that feel stupid or obvious are often useful because they get people to think out loud, question assumptions, and highlight different takes on what might be happening.’

‍

5. It’s a team sport (from John Allspaw)

‘Remember that incident response is a “team sport” - very rarely (if ever) are incidents handled by one person. Ask coworkers for help!’

If you’ve finished reading this and you’re yet to face your first critical incident, good luck!

(and if you’re a seasoned incident responder reading this, may your next incident be smooth and require minimal applications of caffeine).