What is Incident Rust?

Joe Mckevitt
|
November 18, 2026
Tags:
Best Practices
Blog
Incident Management
Simulation
IN THIS ARTICLE

Ready to make incident response your competitive advantage?

See how Uptime Labs builds provable, scalable incident response capability across your organisation.

Incident rust is a concept we’ve created here at Uptime Labs. It describes the loss of fluency that comes from not using the skills needed to respond to a SEV 1 for a long time. It’s not necessarily a lack of experience; it’s that those skills haven’t been exercised recently enough to feel comfortable using them.

As my colleague Stu puts it, it's like getting rusty at any activity you haven't done in a while. Things that were once fluent and instinctive start to falter. Think of playing music in a group after a long break, or playing sports on a team that hasn't played together for a while.

Taking it one step further

I work with customers to help companies practice how to handle major incidents, reducing risk through simulations. When I conceptualise incident rust, the kernel of it is leading a SEV 1 - the most critical type of incident to your org.

There are all sorts of nuances to a SEV 1, in particular:

  • The root cause likely isn’t clear.
  • It's high pressure.
  • There's a lot of business impact & every second counts.

Then the question is: what's your role in the response, and what skills does it require? If you're an incident commander, you're expected to run and lead a SEV 1. Incident rust is therefore about whether you've recently practised the skills you need to do that well. And those skills aren't purely technical. After all, technical skills are cognitive skills too.

The marathon analogy

We first got this from talking to customers. There's expressed anxiety when it's been ages since you've done a SEV 1, and you, your team, your staff, or your org are accountable for handling that SEV 1 effectively. This is accompanied by a sense of tangible dread: it's not if, it's when, and when it does happen, it's going to be awful.

To use an analogy, I used to run a marathon every spring. And I wasn't messing about when I ran my marathon; I spent six months training for it. I was obsessed with it. The marathon was then a celebration lap of all the training I did.

If you asked me to go out now and run a marathon, I would have the same anxiety as I would if I hadn't done a SEV 1 in a while. Even though I've done it before, and I know the skills, they're exercised. So it would be a painful fiasco.

Personally, from running our drills, I notice incident skills start to degrade within a few weeks. Yet some organisations don't have a SEV 1 within a year, so by the time one comes along, those skills have had plenty of time to rust.

My colleague Stu had the same experience: he’s managed incidents after long breaks, and in his experience, confidence takes time to return. And given how fast the system you support can change, it’s easy to feel outdated.

Where does incident rust come from?

If the question is what contributes to incident rust, the frustrating answer is that it depends. But it starts with this: it's been a long time since the staff expected to deal with SEV 1s in the org have actually had that exposure and experience.

So why is that the case? I've seen two examples personally in my own career:

Example 1: the legacy system.

The org was running a legacy system that nobody wanted to touch, as it was critical. People were afraid to restart or change it. So even though we were accountable for the system, for months, we didn't see any SEV 1s.

Example 2: the highly mature team.

Another team I worked with had sophisticated engineering practices, principles and architectures. They baked resiliency into their processes (which is great) and invested a lot of time and effort in it. They used a lot of canary testing, feature-flag testing, blue-green deploys, instant rollback, highly available architecture and a lot of redundancy in their infrastructure. The reward was that SEV 1s didn't happen that often: we could go for months without one.

So you have two extremes there: one organisation where change was glacial, and another with highly mature engineering practices. Both went for a while without a major incident, but for different reasons.

In essence, fewer incidents mean fewer opportunities to practise responding. Whether efforts to improve reliability always lead to fewer incidents is arguable. But whatever the reason for the quiet period, the less practice you get, the rustier you may become.

Will AI SREs make incident rust worse?

At Uptime Labs, we’ve talked a lot about how AI makes things easier, but learning takes effort. When we delegate too much of that cognitive work to AI, we risk losing opportunities to learn. Take post-incident reviews: handled crudely, AI can produce a beautiful 16-page writeup that nobody wrote, few will read and few will learn anything from.

The same applies during incidents themselves. Handling lower-severity incidents helps people build skills they can use in a SEV 1. But if AI SREs handle more of those smaller incidents automatically, when do people get the chance to practise?

With continued progress, AI agents could contribute to incident rust. (Their promise certainly is that). Many orgs aspire to the point where incident rust happens, thanks to good engineering practices and sophisticated incident tools.

Yet, incidents are inevitable. No matter how good your processes & technology are, things outside your control still happen: supply chain issues, third-party outages, etc. And the leftover principle says that, if they do occur, they're probably the hardest ones to resolve. They will require the incident responders to jump on a call and help out.

And then there you are: suddenly you've got a complex, unexpected, ambiguous incident, with the AI SRE in the mix, which could be an enabler or just more noise. But the human responder will still be sitting there, going, ‘uh-oh… It's marathon time’.

What incident rust looks like in the first ten minutes

There are two immediate signs of incident rust during an incident.

The first is general fright and fear. It’s an uncomfortable emotional state. You’re never going to feel awesome going into a SEV 1, but being asked to do something you’re out of practice at is not a nice place to be. For me, that comes out as a lot of expletives.

The second sign is probably frustration that you’re not using the muscle memory you’ve built. As you go through the incident, you’re asking yourself: How do I do this? What do I do again? How does this work? Those little nuggets you’ve learned in your craft come back slower, if they come back at all. So you end up having your practice run in a real incident, and that’s costing the business money.

You can frame all this as a lack of confidence and fluency in which process to follow: where to go, who to call and where to look first.

How it shows up afterwards

After the incident comes the post-incident review (PIR). The biggest signal is likely to be a dominant sense in the PIR that “this could have been easier.” It’s like running a marathon unprepared: people come out saying, “F me, that was harder than it could have been!”

Part of the review is examining how we did as a group and where we can improve. From there, we get into people’s skills, like coordination, diagnosis and communication. People might look back and say, “We could’ve communicated that better in the first fifteen minutes. We could’ve escalated, or done better recruitment, in the first ten minutes. Did we wait too long?”

Some of these may be obvious traits with quick wins. Often, they weren’t used because rust left the person uncomfortable in the situation. All of that adds to how long it takes to resolve the incident.

Going one step further, rust can also mean forgetting how to run an effective PIR, which is a skill in its own right.

What to do if your team is rusty

You have to give your team exposure to incidents. You need to get people match fit. To use the analogy: you need your people to practice, go to the track, do their practice long runs before they take on the marathon - get their mileage up.

There are many ways of doing that. The most expensive and hardest way is to have a string of incidents in your organisation, and share that load across the team so they're all getting that experience. But that's a really expensive, painful and unplanned exercise.

Furthermore, you can’t rely on having the world’s greatest policies sitting on the shelf. Polices/playbooks won’t address incident rust. Nobody wants to refresh themselves with the playbook at 2 am.

So when I was in that position, I thought: right, how can we get in-house drills in there, or practice tabletop exercises and get my team to practise? And that's why we started the Uptime Labs.

Can you measure readiness?

The really straightforward answer is: when's the last time you were in a SEV 1? When's the last time you were in a SEV 1 doing the role you’d be expected to do if a SEV 1 happened tomorrow?

If that answer makes you squirm when you attempt to answer (e.g. you can't remember, or perhaps it was six months ago), then that's a clear signal.

Stu asks the same question: when did they last experience an incident, or a drill?

What you want to be able to say is: we've been practising. We did our practice last month. Here's the data from the SEV 1 that everybody practised on. The next one is coming up in four weeks' time, and we're all gonna practise again. That's a much better readiness state. And ideally, you can also see how we did:

  • What did you find surprising during the incident?
  • What did you find was more challenging than expected?
  • What was confusing at the time?
  • What was easier than you thought it would be?
  • What was obvious after the fact, that was ambiguous at the time?

Stu believes this is possible to a degree. Take two identical companies: one with an incident response team that has never practised, and one with a team that has practised every week for the last six months. Which team would you bet on to manage an incident more effectively?

Incidents are a team sport

A lot of this is about the individual, but you've got to think of the group.

If you have an on-call rota, the incident doesn't care who's on call when it happens. It doesn't care that your most experienced person isn’t on the rota. That's not how it works.

So your incident rust applies to your whole team, or whoever's on call. This is what I’ve sometimes called the exposure problem.

The very important thing is that incident response is a team sport. You might have the sharpest incident commander in your team. But if you're calling someone up because you need them to help you out, and they haven’t done on-call in a year, and they come in and say, "Sorry, what's expected of me here?", that ultimately isn’t going to give you the incident response readiness state your customers and leadership would expect. That's rust.

Rust is a maintenance problem

We believe the true source of resiliency is your staff. If you want your people to be resilient, you, as a proactive leader, have to address the rust like it's a maintenance problem. Incident Rust is one type of cognitive debt, meaning it needs proactive attention in order to counteract the symptoms: lack of creativity, reduced critical thinking, and so forth.

Nobody wants real incidents to address that rust. Lots of continuous incidents are not the ideal state. So you want to do something which is way more cost-effective.

In summary

To pull it together: incident rust is the gap between having done a SEV 1 and being able to do one now. The skills are there; they just haven't been exercised, and the longer the gap, the rustier they get.

That's why it's worth addressing actively, before the incident finds you. The alternative is that your team's practice run happens in production, on a real SEV 1, with the business watching the clock. Production is the worst test system there is: it's expensive, it's unpredictable and you can't repeat it. Here’s what to do instead:

1. Firstly, ask yourself, ‘When's the last time I was in a SEV 1 doing the role that you're expected to do if a SEV 1 occurred tomorrow?’

2. What you want instead is practice. In-house drills, tabletop exercises, whatever gets your people doing the job before they have to do it for real. Then practise again.

That's what we specialise in at Uptime Labs, and you can practise today, right now: try one of the drills I've built. You’ll get a report with all the skills you’ve demonstrated, from fluent to strengthening.

I'd also love to get your feedback. Connect with me on LinkedIn & let me know how you get on.

Joe Mckevitt

Joe is the co-founder and CTO of Uptime Labs. A passionate technologist and developer, he has 17 years’ experience in building and scaling high-performing products and teams. Also a marathon runner, he’s wired for high performance. He loves creating cultures of constant innovation, and coaching people to develop their full potential.

Share this post
Clear blue sky with scattered white clouds.

Ready to make incident response your competitive advantage?

— Chris Voss

See how Uptime Labs builds provable, scalable incident response capability across your financial services organisation.