After the outage: where did the time go?

This article was created with the AI assistant Claude and published by Peter Hoenig.

After a major outage there is usually a meeting, and in that meeting the wrong question usually gets asked. It is: who was to blame? The right one is: where did the time go?

The difference is not moral but practical. The first question yields a name and nothing else. The second yields a figure you can decide something with.

The seven steps

1. occurred – When did it actually begin?
2. noticed – When did someone or something notice?
3. reported – When did it reach the right place?
4. work started – When did somebody begin working on it?
5. cause found – When was it clear what was wrong?
6. fixed – When was it running again?
7. confirmed – When was it confirmed to be really running?

The first point is the one almost everybody skips – and the one that tells you most. A ticket system does not know it: it knows when something was reported, not when it began. The larger part of the downtime often lies in between.

The figure that matters

From the timestamps the tool calculates the spans, among them the downtime (usually called MTTR, mean time to repair, in IT operations). The most revealing figure, though, is a ratio: the share of the detection time in the total.

An example – invented, but typical: a database shuts down at 04:50 after an automatic update. It is noticed at 07:05 by the early shift, reported at 07:20, fixed at 09:30. Downtime: 4 hours 40 minutes. Of that, 2 hours 15 minutes went by simply because nobody noticed – almost half.

Anyone thinking about faster repairs in that situation is optimising the smaller half. The lever is in detection, and it is usually the cheaper one too: monitoring of the most important services and an alert that actually reaches the on-call engineer. The tool points this out as soon as the detection time exceeds 40 per cent.

Why blame-free is not politeness

The review is meant to lead to improvements. As soon as it is about people it stops doing that – and permanently. Anyone who has seen their name in a report will report later next time, choose their words more carefully and keep their own observation to themselves. And then exactly the knowledge you need is missing.

So the tool works with roles and systems instead of names (“service desk”, “on-call”, “monitoring”) and flags wording that points at a person. This is not language policing – it is the precondition for the next review producing anything at all.

The underrated third kind of lesson

Lessons come in three kinds: this helped, not like that again – and we got away with it, but only by luck.

The third is forgotten almost everywhere and is often the most valuable. “It was not month end.” “The backup happened to be from yesterday.” “The colleague who knows that system was there.” Sentences like these describe near misses. They say more about the risk than the damage does, because they show what will no longer be true next time.

Where the data comes from and where it goes

An incident with its history can be taken from the ITSM tool; “occurred” and “noticed” you add in the meeting, because a ticket system does not know them.

If you want to go deeper, the tool hands the incident over to the five whys. And the finished review with its measures becomes an entry in the knowledge base – so that at the next similar case somebody finds what was learned here.

The incident timeline is free of charge and available as free software under the GPL: open the incident timeline. If you would like to arrive at a review culture in which people speak openly after an outage, I am happy to help – get in touch for a free initial consultation.