Writing a Runbook Your On-Call Engineer Will Actually Use at 3am
A runbook that reads well in a design review often fails completely during a real incident. The reason is simple: it was written by someone who understood the system fully, for a reader who will not. At 3am, half-asleep, under pressure, with an alert still firing — that reader needs something closer to a checklist than a document.
Write for the worst version of the reader
When writing a runbook, picture the person reading it at their worst: exhausted, stressed, and unfamiliar with whatever changed in the system since the runbook was last updated. That's not a pessimistic assumption — it's the default condition during a real incident. Google's own Site Reliability Engineering Workbook makes the same point about incident documentation generally: it has to hold up for whoever's actually holding the pager, not for the person who wrote it while calm. A runbook that only works for someone calm and fully caffeinated is a runbook that works for nobody when it actually matters.
Picturing that exhausted, time-pressured reader changes what "good" looks like in a runbook. Prose explanations of why a system is designed a certain way belong in architecture docs, not runbooks. A runbook needs numbered steps, exact commands, and explicit expected output — not context that a tired reader has to parse and interpret under pressure.
What actually belongs in a runbook
- A one-line description of the symptom that tells the reader they're in the right runbook before they start following it.
- Exact commands, copy-pasteable, not paraphrased. "Restart the service" is not a step. "Run
systemctl restart api-workeron the affected host" is a step. - Expected output after each step, so the reader knows whether it worked before moving to the next one.
- A clear escalation point — the exact condition under which to stop following the runbook and escalate instead of continuing to try things.
- Rollback instructions, if the fix involves a change that might need undoing.
- A definition of "done." The specific signal that confirms the incident is actually resolved, not just quiet for the moment — so nobody closes it out prematurely and the same page fires again an hour later.
What to leave out
Background on why the system is architected this way. Alternative approaches that were considered and rejected. Anything that's true but not immediately actionable. All of it belongs somewhere — just not in the document someone is trying to follow while an alert is actively firing. If a runbook needs context to make sense of a step, that's usually a sign the step itself needs to be more explicit, not a sign the runbook needs more prose. A runbook that also tries to function as a design doc ends up serving neither job well: too sparse to explain the system, too padded to follow at 3am.
Test it before an incident forces you to
The only reliable way to know if a runbook actually works is to hand it to someone who didn't write it and watch them follow it literally, step by step, without filling in gaps from memory. The person who wrote the runbook already understands the system, which is exactly why they're the wrong person to judge whether the document is clear on its own. A newer team member, or someone borrowed from an adjacent team, will usually hit the first ambiguous step within a few lines — and that's the point of the exercise, not a failure of it.
Run it as a timed exercise where possible, not just a read-through. If the runbook takes fifteen calm minutes in a test but would obviously take forty in a real incident because of a missing command or an unclear branch point, that's a defect in the document, not a fact about the reader. Treat every point of confusion as a bug to fix, the same way you'd treat a bug in the system the runbook exists to help recover.
Why do runbooks go stale so fast?
Because nobody ties runbook updates to the same process that changes the system underneath them — the document just sits there while the infrastructure moves on. A runbook written for a system as it existed six months ago is a liability, not an asset — it sends someone confidently down a path that no longer applies. The commands might reference a service that's since been renamed. The escalation contact might have left the team. This is the most common way runbooks fail: not because they were badly written, but because nobody updated them after the system changed underneath them.
The fix isn't a quarterly audit calendar reminder that gets ignored. It's tying runbook updates to the same process as the change itself — if a deploy changes how a service restarts, updating the relevant runbook is part of that deploy, not a follow-up task that gets deprioritized indefinitely.
Link the runbook to the alert itself
The gap between "an alert fires" and "someone finds the right runbook" is where a lot of response time disappears, especially for someone newer to the rotation who doesn't already know where things live. PagerDuty's own public incident response documentation treats this link between an alert and its runbook as a baseline part of a usable response process, not an optional nicety. If your paging tool supports it, link directly from the alert to the relevant runbook — PingParrot's webhook payloads carry through custom fields from the originating alert, so a monitoring rule can pass a runbook URL straight into the page itself, putting the right document one tap away instead of a search away. A monitoring rule watching a specific, known-flaky failure — a disk-full alert on a particular host, say — can carry that exact runbook's URL in every page it generates, so the link shows up automatically instead of depending on someone remembering to add it.
A runbook nobody can find fast enough might as well not exist. The best-written runbook in the world doesn't help if the person on-call spends the first five minutes of the incident looking for it instead of following it.