A Runbook Is Not a Document, and Other Confusions

Open the average runbook and you will find a diagram of the system, a paragraph of history, and a list of links. Useful as orientation. Useless at 3am, which is the only time it gets read.

A runbook has one job: take a person from symptom to mitigation with the least possible thinking.

The test: could a tired stranger follow it?

Not a new hire. Not someone who was here when it was written. A tired stranger, paged at 3am, who has never seen this service. If the runbook requires them to already know the system, it is documentation wearing a runbook's coat.

Concretely, a real runbook for an alert contains:

1. What this alert means — one sentence, in terms of user impact.
2. How to confirm — the exact commands or dashboards, with expected output.
3. Mitigation steps — ordered, each with the command to run and how to verify it worked.
4. Escalation — who owns this if mitigation fails, and how to reach them.
5. What NOT to do — the tempting action that makes it worse.

ALERT: api-5xx-rate > 5% for 10m
MEANING: users are getting errors on the API.
CONFIRM: curl -s https://status.internal/api | jq .error_rate
MITIGATE 1: check recent deploys -> kubectl rollout history deploy/api
           if a deploy lands in the window: kubectl rollout undo deploy/api
           VERIFY: error_rate back under 1% within 5m
ESCALATE: #oncall-api, then page the service owner
DO NOT: restart all pods (see health-checks-that-lie; you will cascade)

Runbooks decay, so treat them like code

A runbook that has not been executed since it was written is probably wrong. The commands drift, the dashboards get renamed, the mitigation stops working after an architecture change.

• Put runbooks in the repository, next to the service, so PRs that change behaviour update them.
• Reference them from the alert itself; an alert with no runbook link is an invitation to guess.
• Update the runbook in the incident retro, while the truth is fresh.

A runbook is an interface between your past self and your worst morning. Design it for that reader.

The measure of a runbook is not its length or its diagrams. It is whether the person holding the pager needed to be clever. If they did, the runbook owes them a revision.