Runbooks: Living Documents That Run

A runbook in most organisations is a wiki page nobody trusts. It was correct the day it was written, partially wrong six months later, and dangerously misleading by the time an incident actually invokes it. Runbooks here are different because the scripted steps are the same scripts the platform runs every day; they cannot rot in isolation.

Anatomy of a runbook

A runbook is a sequence of steps. Each step is one of three things:

• A narrative block — context, decision points, escalation criteria.
• A parameter prompt — collect inputs the next step needs.
• A script invocation — run a known script with the collected parameters and capture the result.

Why steps are a graph, not a list

Real incidents branch. A BGP session is down — is the peer reachable? If yes, do this; if no, do that. Linear runbooks force the author to flatten the decision tree into prose nobody reads under stress. Runbook steps can declare conditions on previous step output, so the engine renders the right next step instead of asking the human to remember.

Audit and replay

Every runbook execution is a record: who ran it, which branch was taken, what each script produced, and how long each step took. After-action reviews can replay the execution exactly. Slow steps become obvious; flaky scripts get rewritten.

Generation

Runbooks can be generated from a prompt the same way scripts can. The generator scaffolds the narrative, parameter prompts, and script slots, then asks you to fill in the right scripts from your library. The first version is rarely perfect; the iteration is much faster than starting from a blank page.

Authoring tips

• Write the first step as if the engineer reading it is half-asleep and on a phone, because they often are.
• Put escalation criteria above the heroic-fix steps, not below them. People skim downward.
• Prefer many small runbooks over one large one. A runbook that handles every possible failure handles none of them well.

What to try first

Take the most-paged alert in your environment and write its runbook. Run it the next time the alert fires, even if you already know the fix. The runbook only earns trust by being used.