The best time to write a runbook is before you need one, because the second-best time — during the incident — is when you’re least equipped to write anything clearly. Here’s what actually goes in one, and how to keep it from rotting.

What a runbook is not

It’s not a design doc. It’s not “how the system works.” Nobody reads architecture prose at 3am with an alert screaming — they need the next command to type, in order, with no thinking required. If your runbook requires understanding the system to use it, it’s not a runbook, it’s documentation with a misleading name.

The structure that works under pressure

Here’s the path a good runbook takes someone down, in order:

flowchart TD
    A["Alert fires (the trigger)"] --> B["Immediate mitigation"]
    B --> C{"Did that resolve it?"}
    C -->|Yes| D["Done - note it in\nthe postmortem"]
    C -->|No| E["Diagnosis steps"]
    E --> F{"Resolved within\nthe timebox?"}
    F -->|Yes| D
    F -->|No| G["Escalate to the\nnamed on-call team"]
    G --> H["Roll back if the\nmitigation caused this"]
  

1. The trigger

State exactly what alert or symptom sends someone here. Not “when the database is slow” — that’s a diagnosis, not a trigger. Use the actual alert name:

Trigger: PagerDuty alert "db-connection-pool-exhausted"

2. The immediate mitigation, before the investigation

Most runbooks bury the “make it stop hurting” step under three paragraphs of root-cause investigation. Flip that. The first section after the trigger should be the fastest safe action that reduces user impact — restart a service, fail over, scale up a pool — even if it doesn’t fix the underlying cause.

# Immediate: bump the connection pool ceiling to buy time
kubectl set env deployment/api DB_POOL_MAX=50

3. The diagnosis steps, as copy-pasteable commands

# Check current connection count vs pool max
psql -c "SELECT count(*) FROM pg_stat_activity;"

# Find the query holding connections longest
psql -c "SELECT pid, now() - query_start AS duration, query
          FROM pg_stat_activity ORDER BY duration DESC LIMIT 5;"

Every command should be something a half-awake human can paste without editing, or with exactly one obvious substitution (a hostname, an ID).

4. The escalation path

Who do you page next if the mitigation doesn’t work, and after how long do you page them? Vague answers like “escalate if needed” are useless under pressure — put a name, a team, and a timebox: “if unresolved after 15 minutes, page #database-oncall.”

5. The rollback

If the mitigation involves a config change or deploy, the runbook needs the exact command to undo it. People forget to write this down because they’re focused on the fix, but the rollback is what prevents the mitigation itself from becoming the next incident.

Keeping it from going stale

A runbook that references a command that no longer exists is worse than no runbook — it costs someone precious minutes discovering it’s wrong mid-incident. Two habits that actually work:

1. Run it during the postmortem. After every incident that used a runbook, walk through it literally, step by step, in the retro. If a command was wrong or a step was missing, fix it right there — don’t file a ticket for “update the runbook,” because that ticket won’t get done.
2. Attach an owner and a review cadence, not a wiki page nobody watches. One person’s name at the top, reviewed every quarter along with the on-call rotation handoff, is more reliable than “the team” owning it, which means no one does.

The test

Hand the runbook to someone who’s never touched this system and ask them to follow it, cold, with you watching and saying nothing. Every place they hesitate or ask “wait, what does this mean” is a gap. Fix those gaps now — they’re exactly what will cost you five extra minutes of downtime the next time this actually pages someone at 3am.