Runbooks¶
Procedures for planned work — the things you do deliberately, on a schedule or in response to a decision, as opposed to troubleshooting something already broken.
Runbook format¶
Every runbook follows the same shape, so they are safe to hand to someone who has not run it before:
| Section | Purpose |
|---|---|
| When to use this | The trigger — what makes this the right procedure |
| Impact | What is affected, and whether there is downtime |
| Prerequisites | Access, approvals and checks required before starting |
| Procedure | Numbered steps, each with its expected result |
| Verification | How to confirm success |
| Rollback | How to undo it if verification fails |
Planned runbooks¶
| Runbook | Covers |
|---|---|
| Rotate credentials | Service account tokens, provider keys, database passwords |
| Restore from backup | Velero restore of resources and volumes |
| Scale a workload | Adding replicas, worker pool capacity, quota changes |
| Upgrade a cluster | Coordinating a managed upgrade with your workloads |
| Drain a node | Safely moving workloads for maintenance |
| Fail over a database | Promoting a replica, and coming back |
Prerequisites for any runbook¶
Confirm which cluster and project you are connected to before running any procedure below. Running a runbook against the wrong context is the most common way one goes wrong.