Everyone wants a one-button rollback. Very few teams have one, and the reason is rarely the button.
A rollback is not a single action. It is a set of changes unwound in reverse, and the set almost always contains something that does not unwind cleanly: a schema migration, a feature-flag dependency, a queue format, a cached asset. The deployment tooling is the easy ten percent.
The three things that make a rollback slow
- State that moved forward. Code rolls back in seconds. Data does not.
- Coupled releases. If service B now expects service A's new payload, rolling back A breaks B.
- Manual verification. If "is it fixed?" requires a human staring at a dashboard, your rollback time is bounded by how fast that human wakes up.
Design for the unwind, not the rollout
The trick is to make every release reversible by construction. That means shipping changes in an order where each step is independently safe to remove.
A deployment you cannot reverse is not a deployment, it is a migration with extra steps.
The pattern that works is expand-contract:
- Expand: add the new capability alongside the old one. Both code paths run.
- Migrate: move traffic and data to the new path, gradually, behind a flag.
- Contract: delete the old path in a later release, once nothing depends on it.
At any point during expand or migrate, rolling back is just "stop sending traffic to the new path". Nothing has been destroyed yet.
Schema changes are the real boss fight
The deploy that hurts is the one carrying ALTER TABLE. A backwards-incompatible migration turns your rollback into a restore, and a restore is an incident, not a button.
The discipline is boring and non-negotiable:
| Change | Safe to roll back? | Why |
|---|---|---|
| Add nullable column | Yes | Old code ignores it |
| Add column with default | Usually | Check read paths first |
| Rename column | No | Old code reads the old name |
| Drop column | No | Data is gone |
| Change column type | No | Values may not convert back |
For renames and drops, use the expand-contract cycle over two or three releases instead of one heroic migration. The Kubernetes documentation makes the same point about rollout strategy: the orchestration is fine, it is your state that bites.
Make the button honest
Measure rollback as a number, not a vibe. Time from "decision to roll back" to "error rate back to baseline". Put it on the same dashboard as deploy frequency. Teams that track it start designing for it, and that — not any tool — is where zero-downtime actually comes from.
If your rollback requires a person to remember the order of operations, you do not have a rollback. You have a ritual.
Conversation
3 comments