Planning a Migration: Rehearsal, Cut-over, Rollback

System Design for DevOps · Reliability, Deployments and Migrations · Lesson 4 of 5

A migration moves a system to new infrastructure or a new database; good plans have a rehearsal, data sync, a defined cut-over (often by DNS), clear go/no-go criteria, and both roll-back and roll-forward paths.

Migrations fail less on technology than on planning: unknown dependencies, no rehearsal, no way back.

flowchart LR
  INV["Inventory and dependencies"] --> SYNC["Sync data: replication / CDC"]
  SYNC --> REH["Dress rehearsal"]
  REH --> GO{"Go / no-go"}
  GO -->|"go"| CUT["Cut-over: DNS weights 10 → 50 → 100%"]
  CUT --> WATCH["Watch SLOs"]
  WATCH -->|"problem"| RB["Roll back to old"]
  WATCH -->|"small issue"| RF["Roll forward with a fix"]
  %% step INV: List every component, client, cron job, hard-coded IP and integration.
  %% step SYNC: Keep the new database in sync with the old one (replication or change data capture).
  %% step REH: Rehearse the whole cut-over on a copy and time every step.
  %% step GO: Decide with clear criteria: data in sync, tests green, people on call.
  %% step CUT: Move traffic gradually, often with weighted DNS or the load balancer.
  %% step RB: Rolling back must be possible until the old system is retired.
  %% step RF: Sometimes fixing forward is faster and safer than going back.

The checklist

Step	Questions
Inventory	What talks to the system? Batch jobs, partners, IP allow-lists, certificates, DNS records
Data	How big? How do we sync and verify (row counts, checksums)? What is the write freeze window, if any?
Rehearsal (dress rehearsal)	Did we run the full plan on a copy? How long did each step take?
Cut-over	Big bang (one switch in a maintenance window) or gradual (DNS weights, per tenant)?
Go/no-go criteria	Data lag < 5 s, smoke tests pass, error budget available, rollback tested
Roll-back	How do we return? What happens to data written to the new system meanwhile?
Roll-forward	When is fixing on the new side better than going back?
Communication	Status page, stakeholders, who decides
Decommission	When is the old system switched off and its data deleted?

DNS redirection

Lower the TTL (e.g. from 3600 to 60 s) a day before, so changes spread fast. Then shift weighted records 10% → 50% → 100%. Remember some clients cache DNS longer than the TTL — keep the old system running for a while.

Database migration patterns

Replication / CDC (AWS DMS, Debezium, native logical replication) keeps the target in sync until cut-over.
Dual writes are risky without careful idempotency — prefer one writer plus replication.
Reverse replication after cut-over keeps the old database usable for rollback.

Roll-back vs roll-forward

Roll back when the new system is fundamentally broken and the old one still has current data. Roll forward when the issue is small and well understood, or when rolling back would lose new writes. Decide the rules before the night of the migration.

Traps: forgetting to lower the DNS TTL in advance; no rehearsal; a rollback plan that ignores writes made after cut-over; turning off the old system the same day.