Zero-Downtime Deployments

System Design for DevOps · Reliability, Deployments and Migrations · Lesson 3 of 5

Rolling updates replace instances gradually, blue/green switches all traffic between two full environments, and canary releases send a small share of traffic to the new version first; readiness probes, draining and backward-compatible changes make all of them safe.

A classic question: "How would you update every service behind foo.com — CDN, load balancer, app servers, database — without downtime?" The answer is a set of habits, not one tool.

flowchart LR
  subgraph Rolling["Rolling"]
    R1["v1 v1 v1"] --> R2["v2 v1 v1"] --> R3["v2 v2 v2"]
  end
  subgraph BG["Blue/green"]
    LB["Router"] -->|"100%"| BLUE["blue v1"]
    LB -.->|"switch"| GREEN["green v2"]
  end
  subgraph CAN["Canary"]
    LB2["Router"] -->|"95%"| OLD["v1"]
    LB2 -->|"5%"| NEW["v2"]
  end
  %% step R1: Rolling: replace a few instances at a time while the rest serve traffic.
  %% step LB: Blue/green: run the new version beside the old one, test it, then switch all traffic at once.
  %% step GREEN: Rollback is just switching back to blue.
  %% step LB2: Canary: send a small share of real traffic to v2 and watch error rate and latency.
  %% step NEW: Increase the share step by step, or roll back automatically if metrics get worse.

Comparison

Strategy	Extra capacity	Rollback	Risk exposure
Rolling	Small (maxSurge)	Roll again to v1 (minutes)	All users gradually
Blue/green	2× during the switch	Instant switch back	All users at once, after testing
Canary	Small	Instant (shift traffic back)	Only the canary share

Requirements for any of them

Readiness probes so new instances get traffic only when ready.
Connection draining / graceful shutdown so old instances finish in-flight requests (handle SIGTERM, preStop, deregistration delay).
At least 2 instances and maxUnavailable: 0 for rolling updates.
Backward-compatible changes: v1 and v2 run at the same time, so APIs and database schemas must work for both.

Database changes: expand and contract

Expand: add the new column/table; both versions work.
Deploy code that writes both and reads the new one; backfill.
Contract: after all instances run the new code, remove the old column.

Never rename or drop a column in the same release that stops using it.

Updating the edge layers

CDN: versioned asset names; update config in a staging distribution first.
Load balancer: managed LBs update without downtime; for self-managed pairs, update one node at a time behind a floating IP or DNS.
DNS: lower TTLs before a switch.

Tools: Kubernetes Deployments (rolling), Argo Rollouts and Flagger (canary/blue-green with metric analysis), ECS/CodeDeploy, feature flags to decouple deploy from release.

Traps: replicas: 1 with rolling updates; no graceful shutdown, so each deploy drops requests; schema change that breaks the old version still running; canary without metrics to judge it.