DevOps Peach

System Design for DevOps · Reliability, Deployments and Migrations · Lesson 1 of 5

Fault Tolerance, High Availability and Disaster Recovery

High availability removes single points of failure so a component failure causes no outage; disaster recovery restores service after a large event such as a region loss. RPO and RTO set how much data and time you may lose, and drive the DR strategy.

Practice interactively →

These words are often mixed up. They answer different questions.

flowchart LR
  FT["Fault tolerance: keep working through a failure"] --> HA["High availability: no SPOF in one region"]
  HA --> DR["Disaster recovery: restore after region or data loss"]
  DR --> RPO["RPO: how much data may be lost"]
  DR --> RTO["RTO: how long until service is back"]
  %% step FT: Fault tolerance means the system withstands a component failure and heals itself.
  %% step HA: High availability removes single points of failure, usually across zones in one region.
  %% step DR: Disaster recovery picks up when HA is not enough: region outage, corruption, ransomware, human error.
  %% step RPO: Recovery Point Objective — the maximum acceptable data loss, measured in time.
  %% step RTO: Recovery Time Objective — the maximum acceptable time to restore service.

Definitions

Term Meaning Example
Fault tolerance Withstand a failure and keep working, return to normal capacity A pod crashes; Kubernetes restarts it, others serve meanwhile
High availability Eliminate SPOFs so the service stays reachable Multi-AZ database, app in three zones
Disaster recovery Process to restore critical systems and data after a disruptive event Restore from backups into another region

"HA first, DR when HA fails." HA does not protect against a bad migration or DELETE without WHERE — replicas copy the mistake instantly. Only backups and point-in-time recovery do.

DR strategies (cheapest → fastest)

Strategy Standby in the DR region RTO RPO
Backup & restore Backups only Hours–days Hours (last backup)
Pilot light Data replicated, core services off or minimal Tens of minutes–hours Minutes
Warm standby Scaled-down full copy running Minutes Seconds–minutes
Active-active (multi-site) Full capacity serving traffic Near zero Near zero

Lower RPO/RTO costs more. Set them per system with the business: checkout may need RTO 15 min, the reporting tool can wait a day.

A DR plan contains

  1. Scope, RPO/RTO per system, owners and contacts.
  2. Backups: what, how often, where (another account and region), immutable copies (ransomware).
  3. Step-by-step runbook: restore data, deploy infrastructure (IaC), switch DNS.
  4. Regular tests — a backup you never restored is a hope, not a backup.

Useful building blocks

Database point-in-time recovery, snapshots copied cross-region, S3 Cross-Region Replication and Object Lock, AWS Backup / Azure Backup policies, Terraform to rebuild infrastructure, Route 53 health-check failover.

Traps: calling Multi-AZ "DR"; backups in the same account as production; never testing a restore; RTO promised without the infrastructure code to rebuild the region.

Practice interactively →