Resiliency Patterns: Timeouts, Retries, Circuit Breakers

System Design for DevOps · Reliability, Deployments and Migrations · Lesson 2 of 5

Resiliency is the ability to absorb failures and recover; timeouts bound waiting, retries with exponential backoff and jitter handle transient errors, circuit breakers stop calling a failing dependency, and bulkheads keep one failure from consuming every resource.

In a distributed system, something is always failing a little. Resilient systems expect it: they degrade gracefully and recover on their own.

flowchart LR
  A["checkout"] -->|"timeout 800 ms"| CB{"Circuit breaker"}
  CB -->|"closed"| P["payments"]
  CB -->|"open: fail fast"| FB["Fallback: queue payment, show pending"]
  P -.->|"5xx / slow"| RT["Retry with backoff + jitter"]
  RT --> CB
  BH["Bulkhead: separate pool per dependency"] -.-> A
  %% step A: Every outgoing call has a timeout, so a slow dependency cannot hold threads forever.
  %% step CB: The circuit breaker counts failures; above a threshold it opens.
  %% step P: While closed, calls go through normally.
  %% step RT: Transient errors are retried a few times with growing, randomised delays.
  %% step FB: When open, calls fail fast and a fallback keeps the user flow alive.
  %% step BH: Bulkheads give each dependency its own pool, so one slow service cannot exhaust all connections.

The patterns

Pattern	Problem it solves	Rule of thumb
Timeout	Threads stuck waiting on a slow call	Always set one; shorter than the caller's own timeout
Retry + exponential backoff + jitter	Transient errors (network blip, 503, throttling)	2–3 attempts; delays like 100 ms, 200 ms, 400 ms ± random; only for idempotent calls
Circuit breaker	Hammering a dependency that is down	Open after e.g. 50% failures in 30 s; half-open to test recovery
Bulkhead	One slow dependency eating all threads/connections	Separate pools or concurrency limits per dependency
Fallback / graceful degradation	Hard failure for a non-critical feature	Cached recommendations, "payment pending", hide the widget
Rate limiting / load shedding	Overload	Reject early with 429/503 instead of collapsing

Retry storms

If every layer retries 3 times, one failing call at the bottom of a 3-layer chain becomes 27 calls. Retry in one layer, use a retry budget (for example at most 10% extra traffic), and always add jitter so clients do not retry in sync.

Health and self-healing

Liveness probe: restart a stuck process.
Readiness probe: stop sending traffic until the pod is ready (and during overload).
PodDisruptionBudget: keep enough replicas during maintenance.

Ways to make a system more resilient

Redundancy across zones, the patterns above, idempotent operations, queues to absorb spikes, autoscaling, chaos testing (kill pods or block a dependency on purpose and watch), and runbooks for what automation cannot fix.

Traps: no timeout on HTTP clients (defaults can be minutes or infinite); retrying non-idempotent POSTs; retries without jitter; a liveness probe that checks the database, so a DB blip restarts every pod.