Elasticity and Autoscaling

Elasticity is scaling out and back in automatically with demand; autoscalers such as the Kubernetes HPA, cluster autoscaler and cloud Auto Scaling groups react to metrics like CPU, requests per target or queue depth.

Elasticity means capacity follows demand in both directions — up for the evening peak, down at night — without anyone clicking buttons.

flowchart LR
  M["Metrics: CPU, RPS, queue depth"] --> HPA["Pod autoscaler"]
  HPA -->|"more pods"| D["Deployment"]
  D -->|"pods pending"| CA["Cluster autoscaler / Karpenter"]
  CA -->|"more nodes"| N["Nodes"]
  M -.-> ASG["Cloud Auto Scaling group"]
  %% step M: Autoscaling starts with a metric that rises with load.
  %% step HPA: The Horizontal Pod Autoscaler changes the replica count to reach a target value.
  %% step D: New pods need room; if none fits, they stay Pending.
  %% step CA: The cluster autoscaler adds nodes for pending pods and removes empty ones.
  %% step ASG: Outside Kubernetes, Auto Scaling groups or scale sets do the same for VMs.

Layers of autoscaling

Layer	Tool	Scales
Pods	HPA (CPU, memory, custom metrics), KEDA (queues, events)	Replicas
Pod size	VPA	Requests/limits
Nodes	Cluster autoscaler, Karpenter	Node count and type
VMs	AWS Auto Scaling groups, Azure VM scale sets, GCP managed instance groups	Instances
Serverless	Lambda, Cloud Run, Fargate	Per request or per task

Choosing the metric

Workload	Good metric
CPU-bound API	Average CPU (target ~60–70%)
I/O-bound API	Requests per second per pod, or latency
Queue workers	Queue depth or age of oldest message
Batch	Pending jobs

Policies

Target tracking: "keep CPU at 65%" — simplest and usually best.
Step scaling: add N instances when a threshold is crossed.
Scheduled scaling: warm up before a known event (sale at 20:00).
Cooldown / stabilisation window: avoid flapping up and down.
Min/max: min ≥ 2 for availability, max to cap cost and protect the database.

Speed matters

Scaling takes time: a new node can need 1–3 minutes, an app may need 30 s to warm up. Keep headroom, use readiness probes so pods get traffic only when ready, and pre-scale for predictable peaks.

Traps: scaling on CPU for an I/O-bound service; no max replicas, so autoscaling overwhelms the database; min of 1; pods without resource requests (HPA cannot compute CPU percentage).