Service Discovery and Service Mesh

System Design for DevOps · Communication and Asynchronous Processing · Lesson 4 of 4

Services find each other through DNS-based discovery; a service mesh adds a sidecar or node proxy to every workload to handle mTLS, retries, timeouts, traffic splitting and telemetry without changing application code.

With dozens of services and pods that come and go, nobody can hard-code IP addresses. Service discovery answers "where is orders right now?"; a service mesh manages how the call happens.

flowchart LR
  A["checkout pod"] --> PA["proxy (sidecar)"]
  PA -->|"mTLS, retries, timeout"| PB["proxy (sidecar)"]
  PB --> B["orders pod"]
  CP["Mesh control plane"] -.->|"config, certificates"| PA
  CP -.-> PB
  PA -.->|"metrics, traces"| OBS["Observability"]
  %% step A: The app calls http://orders as if it were local.
  %% step PA: A proxy beside the app intercepts the call.
  %% step PB: Proxies encrypt with mutual TLS and apply retries, timeouts and traffic rules.
  %% step CP: The control plane pushes routing rules and rotates certificates.
  %% step OBS: Every call produces metrics and traces, with no code change in the apps.

Service discovery

Mechanism	How
Kubernetes Service + DNS	`orders.shop.svc.cluster.local` → ClusterIP → ready pod endpoints
Cloud	AWS Cloud Map, ECS Service Connect, internal load balancers
Registry	Consul, Eureka: services register and send heartbeats

Only ready pods (passing readiness probes) appear in endpoints — discovery and health go together.

What a service mesh adds

Feature	Benefit
mTLS everywhere	Encrypted, authenticated service-to-service traffic; zero-trust policies ("only checkout may call payments")
Retries, timeouts, circuit breaking	Resilience without coding it in every language
Traffic splitting	Canary 5% to v2, header-based routing, mirroring
Telemetry	Golden metrics and traces per service pair

Examples: Istio (sidecar or ambient mode), Linkerd, Consul, AWS App Mesh successors such as ECS Service Connect / VPC Lattice.

Costs

An extra hop per call (latency in the low milliseconds), CPU and memory for proxies, a control plane to upgrade, and new failure modes. Many teams start with Kubernetes Services plus a few libraries and add a mesh when mTLS and traffic control across many services justify it.

Mesh vs API gateway

The gateway handles north–south traffic (clients → system): auth, rate limits, public API. The mesh handles east–west traffic (service → service). Most platforms use both.

Traps: retries in both the app and the mesh (retry storms); a mesh for three services; forgetting readiness probes, so discovery sends traffic to pods that are still starting.