Backpressure, Batching and Throughput

System Design for DevOps · Communication and Asynchronous Processing · Lesson 2 of 4

When input grows faster than a pipeline can process it, queues fill, latency explodes and nodes fall over; backpressure, bounded queues, batching, parallel workers and autoscaling on backlog keep throughput stable.

A common report: "adding a little more input makes processing time explode." Usually one stage has hit its capacity and work is queuing in front of it.

flowchart LR
  IN["Input"] --> S1["Stage 1: parse"]
  S1 --> B{"Stage 2 at capacity?"}
  B -->|"no"| S2["Stage 2: enrich"]
  B -->|"yes"| WAIT["Queue grows, latency explodes"]
  S2 --> OUT["Output"]
  BP["Backpressure: slow the producer or shed load"] -.-> S1
  %% step IN: Input rate rises a little.
  %% step S1: Earlier stages keep pushing work forward.
  %% step B: The slowest stage is the bottleneck; once utilisation nears 100%, waiting time grows sharply.
  %% step WAIT: Unbounded buffers hide the problem until memory runs out.
  %% step BP: Backpressure tells upstream to slow down instead of drowning downstream.

Why small increases hurt so much

Queueing theory: as a stage's utilisation goes from 70% to 95%, average waiting time does not grow by 25% — it grows many times over. Other usual suspects:

An O(n²) step (nested loops, a join without an index) that was fine for small inputs.
Memory running out → swapping or garbage collection pauses.
A shared resource limit: database connections, API rate limits, disk IOPS.

Profile and measure each stage before scaling anything.

Tools for stable throughput

Technique	What it does
Bounded queues	Reject or block when full instead of growing until OOM
Backpressure	Producer slows down when consumers lag (TCP windows, reactive streams, Kafka consumers pulling)
Load shedding	Return 429/503 for low-priority work to protect the core
Batching	Process 100 records per database call instead of 1 — fewer round trips
Parallelism	Partition work and run N workers; scale on backlog (KEDA, SQS metrics)
Async I/O	Do not hold a thread while waiting for the network

Scaling to hundreds of users

Turn a single input → process → output box into: an API that accepts work and enqueues it, a queue, a pool of stateless workers that autoscale on queue depth, results in object storage or a database, and a status endpoint or notification when done. Each piece scales on its own.

Measure

Track per stage: throughput (items/s), latency percentiles, queue depth and age of the oldest item, error rate. Age of oldest message is the clearest "we are falling behind" signal.

Traps: unbounded in-memory queues; one-by-one database inserts in a loop; scaling workers until the database runs out of connections; autoscaling on CPU when the real bottleneck is I/O.