Message Queues and Event Streams

System Design for DevOps · Communication and Asynchronous Processing · Lesson 1 of 4

Message queues and streams let producers hand off work without waiting for consumers; consumers pull at their own pace, so a slow or crashed consumer no longer breaks the producer. Queues distribute tasks, streams keep an ordered, replayable log.

In a synchronous design, service A calls B and waits. If B is slow, A is slow; if B is down, A fails. A message queue puts a buffer in between.

flowchart LR
  P1["Producer: web"] --> Q["Queue / stream"]
  P2["Producer: mobile API"] --> Q
  Q -->|"pull when ready"| C1["Consumer 1"]
  Q -->|"pull when ready"| C2["Consumer 2"]
  C1 -.->|"failed 5×"| DLQ["Dead-letter queue"]
  %% step P1: Producers publish a message and return immediately.
  %% step Q: The queue stores messages durably until a consumer takes them.
  %% step C1: Consumers pull messages at their own pace and acknowledge when done.
  %% step DLQ: Messages that keep failing go to a dead-letter queue for inspection, so they do not block the rest.

The push-overload problem

A classic design question: producers push tasks straight to consumers. Under load, consumers receive more than they can handle and hang or crash — and producers fail with them.

Fix: let consumers pull only when ready, through a queue or streaming platform (SQS, RabbitMQ, Kafka, Kinesis). The queue absorbs the spike; the backlog shrinks when load drops or when you add consumers.

Queue vs stream

	Message queue (SQS, RabbitMQ)	Event stream / log (Kafka, Kinesis)
Model	Each message handled by one consumer, then deleted	Ordered log kept for days; many consumer groups read independently
Ordering	Best effort (FIFO queues available)	Ordered per partition
Replay	No	Yes — rewind the offset
Scale consumers	Add workers	Up to one consumer per partition per group
Typical use	Background jobs: resize image, send email	Event-driven systems, analytics, change data capture

Delivery guarantees

At-most-once: may lose messages.
At-least-once: may deliver twice (most common) → consumers must be idempotent.
Exactly-once: possible in limited scopes (Kafka transactions); design for at-least-once.

Key settings

Setting	Why
Visibility timeout (SQS)	Longer than the processing time, or another worker picks the message up too
Dead-letter queue + max receives	Poison messages do not loop forever
Partitions (Kafka)	Upper bound on parallel consumers; choose the key for ordering (e.g. order_id)
Retention	How long you can replay or recover

Pub/sub

A topic (SNS, Kafka topic, Google Pub/Sub) fans one event out to many subscribers: order-placed → billing, email, analytics, each with its own queue. Producers do not need to know who listens — loose coupling.

Traps: consumers that are not idempotent; visibility timeout shorter than the job; no dead-letter queue; one Kafka partition limiting you to one consumer; using a queue where the caller really needs an immediate answer.