Pub/Sub: Topics, Subscriptions and Delivery

GCP Basics · Data on GCP – BigQuery, Pub/Sub, Dataflow and a First Pipeline · Lesson 3 of 5

Pub/Sub decouples producers from consumers: publishers send messages to a topic, each subscription gets every message, and subscribers acknowledge them; dead-letter topics, ordering keys and retention handle the hard cases.

Pub/Sub is a global, serverless messaging service. Publishers and subscribers never talk directly, so each side can scale, fail and deploy independently.

flowchart LR
  API["Order API"] -- "publish order.created" --> T["Topic orders"]
  T --> S1["Subscription orders-email<br/>push → Cloud Run"]
  T --> S2["Subscription orders-bq<br/>BigQuery subscription"]
  T --> S3["Subscription orders-worker<br/>pull"]
  S3 -- "after 5 failed attempts" --> DLQ["Dead-letter topic"]
  %% step API: The order API publishes one message per new order and returns to the user immediately.
  %% step T: The topic stores the message and fans it out; publishers do not know who consumes it.
  %% step S1: A push subscription sends each message to the email service’s HTTPS endpoint.
  %% step S2: A BigQuery subscription writes messages straight into a table with no code.
  %% step S3: A pull subscription lets workers fetch messages at their own pace and acknowledge them.
  %% step DLQ: Messages that keep failing go to a dead-letter topic instead of blocking the queue.

Core ideas

Term	Meaning
Topic	Named channel that receives messages
Subscription	A queue attached to one topic; every subscription gets every message
Ack	The subscriber confirms it processed a message; unacked messages are redelivered after the ack deadline (10–600 s)
Retention	Unacked messages are kept 7 days by default (configurable); topics can also retain messages for replay

Subscription types

Type	Use
Pull	Workers fetch messages (StreamingPull); best throughput and flow control
Push	Pub/Sub POSTs to an HTTPS endpoint (Cloud Run, functions) with an OIDC token
BigQuery	Writes directly into a BigQuery table
Cloud Storage	Writes batches of messages into files

Delivery guarantees

Default is at-least-once: handlers must be idempotent (use a message ID or business key).
Exactly-once delivery is available for pull subscriptions in a region.
Ordering keys keep messages with the same key in order (for example all events of one order).
Dead-letter topics and retry policies (exponential backoff) handle poison messages.
Filters let a subscription receive only messages with matching attributes.
Seek replays messages from a snapshot or timestamp.

gcloud pubsub topics create orders
gcloud pubsub subscriptions create orders-worker --topic=orders --ack-deadline=60   --dead-letter-topic=orders-dlq --max-delivery-attempts=5
gcloud pubsub topics publish orders --message='{"order_id": 1042}' --attribute=type=created
gcloud pubsub subscriptions pull orders-worker --auto-ack --limit=5

Pub/Sub or something else?

Need	Use
Fan-out events to many independent consumers	Pub/Sub
Run a task later with rate limits and per-task deduplication	Cloud Tasks
Kafka protocol compatibility	Managed Service for Apache Kafka
React to Google Cloud events	Eventarc (often uses Pub/Sub underneath)

Traps: a subscription only receives messages published after it was created; slow handlers beyond the ack deadline cause duplicates; the dead-letter feature needs the Pub/Sub service agent to have publisher rights on the dead-letter topic and subscriber rights on the source subscription.