Dataflow and Dataproc: Batch and Streaming Processing

GCP Basics · Data on GCP – BigQuery, Pub/Sub, Dataflow and a First Pipeline · Lesson 4 of 5

Dataflow runs Apache Beam pipelines for batch and streaming with autoscaling and no cluster to manage; Dataproc runs Spark and Hadoop clusters or serverless batches when you already have Spark code.

When SQL in BigQuery is not enough (complex transforms, streaming enrichment, existing Spark jobs), use a processing engine.

flowchart LR
  PS["Pub/Sub topic clicks"] --> DF["Dataflow streaming job<br/>Apache Beam"]
  DF -- "windows of 1 min" --> BQ["BigQuery table clicks_per_minute"]
  GCS["gs://dl-raw/logs/*.json"] --> DP["Dataproc Serverless<br/>PySpark batch"]
  DP --> PQ["Parquet in gs://dl-curated"]
  PQ --> BQ
  %% step PS: Click events stream in through Pub/Sub.
  %% step DF: A Dataflow job built with Apache Beam parses, enriches and aggregates them; workers autoscale with the load.
  %% step BQ: One-minute windows of aggregated clicks are written to BigQuery for dashboards.
  %% step GCS: Raw log files arrive daily in Cloud Storage.
  %% step DP: An existing PySpark job runs on Dataproc Serverless without creating a cluster.
  %% step PQ: Clean Parquet files are written to a curated bucket and loaded or queried in BigQuery.

Dataflow

Fully managed runner for Apache Beam (Java, Python, Go); the same pipeline code runs batch and streaming.
Autoscaling workers, dynamic work rebalancing, exactly-once processing in streaming.
Streaming concepts: windows (fixed, sliding, session), watermarks and triggers for late data.
Templates (Google-provided or your own Flex Templates) let you start common pipelines without code: Pub/Sub → BigQuery, Cloud Storage text → BigQuery, JDBC → BigQuery.
Runs workers in your VPC; give them a dedicated worker service account and private IPs.

gcloud dataflow jobs run clicks-to-bq --region=asia-southeast1   --gcs-location=gs://dataflow-templates-asia-southeast1/latest/PubSub_Subscription_to_BigQuery   --parameters=inputSubscription=projects/dl-data-prod/subscriptions/clicks-df,outputTableSpec=dl-data-prod:web.clicks

Dataproc

Managed Spark, Hadoop, Hive, Flink, Presto clusters that start in about 90 seconds.
Use ephemeral clusters (create, run job, delete) or autoscaling clusters; store data in Cloud Storage, not HDFS, so clusters can be deleted.
Dataproc Serverless runs Spark batches and interactive sessions with no cluster at all.
Good for lift-and-shift of existing Spark/Hadoop jobs and Spark-based ML.

gcloud dataproc batches submit pyspark clean_logs.py --region=asia-southeast1   --deps-bucket=gs://dl-dataproc-deps -- --input=gs://dl-raw/logs/ --output=gs://dl-curated/logs/

Choosing

Situation	Choose
New pipeline, streaming or both, minimal operations	Dataflow
Existing Spark/Hadoop code or Spark skills	Dataproc (Serverless if possible)
Transformations expressible in SQL on data already in BigQuery	BigQuery SQL (scheduled queries or Dataform)
Kafka-based streaming already in place	Managed Service for Apache Kafka + Dataflow or Spark

Command	Meaning
`gcloud dataflow jobs list --region=… --status=active`	Running pipelines
`gcloud dataflow jobs drain JOB_ID --region=…`	Stop a streaming job after in-flight data
`gcloud dataproc clusters create spark-tmp --region=… --single-node`	Small test cluster

Traps: cancelling a streaming job drops in-flight data, draining finishes it; keeping data in HDFS ties you to a cluster; a long-running Dataproc cluster nobody deletes is a classic cost leak.