GCP Basics · Module 09

Data on GCP – BigQuery, Pub/Sub, Dataflow and a First Pipeline

Store and query data in BigQuery and keep its cost under control with partitioning and clustering, decouple services with Pub/Sub, process batch and streaming data with Dataflow and Dataproc, and build a simple pipeline from Cloud Storage to a Looker Studio dashboard.

Practice interactively →

What you will learn

• Create BigQuery datasets and tables, load data and query it with the bq CLI
• Lower BigQuery cost with partitioning, clustering, dry runs and the right pricing model
• Publish and consume messages with Pub/Sub topics and subscriptions
• Choose between Dataflow and Dataproc for batch and streaming processing
• Build a simple pipeline from Cloud Storage to BigQuery to a dashboard

Lessons

1. BigQuery: Datasets, Tables, Loading and Queries

   BigQuery is a serverless data warehouse: data lives in tables inside datasets, you query it with standard SQL, and you pay for storage plus either bytes scanned (on-demand) or slot capacity (editions).

2. BigQuery Cost and Performance: Partitioning, Clustering, Slots

   Partition tables by date so queries read only the days they need, cluster by frequent filter columns, check bytes with dry runs, cap them with maximum bytes billed, and choose on-demand or editions with slot reservations.

3. Pub/Sub: Topics, Subscriptions and Delivery

   Pub/Sub decouples producers from consumers: publishers send messages to a topic, each subscription gets every message, and subscribers acknowledge them; dead-letter topics, ordering keys and retention handle the hard cases.

4. Dataflow and Dataproc: Batch and Streaming Processing

   Dataflow runs Apache Beam pipelines for batch and streaming with autoscaling and no cluster to manage; Dataproc runs Spark and Hadoop clusters or serverless batches when you already have Spark code.

5. A Simple Data Pipeline: Cloud Storage → BigQuery → Dashboard

   Land raw files in Cloud Storage, load or query them in BigQuery, transform with scheduled SQL or Dataform, orchestrate with Workflows or Cloud Composer, and publish a Looker Studio dashboard.