RAG on GCP: Embeddings, Vector Search and RAG Engine

GCP Basics · AI on GCP – Gemini, RAG, BigQuery ML and Agents · Lesson 3 of 5

Retrieval-augmented generation finds relevant chunks of your documents with embeddings and vector search, then gives them to the model so answers are grounded and cite sources; Google Cloud offers managed and do-it-yourself options.

Models do not know your private documents and can make things up. Retrieval-augmented generation (RAG) fixes this: retrieve relevant passages first, then ask the model to answer only from them.

flowchart LR
  DOCS["Policies and manuals<br/>gs://dl-ai-docs"] --> CH["Chunk and embed"]
  CH --> VS["Vector store<br/>RAG Engine, Vector Search, BigQuery, AlloyDB"]
  Q["User question"] --> QE["Embed the question"]
  QE --> VS
  VS -- "top 5 chunks" --> GEM["Gemini with retrieved context"]
  GEM --> A["Answer with citations"]
  %% step DOCS: Source documents sit in Cloud Storage, Drive or a database.
  %% step CH: They are split into chunks and each chunk is turned into an embedding vector by an embedding model.
  %% step VS: Vectors and the chunk text are stored in a vector store that supports nearest-neighbour search.
  %% step Q: A user asks a question in natural language.
  %% step QE: The question is embedded with the same model.
  %% step GEM: The closest chunks are added to the prompt, with an instruction to answer only from them.
  %% step A: The answer cites its sources, and says “I don’t know” when the documents do not cover the question.

Options on Google Cloud

Option	Effort	Notes
Vertex AI Search (search and agent search)	Lowest	Managed ingestion, chunking, ranking and grounded answers over websites, Cloud Storage, BigQuery
RAG Engine	Low	Managed corpus: import files, chunking, embeddings and retrieval; plug into Gemini as a retrieval tool
Vector Search	Medium	Managed approximate nearest-neighbour index at very large scale
BigQuery vector search	Medium	`ML.GENERATE_EMBEDDING` + `VECTOR_SEARCH` on data already in BigQuery
AlloyDB / Cloud SQL with pgvector	Medium	Vectors next to your relational data
Grounding with Google Search	Lowest	For public, fresh web information

Design decisions

Chunk size and overlap: too small loses context, too large adds noise and tokens (a few hundred tokens with some overlap is a common start).
Embedding model: use one model for both documents and questions; re-embed everything if you change it.
Metadata filters: department, language, access level, date.
Access control: only retrieve chunks the user may see (filter by ACL metadata or keep separate corpora).
Freshness: re-index on document changes (event-driven with Eventarc on Cloud Storage).
Evaluation: measure retrieval quality (did the right chunk come back?) and answer quality (grounded, complete) with an evaluation set.

A grounded prompt

System: You answer questions about Example Logistics policies.
Use ONLY the context below. If the answer is not in the context, say you don't know.
Cite the document name for each fact.
Context:
[1] refund-policy.pdf: "Refunds are issued within 7 days ..."
[2] delivery-sla.pdf: "Same-day delivery in Ho Chi Minh City ..."
Question: How long do refunds take?

gcloud storage cp ./policies/*.pdf gs://dl-ai-docs/policies/
gcloud ai indexes list --region=asia-southeast1

Command	Meaning
`gcloud ai index-endpoints list --region=…`	Vector Search endpoints
`bq query --use_legacy_sql=false 'SELECT * FROM VECTOR_SEARCH(…)'`	Vector search in BigQuery

Traps: RAG without access filtering can leak documents to the wrong users; mixing embeddings from different models gives meaningless distances; retrieved text can contain prompt-injection instructions, so screen it like user input.