When RAG needs to scale to hundreds of millions of vectors with millisecond latency, the vector database stops being an implementation detail and becomes the critical piece of the architecture.
Vertex AI Vector Search (the direct successor to what Google called Matching Engine) is Google Cloud's managed vector similarity search service. It's built on ScaNN (Scalable Nearest Neighbors), the approximate nearest-neighbor search algorithm developed internally by Google Research -- the same algorithmic lineage that powers search at scale in products like Google Search and YouTube.
The core value proposition is serving similarity queries over indexes of hundreds of millions or billions of vectors with single-digit-millisecond latency, something general-purpose vector databases start to struggle to sustain without considerable tuning at that volume.
The typical flow starts by generating embeddings with Vertex AI's `text-embedding-005` or `multimodalembedding` model (for mixed text-image content), then creating an index over those vectors.
Vertex AI Vector Search offers two indexing strategies. The Tree-AH index (based on ScaNN) uses quantization and tree structures to approximate nearest-neighbor search with a configurable trade-off between precision and speed, suited for large-scale production. The brute-force index computes exact distance against every vector, useful as a baseline for measuring the real recall you're sacrificing with the Tree-AH approximation, or for small catalogs where exactness matters more than latency.
Unlike an index that only rebuilds in batches, Vertex AI Vector Search supports streaming updates: inserting, updating, or deleting individual vectors without rebuilding the entire index, with changes reflected in queries within seconds. This is critical for e-commerce catalogs with changing inventory or recommendation systems with continuously published new content, where rebuilding the full index every time would be operationally infeasible.
Beyond pure vector similarity, Vector Search supports filtering by numeric and categorical metadata in the same query -- for example, "the 10 most similar products within category X and priced below Y" -- with no need to post-process results outside the index, plus namespaces with "allow/deny" restrictions for implementing document-level access control within the same shared index.
The frequent question is when to build RAG directly on Vector Search instead of using Vertex AI Search. Vector Search gives full control: you choose your chunking strategy, your embedding model, your re-ranking logic, and your filtering metadata schema. That flexibility has implementation and maintenance cost. Vertex AI Search abstracts all of that in exchange for less control. The practical rule: if the team already has experience operating search infrastructure and the use case has particular chunking or filtering requirements, go with Vector Search directly. If the priority is time-to-market, Vertex AI Search.
Vertex AI Vector Search charges for the compute nodes serving the deployed index (per hour, based on machine type) plus a cost associated with index storage, independent of query volume. This means that, unlike a pay-per-query model, cost is predictable but fixed: correctly sizing the machine type and the number of index replicas based on the actual expected query volume avoids both over-provisioning and latency bottlenecks during traffic spikes.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel