The component that most determines whether a RAG system works well isn't the language model, it's the retrieval engine. Azure AI Search is the piece Microsoft designed specifically for that job.
Azure AI Search runs a traditional lexical search (BM25, good for exact matches on terms, codes, or proper names) and a vector search over embeddings (good for semantic similarity) simultaneously, and fuses both rankings with Reciprocal Rank Fusion. In practice, purely vector search fails on queries with exact terms like policy numbers or product codes; hybrid search is the recommended default configuration for enterprise RAG, not an advanced option.
On top of the initial hybrid search results, the semantic ranker applies a Microsoft-trained re-ranking model that reorders by true semantic relevance to the query, and can generate captions and extractive answers directly. This noticeably improves precision in the top 3-5 results, which are typically what gets injected as context to the LLM, and reduces the risk of a marginally relevant document with a good lexical score contaminating the prompt.
Integrated vectorization lets the indexer itself call an embedding model (from Azure OpenAI or another connected provider) during ingestion and at query time, without the team having to maintain an external service to generate vectors. This substantially simplifies the architecture: previously, a separate pipeline (an Azure Function or similar) was needed just to vectorize new content before indexing it.
An indexer connects a data source (Blob Storage, Azure SQL, Cosmos DB) to the index, and a skillset applies a chain of cognitive skills to each document during ingestion: OCR for scanned PDFs, entity extraction, language detection, and custom skills via Azure Functions for your own business logic. All of this runs automatically on every scheduled indexer run, with no manual intervention.
A critical problem in enterprise RAG is preventing a user from receiving, via the LLM, information from a document they wouldn't have direct access to. Azure AI Search solves this with document-level security filters (using Microsoft Entra ID group claims as filterable metadata) or by integrating native permissions from the data source, so the filter applies at query time, before the document ever reaches the model's context.
A RAG system's quality depends directly on how a document is chunked before vectorizing. Azure AI Search supports fixed-size chunking with overlap, and document-structure-based chunking (by section, by page) via layout skills. Legal documents and contracts, for example, respond better to chunking by clause or section than to fixed-size cuts, because it preserves each fragment's full semantic context.
Microsoft's reference pattern combines Azure AI Search for retrieval, Azure OpenAI for generation, and optionally AI Foundry Agent Service or Semantic Kernel as an orchestration layer when the answer requires multiple search steps or combining results from more than one index. This pattern is available as a reference accelerator (RAG accelerator) in Microsoft's official repository, which saves weeks of initial architecture work.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel