A basic RAG with pure vector similarity fails in production more often than demos suggest. This guide covers the two techniques that raise precision the most: hybrid search and re-ranking.
A retrieval pipeline based solely on cosine similarity over embeddings has a well-known blind spot: it fails with exact terms — product codes, proper names, item numbers — because embeddings capture semantics, not lexical matching. A customer searching for "error E-4021" might not retrieve the right document if the embedding semantically associates it with other similar errors. The standard solution in production systems is combining lexical retrieval (BM25) with vector retrieval.
LangChain solves this with `EnsembleRetriever`, which combines multiple retrievers and fuses results using Reciprocal Rank Fusion (RRF):
The `weights` aren't universal: in technical domains with lots of exact vocabulary (logs, error codes, SKUs), raising BM25's weight to 0.5-0.6 usually improves recall. In more conversational domains, a dominant vector weight works better.
Hybrid search improves recall (bringing relevant documents into the candidate set), but not necessarily the ordering. That's what a re-ranker is for: a cross-encoder that evaluates the (query, document) pair directly, much more precise but computationally more expensive than vector search, which is why it's applied only to the top-k candidates, not the entire base.
`FlashrankRerank` is a lightweight option that runs locally with no external API call, useful when the latency or cost of a managed re-ranker (like Cohere Rerank) isn't justified for the project's volume. For higher precision, `CohereRerank` via `langchain-cohere` delivers better results on multilingual benchmarks, relevant if the content is in Spanish.
The final pipeline combines hybrid retrieval, re-ranking, and generation into a single declarative chain:
Including the source (`metadata.get("source")`) in the formatted context isn't cosmetic: it lets the model cite its source in the answer, which reduces perceived hallucinations and makes auditing easier in regulated domains (finance, healthcare, legal) — something that, in our projects with clients in the region, turns out to be a non-negotiable requirement.
No re-ranking technique compensates for poorly done chunking. We recommend `RecursiveCharacterTextSplitter` with a `chunk_size` between 500-800 tokens and 10-15% `chunk_overlap` for general text, but for structured documents (contracts, technical manuals) a structure-aware splitter like `MarkdownHeaderTextSplitter` is better, preserving the section hierarchy as metadata on each chunk, dramatically improving the relevance of the retrieved context.
Any change to the retrieval pipeline should be measured against an evaluation set with known "gold" questions and documents, using metrics like recall@k and MRR (Mean Reciprocal Rank) before and after the change. LangSmith lets you version these evaluation datasets and run automatic comparisons, a topic we cover in detail in our article dedicated to observability.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel