A RAG system that retrieves the right documents but presents them poorly to the model gets the same result as one that retrieves badly: imprecise answers. Retrieval is half the problem -- how you structure what's retrieved is the other half, and it's the half fewer teams take care of.
Language models don't pay uniform attention to all of their context. The consistent evidence across the industry is that information at the beginning and end of the context gets processed more faithfully than what sits in the middle. If your RAG stuffs in 10 documents and the critical fact is in document number 6, the model is more likely to miss it than if it were in document 1 or 10.
The practical implication: it's not enough to retrieve the right documents by raw relevance -- the **order** in which you present them matters. Put the most relevant document (per your reranker) at the beginning and end, not buried in the middle of a long list.
Embedding similarity search retrieves "semantically similar" documents, but that's not the same as "the documents that actually answer the question." A reranking step -- a model (or even the same LLM) that reorders by true relevance to the specific query -- drastically reduces the noise that reaches the final context.
The "broad recall + narrow reranking" pattern is cheaper and more precise than raising the vector retrieval's `k` directly, because the reranker operates over a small, already pre-filtered set.
A common mistake: concatenating retrieved chunks as plain text, with no marker for where each source starts and ends. This keeps the model from distinguishing where each claim comes from, hurting both precision and the ability to cite sources.
Delimiting each source with XML or markdown isn't cosmetic -- it measurably improves the model's ability to correctly attribute each fact to its origin, and makes it possible to ask for verifiable citations.
Some models support a native citations mode: instead of asking the model to "cite the source" in free text (which can hallucinate the citation), the system automatically returns which exact document fragment backs each part of the response.
This eliminates an entire class of hallucination: the "invented" citation that sounds plausible but doesn't correspond to the real text.
It's common for retrieval to return multiple chunks saying essentially the same thing (the same fact repeated across different sections of a document, or in near-duplicate documents). Feeding all three into the context doesn't improve the answer -- it inflates tokens and increases the surface for contradictions if the versions have slightly different nuances.
Simple deduplication by similarity across retrieved chunks, before building the final prompt, usually shrinks context size without losing coverage.
If multiple users ask about the same base document, or the same user asks several questions over the same retrieved context in a session, that context block is a direct candidate for `cache_control` -- exactly the same principle from the prompt caching article, applied specifically to RAG-retrieved content instead of the system prompt.
The most common final mistake: when RAG fails, people assume it's the model generating a bad answer. Before touching the prompt, verify whether the problem is that retrieval never brought back the right document in the first place. An evaluation set with questions whose correct answer you know, and the document that contains it identified beforehand, lets you measure separately:
- **Retrieval recall:** was the correct document among the retrieved ones? - **Generation precision:** given that the correct document was present, did the model use it well?
Mixing these two metrics is the number one reason teams "fix" the prompt when the real problem was in the search index.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel