Few-shot examples are one of the most effective ways to improve a model's quality without fine-tuning -- but also one of the easiest ways to double your bill if you throw them into every request without discipline.
The common mistake is thinking the few-shot problem is "how many to include." The real problem is that, without caching, every example gets billed at full price on every request -- even if it's the same 3,000-token block you sent a second ago. If your prompt has 10 well-chosen examples taking up 4,000 tokens, and you make 1,000 requests a day with no caching, you're paying 4 million input tokens just on examples that never changed.
The highest-impact fix is the simplest: put the examples block in a stable position in the prompt and mark it with `cache_control`.
With this, you only pay full price for the examples on the first request within each cache window (5 minutes by default); the rest are served at ~10% of the cost.
Before optimizing caching, optimize the quantity. The intuition of "more examples = better" doesn't hold past a certain point -- and each extra example dilutes the model's attention away from the actual pattern you want to teach. In practice:
- **3-5 well-chosen examples** covering real edge cases usually beat 15 generic ones. - Prioritize examples that cover **the case the model usually gets wrong**, not the trivial case it already handles fine without help. - One example of an ambiguous case with its correct resolution is worth more than three examples of the obvious case.
If you're using 15+ examples and still getting errors, the problem is probably not the number of examples -- it's that the task needs an explicit instruction that no example alone will communicate (disambiguation criteria, exact format, business constraints).
If you have a large library of examples (50, 100, several hundred) covering different task subtypes, sending all of them on every request is both expensive and counterproductive. The alternative is dynamically retrieving only the examples most relevant to the current input -- the same retrieval principle you'd use for RAG, applied to your example bank.
The trade-off: dynamic selection breaks caching for that specific block, because it changes per request. The right strategy is usually hybrid: a small, fixed, cached core of examples (the 2-3 universal cases), plus a dynamic, uncached handful for the specific case.
Often the original example comes from a full real conversation (with a greeting, business context, back-and-forth). The model doesn't need that full conversation -- it needs the clean input→output pattern.
Compressing real examples to their essential pattern usually cuts the few-shot block's size by 60-80% without losing the signal the model needs.
If you find yourself needing dozens of examples to cover the case variety, and the model still fails on new cases outside that distribution, that's a sign the problem isn't context -- it's the task. At that point there are two better paths than stacking more examples:
- **Fine-tuning**, if the pattern is stable and high-volume enough to justify training cost. - **Decomposing the task** into simpler subtasks with their own specialized prompts, instead of trying to make one prompt with 30 examples cover all the variety.
- Does this example cover a case the model actually fails on, or is it redundant with one you already have? - Is it in the prompt's cacheable position, or is it being regenerated on every request? - Is it compressed to the essential pattern, or dragging unnecessary conversational context? - Is the problem this example tries to solve really "didn't see enough cases," or is it ambiguous instruction that no example will fix?
The combination of fewer-but-better examples, aggressive caching of the stable block, and dynamic selection only when the volume of variants justifies it, is what separates a few-shot pipeline that scales from one that becomes unaffordable in production.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel