Serving LLM requests one at a time wastes most of the GPU. Batching done right is the difference between paying for idle capacity and squeezing every bit of it.
LLM inference is dominated by memory reads (moving the model's weights from GPU memory to the compute units), not by the compute itself -- it's "memory-bound." Processing one request at a time means the GPU loads the entire model's weights just to generate a single token for a single sequence, leaving most of its parallel compute capacity unused. Grouping several requests into the same compute step (a "batch") lets that same weight load be reused to generate tokens for multiple sequences at once -- the cost of moving the weights gets amortized across every request in the batch.
Classic static batching groups a fixed set of requests, waits for all of them to finish generating (including the longest one), and only then frees the whole batch to accept new requests -- if one request finishes fast but another generates 500 tokens, the first one's slot sits idle until the entire batch closes. Continuous batching (also called dynamic batching), popularized by engines like vLLM and TGI, fixes this at the scheduler level: the moment a request finishes, its slot is immediately freed for a new request from the queue, without waiting for the rest of the batch to wrap up.
The gap between static batching and continuous batching can mean several times more throughput on the same hardware for workloads with variable response length -- which is the norm in production, where some questions get answered in 20 tokens and others in 500. That translates directly into cost: if you can serve 5x more requests per second on the same GPU, cost per request drops proportionally, without changing the model or meaningfully degrading any individual request's latency.
Batching isn't free: adding more requests to a batch already in flight can increase how long it takes any single request to get its first token, because the scheduler has to split available compute across more concurrent sequences. Production engines expose parameters to cap batch size or prioritize certain requests, and finding the right point depends on whether your application can tolerate more latency in exchange for more throughput (an overnight batch job) or needs guaranteed fast response (a live chat).
If you consume LLMs via a provider's API (OpenAI, Anthropic, Bedrock), the provider already applies batching internally across its shared infrastructure -- it's not something you configure directly. Where you do have control is in how you structure your own calls: using the batch endpoints several providers offer for non-interactive jobs (processing thousands of documents with no need for an immediate response) usually comes with significant discounts over the synchronous API price, precisely because the provider can group them with much more freedom when there's no real-time latency constraint.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel