vLLM: How to Serve Open Source LLMs at Production Scale

The difference between an LLM that runs fine on your laptop and one that serves thousands of requests per second without dying of OOM is called vLLM, and understanding why requires understanding a specific memory problem.

The problem vLLM solves: fragmented KV cache memory

When an LLM generates text token by token, it keeps a "KV cache" (key-value cache) per active request to avoid recomputing attention over already-processed tokens. Naive inference engines reserve contiguous memory for the maximum possible size of this cache for every request, which wastes enormous amounts of VRAM when real sequences are shorter than the maximum -- a problem analogous to memory fragmentation in operating systems.

vLLM, originally developed at UC Berkeley, solves this with PagedAttention: instead of contiguous memory, it splits the KV cache into fixed-size blocks (similar to virtual memory pages) allocated dynamically as needed. This cuts memory waste from up to 60-80% in naive implementations down to less than 4%, which translates directly into serving far more concurrent requests with the same hardware.

Continuous batching: the second performance pillar

Traditional batching (static batching) waits for every sequence in a batch to finish before processing the next batch, which wastes compute when some sequences finish much earlier than others (a short response vs. a long one in the same batch). vLLM implements continuous batching (also called iteration-level batching): as soon as a sequence finishes, its slot immediately frees up for a new request, with no need to wait for the rest of the batch to finish.

The combination of PagedAttention plus continuous batching is what gives vLLM throughput advantages of 2x to 24x over naive implementations in plain Hugging Face Transformers, according to the original paper's benchmark and repeatedly validated by the community under real production loads.

Standing up a production server

vLLM exposes a production-ready, OpenAI-compatible server with a single command, which greatly simplifies migrating code that already uses the OpenAI SDK to point at your own endpoint.

The `--tensor-parallel-size` flag distributes the model across multiple GPUs when it doesn't fit on one, and `--gpu-memory-utilization` controls what fraction of VRAM gets reserved for the paged KV cache -- raising it increases the number of concurrent requests you can serve at the cost of a smaller safety margin.

Quantization and parallelism: tuning for your hardware

vLLM supports multiple production quantization schemes -- AWQ, GPTQ, and FP8 on GPUs that natively support it (H100, L40S) -- letting you serve larger models in less VRAM without the latency penalty some storage-only-oriented quantization formats carry. For models exceeding a single GPU, besides tensor parallelism (splitting each layer across GPUs) it supports pipeline parallelism (splitting whole layers across GPUs), relevant for the largest MoE models like Mixtral or DeepSeek-V3.

It also supports "prefix caching" -- caching the KV cache of prompts shared across requests (for example, a common system prompt or repeated RAG context) to avoid recomputing that portion on every request, an optimization with a direct cost impact when many requests share initial context.

vLLM vs. the alternatives: TGI, TensorRT-LLM, Ollama

Hugging Face's Text Generation Inference (TGI) solves a similar problem with a similar philosophy (continuous batching, quantization), and on several benchmarks it's very close to vLLM in throughput -- the choice between the two usually depends more on how well each supports the specific model you want to serve and integration with the rest of your stack. NVIDIA's TensorRT-LLM gives the highest possible performance but requires compiling the model specifically for the exact GPU architecture you'll use, with a more rigid, less portable development cycle.

Against Ollama: the difference is one of purpose. Ollama prioritizes development simplicity and low traffic; vLLM prioritizes throughput under real concurrent load. For a production service with dozens or hundreds of simultaneous users, vLLM is virtually always the right choice over Ollama.

Kubernetes deployment considerations

For horizontal scaling, vLLM typically deploys behind a load balancer with multiple replicas, each serving the model on its own set of GPUs, coordinated with a queueing system (or directly with Kubernetes' standard load balancing) when traffic exceeds a single instance's capacity. Observability is handled with vLLM's native Prometheus metrics (throughput, time-to-first-token latency, KV cache usage), critical for correctly sizing how many replicas you need before latency degrades under real load.

In a recent deployment serving a 32B-parameter model for a client with constant production traffic, migrating from a plain-Transformers-based server to vLLM on the same two A100 GPUs multiplied sustainable throughput by roughly 6x, eliminating the need to provision additional hardware that had initially been budgeted.