vLLM is a high-throughput, memory-efficient open-source inference and serving engine for large language models, developed by UC Berkeley's Sky Computing Lab. It features innovations like PagedAttention for KV cache management and continuous batching for dynamic request handling.¹⁾

PagedAttention treats GPU memory like an operating system's virtual memory, breaking the KV cache into smaller, reusable non-contiguous pages instead of large per-request blocks.²⁾ This approach:

• Reduces memory waste to near-zero through efficient allocation
• Enables sharing of KV cache pages across requests
• Supports larger batch sizes on the same hardware
• Achieves up to 14-24x higher throughput than traditional methods

The mechanism separately optimizes prefill (initial prompt processing) and decode (iterative token generation) phases, with features like prefix caching and chunked prefill for long sequences.

vLLM uses continuous (dynamic) batching, processing incoming requests in a continuous stream rather than static batches.³⁾ This:

• Maximizes GPU utilization by dynamically adding and removing sequences mid-execution
• Handles heterogeneous requests with varying lengths
• Supports streaming outputs and multi-LoRA serving
• Reduces latency under real-world bursty traffic patterns

vLLM supports tensor parallelism and pipeline parallelism for distributed inference across multiple GPUs, alongside:⁴⁾

• Optimized CUDA/HIP graphs for reduced kernel launch overhead
• Custom CUDA kernels integrating FlashAttention and FlashInfer
• Quantization support (AWQ, GPTQ, INT4/INT8/FP8)
• Speculative decoding for further throughput improvements
• Dual batch overlap for concurrent prefill and decode

2026 benchmarks on H100 GPUs show vLLM alongside newer competitors:⁵⁾

While newer engines show throughput advantages, vLLM remains widely adopted for its maturity, ecosystem, and production reliability.

vLLM provides an OpenAI-compatible API server supporting:⁶⁾

• Chat completions and text completions endpoints
• Parallel sampling and beam search
• Token streaming for real-time responses
• High-throughput batch serving
• Drop-in replacement for OpenAI API clients

• Hugging Face Transformers models (LLaMA, Mistral, Falcon, GPT-NeoX, and instruction-tuned variants)
• Quantized formats (AWQ, GPTQ, INT4/INT8/FP8)
• Vision encoders (ViT) for multimodal models
• Multi-LoRA serving for concurrent adapter inference
• Prefix caching for shared prompt prefixes

• Hardware – NVIDIA GPUs, AMD GPUs (ROCm), Intel/AMD/PowerPC CPUs, Gaudi accelerators, TPUs, AWS Trainium/Inferentia⁷⁾
• API – simple Python API for programmatic use
• Docker – containerized deployment for production environments
• Kubernetes – scalable orchestration for multi-replica serving

• Text Generation Inference
• llama.cpp
• Ollama
• Hugging Face