Prefill/Decode Disaggregation is an architectural optimization in large language model (LLM) inference that separates the processing of input tokens (prefill phase) from the generation of output tokens (decode phase). This separation enables specialized computational strategies and resource allocation for each phase, addressing fundamental differences in their computational characteristics and memory access patterns. The approach has become industry standard across major hardware and software providers, with significant investments from NVIDIA, Intel, and Amazon in dedicated optimization stacks.
The traditional LLM inference pipeline processes all input tokens through the model sequentially before entering token generation. Prefill/Decode Disaggregation bifurcates this pipeline into two distinct phases with different optimization requirements.
The prefill phase processes large batches of input tokens simultaneously, characterized by high arithmetic intensity and parallelizable matrix operations. During this phase, the model computes attention mechanisms across all input tokens, where each token attends to all previous tokens in the input sequence. This phase exhibits favorable properties for GPU acceleration—high compute-to-memory ratios enable efficient tensor operations and batch parallelization across thousands of tokens.
The decode phase generates output tokens one at a time (or in small groups), where each new token depends on all previously generated tokens. This autoregressive generation exhibits fundamentally different computational characteristics: memory-bound operations dominate, with relatively small matrix multiplications repeated iteratively. Each decode step requires loading the entire model weights from memory to produce a single token, creating a memory bandwidth bottleneck rather than a compute bottleneck.
By disaggregating these phases, systems can apply specialized optimizations to each1). Prefill operations benefit from large-batch matrix multiplication kernels and standard GPU acceleration strategies. Decode operations require different approaches: smaller batch sizes, memory-efficient attention implementations, and token-server architectures that prioritize latency reduction.
Prefill and decode phases differ substantially in their computational bottlenecks. Prefill operations feature compute-bound characteristics—the ratio of arithmetic operations to memory accesses is high, allowing GPUs to achieve near-peak compute utilization. Hardware accelerators can parallelize across input tokens with minimal memory access overhead.
Decode operations exhibit memory-bound characteristics—token generation requires loading model parameters repeatedly with minimal arithmetic per memory access. For a model with B parameters and generating T tokens, decode requires O(B·T) memory accesses for O(B·T) arithmetic operations, yielding a ratio near one. This memory bandwidth limitation means decode latency is primarily determined by memory speed rather than compute speed2).
Specialized decode optimizations address this constraint through:
- Quantization: Reducing model parameter precision (INT8, INT4) to fit larger portions in cache and reduce memory bandwidth requirements - Paged attention: Allocating GPU memory in fixed-size pages to reduce memory fragmentation and improve cache efficiency3) - Token batching: Grouping multiple token generation requests to amortize model loading costs - Speculative decoding: Using smaller models to predict tokens and validate with larger models, reducing forward passes needed4)
Prefill/Decode Disaggregation has become the dominant paradigm in production LLM inference systems, with major technology companies developing specialized hardware and software architectures around this separation.
NVIDIA integrated disaggregation principles into optimization frameworks, with NVIDIA TensorRT-LLM supporting distinct kernel implementations for prefill and decode phases. The company's approach recognizes that token-per-cycle throughput differs fundamentally from token generation latency, requiring separate tuning strategies.
Intel developed dedicated inference accelerators around disaggregation principles, optimizing for throughput-oriented prefill operations on standard compute cores while using specialized decode acceleration units. This architectural split reflects the fundamental computational differences between phases.
Amazon's neuron compiler for AWS Trainium chips incorporates prefill/decode separation, enabling automatic kernel selection and memory layout optimization based on phase detection. This approach allows flexible deployment across different inference scenarios—high-throughput batch prefilling for content recommendations or search reranking, versus low-latency single-token generation for conversational applications.
This standardization reflects industry recognition that treating prefill and decode identically wastes computational resources and introduces unnecessary latency.
Disaggregation enables more efficient batch utilization. Operators can separate prefill and decode workloads across hardware pools, running high-throughput batch prefilling on GPUs optimized for parallel computation while routing decode operations to hardware tuned for memory bandwidth. This flexibility improves overall inference throughput and reduces latency tail percentiles5).
However, disaggregation introduces operational complexity. Systems must manage queue scheduling across disaggregated compute pools, balance latency and throughput objectives, and handle variable request distributions. Request batching optimization becomes more complex when routing decisions must account for whether requests are in prefill or decode phase.
Memory management presents additional challenges. Prefill phase typically creates large KV (key-value) cache allocations, while decode phase reuses and extends these caches. Managing cache coherence and eviction policies across disaggregated hardware requires sophisticated memory management systems.