====== Prefill-as-a-Service (PrfaaS) ======
**Prefill-as-a-Service (PrfaaS)** is a distributed serving architecture that optimizes the inference pipeline of large language models (LLMs) by decoupling prefill and decode operations across geographically distributed compute clusters. The architecture selectively offloads long-context prefill computation to specialized, compute-dense prefill clusters and transfers the resulting key-value (KV) cache over standard commodity Ethernet to local decode clusters, enabling independent scaling of these two distinct operational phases without requiring shared low-latency RDMA (Remote Direct Memory Access) fabric.

===== Overview and Architecture =====
Modern LLM inference consists of two distinct computational phases: the **prefill phase**, where the model processes all input tokens to generate the key-value cache, and the **decode phase**, where the model autoregressively generates one token at a time using the precomputed cache. These phases have fundamentally different computational characteristics and resource requirements. PrfaaS exploits this distinction by implementing a cross-datacenter serving pattern that physically separates these workloads across loosely coupled clusters (([[https://arxiv.org/abs/2305.05374|Kwon et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention" (2023]])).

The PrfaaS architecture comprises three primary components: specialized prefill clusters optimized for throughput and long-context processing, decode clusters designed for low-latency token generation, and a network layer handling KV cache transmission. When a request arrives, the system determines whether prefill computation should be executed locally or offloaded to a remote prefill cluster based on factors including input length, current local capacity, and network bandwidth constraints. After prefill completion, the resulting KV cache is serialized and transferred via standard Ethernet rather than requiring dedicated RDMA infrastructure.

===== Computational Decoupling and Resource Optimization =====
The prefill phase is inherently compute-dense, with computational intensity increasing proportionally to context length. In contrast, the decode phase exhibits memory-bound characteristics with minimal arithmetic operations per memory access. Traditional co-located serving architectures force these incompatible workloads onto the same hardware, resulting in suboptimal resource utilization. PrfaaS resolves this constraint by enabling **independent cluster scaling**: prefill clusters can be provisioned with hardware optimized for matrix multiplication throughput (such as higher compute-to-memory ratios), while decode clusters prioritize low-latency memory access and smaller model replicas (([[https://arxiv.org/abs/2104.04473|Zhou et al. "Towards Efficient Transformer Decoding" (2021]])).

This architectural separation provides several practical benefits. Prefill clusters can maintain higher batch sizes without latency constraints, approaching maximal GPU utilization. Decode clusters focus exclusively on minimizing time-to-first-token (TTFT) and inter-token latency, critical user-experience metrics. The system can independently adjust cluster sizes in response to workload fluctuations—scaling prefill capacity during periods of long-context requests while maintaining constant decode capacity. By offloading long-context prefill computation to specialized compute-dense clusters, PrfaaS optimizes overall serving efficiency and enables better resource utilization across distributed infrastructure (([[https://tldr.tech/ai/2026-04-20|TLDR AI - Long-Context Prefill (2026]])).

===== KV Cache Transfer and Network Considerations =====
The central technical challenge in PrfaaS is efficiently transferring KV cache across the network boundary. The KV cache size is proportional to context length and batch size, scaling as O(context_length × hidden_dimension × batch_size). For typical configurations with 4K-token contexts and batch sizes of 32, KV cache sizes may reach hundreds of megabytes per request. Rather than requiring specialized RDMA fabric, PrfaaS leverages commodity Ethernet, reducing infrastructure costs and improving geographical flexibility (([[https://arxiv.org/abs/2309.06852|Zhang et al. "DistServe: Disaggregating Prefill and Decode for Goodput-Optimized LLM Serving" (2023]])).

Network bandwidth becomes a limiting factor when context lengths exceed certain thresholds. For a 100 Gbps Ethernet connection, transferring a 200 MB KV cache requires approximately 16 milliseconds. Systems implementing PrfaaS must balance prefill offloading decisions against network transmission time. Optimization strategies include [[kv_cache_compression|KV cache compression]] techniques, selective attention mechanisms that reduce cache dimensionality, and intelligent routing to minimize cross-datacenter transfers through localized prefill processing (([[https://arxiv.org/abs/2310.08059|Li et al. "PowerInfer: Fast Large Language Model Serving with a Mobile GPU" (2023]])).

===== Practical Applications and Deployment Scenarios =====
PrfaaS proves particularly valuable in deployments with diverse request characteristics and geographic distribution requirements. Multi-tenant serving environments benefit from independent scaling—a single tenant may submit many short requests suitable for local decode processing while another submits occasional long-context queries. Content delivery and edge-computing scenarios leverage PrfaaS to maintain regional decode clusters while centralizing expensive prefill computation.

The architecture enables cost-effective serving of variable-length prompts. Datacenters can provision fewer prefill-optimized nodes relative to decode nodes, amortizing high prefill computational costs across multiple geographically distributed decode locations. This pattern suits services like document analysis, code generation, and reasoning-intensive applications that accept moderate latency for prefill phases in exchange for fast interactive decode.

===== Limitations and Current Challenges =====
Despite architectural advantages, PrfaaS introduces complexity and constraints. Network bandwidth represents a hard limit on context length and batch size combinations. Datacenter networking, while substantially cheaper than RDMA fabric, adds non-trivial latency (typically 0.5-2 milliseconds cross-datacenter compared to 1-10 microseconds within a single machine). For extremely latency-sensitive applications, this overhead may prove prohibitive (([[https://arxiv.org/abs/2305.05374|Kwon et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention" (2023]])).

Operational complexity increases through additional components: prefill cluster management, KV cache serialization and deserialization, network monitoring, and failure handling across cluster boundaries. Debugging distributed inference systems becomes more challenging when prefill and decode failures occur in different geographic locations.


===== See Also =====

  * [[prefill_vs_decode_scaling|Prefill vs Decode Capacity Scaling]]
  * [[pretraining_scaling|Pretraining Scaling]]
  * [[advisor_pattern|Advisor Pattern]]
  * [[llama_cpp|llama.cpp]]

===== References =====