Bandwidth-aware scheduling is a computational resource optimization strategy that incorporates cross-datacenter bandwidth constraints into request routing decisions for distributed large language model (LLM) inference systems. This approach addresses a critical challenge in serving language models at scale: balancing the throughput demands of inference workloads against the limited and expensive bandwidth available between geographically distributed computing clusters.

In modern distributed inference architectures, LLM serving systems typically separate the computational workload into two distinct phases: the prefill phase, where input tokens are processed and the key-value cache is generated, and the decode phase, where output tokens are generated sequentially. These phases have fundamentally different computational characteristics and resource requirements.

Bandwidth-aware scheduling emerges from the practical challenge that the bandwidth connecting separate prefill and decode clusters represents a scarce, expensive resource. Rather than treating routing decisions as purely a function of compute capacity or latency, bandwidth-aware approaches explicitly model network congestion as a constraint on the scheduling problem ¹⁾. This consideration becomes increasingly important as inference workloads scale and multiple requests compete for limited inter-datacenter connectivity.

The technique is particularly relevant in Prefix-as-a-Service (PrfaaS) architectures, where the prefix (accumulated key-value cache) represents the primary artifact being transferred between computational stages. By making scheduling decisions cognizant of current bandwidth availability, systems can avoid bottleneck conditions that would otherwise degrade system throughput and increase request latency.

Bandwidth-aware scheduling integrates bandwidth metrics into the request routing algorithm through several mechanisms:

Constraint-Based Routing: The scheduler maintains real-time estimates of available bandwidth between prefill and decode clusters. When assigning an incoming request to a prefill cluster, the scheduler considers not only whether compute capacity exists, but whether sufficient bandwidth headroom remains to transfer the resulting key-value cache to available decode capacity within acceptable latency bounds.

Cost Function Integration: Many implementations augment traditional load-balancing cost functions with bandwidth utilization penalties. Rather than minimizing purely compute-related metrics (such as queue depth or response time), the cost function incorporates terms representing the expected bandwidth consumption for each potential routing decision ²⁾.

Predictive Bandwidth Management: Effective implementations employ predictive models of bandwidth demand based on request characteristics. For instance, requests with longer input sequences will generate larger key-value caches, requiring proportionally more bandwidth to transfer to decode clusters. By predicting these bandwidth demands, schedulers can make proactive routing decisions that prevent congestion rather than reacting to it reactively.

Admission Control: Bandwidth-aware schedulers frequently implement admission control policies that reject or queue incoming requests when accepting them would predictably cause bandwidth saturation. This ensures that accepted requests can be routed through the system without suffering degraded performance due to network congestion.

The practical importance of bandwidth-aware scheduling manifests in several key deployment scenarios:

Multi-Datacenter Inference: Organizations serving LLMs across geographically distributed datacenters must route requests through limited inter-datacenter links. Bandwidth-aware scheduling optimizes this routing to maintain consistent service quality while managing costs associated with expensive wide-area network capacity.

Inference Batching and Request Queueing: When multiple requests are batched together for efficiency, their collective key-value cache output can consume significant bandwidth. Schedulers that are bandwidth-aware can determine optimal batch compositions and sizes that balance compute efficiency against network utilization ³⁾.

Heterogeneous Cluster Architectures: In systems where prefill and decode clusters may have different network characteristics (for example, prefill running on high-memory, lower-bandwidth hardware while decode runs on high-bandwidth, optimized serving hardware), bandwidth-aware scheduling accounts for these asymmetries in routing decisions.

Despite its value, bandwidth-aware scheduling faces several implementation challenges:

Prediction Accuracy: Bandwidth demand prediction requires accurate modeling of how request characteristics (input length, batch size, model parameters) translate to actual network traffic. Mispredictions can lead to suboptimal routing or unnecessary request rejection.

Bandwidth Measurement Overhead: Continuously monitoring inter-cluster bandwidth availability introduces monitoring overhead and measurement latency. Stale bandwidth estimates can lead to poor scheduling decisions, while continuous high-resolution monitoring may itself consume significant resources.

Dynamic Network Conditions: Real-world networks exhibit time-varying congestion, packet loss, and link failures. Schedulers must adapt to these dynamics while maintaining decisions that remain valid even as network conditions shift.

Tradeoff Complexity: Bandwidth-aware scheduling introduces additional optimization objectives (minimize bandwidth congestion, minimize latency, maximize throughput, reduce cost) that may conflict. Balancing these tradeoffs requires careful tuning of scheduling algorithms and cost function weights.

Bandwidth-aware scheduling has become an increasingly important research area as inference workloads at scale reveal bandwidth as a bottleneck ⁴⁾. Recent work focuses on tighter integration of bandwidth awareness with other optimization objectives like latency-bounded serving and cost minimization.

Emerging approaches explore learned scheduling policies that implicitly model bandwidth constraints through training on simulated or real datacenter workloads, potentially capturing complex bandwidth-latency-throughput tradeoffs more effectively than hand-designed heuristics. Additionally, advances in in-network computing and programmable switching fabrics may enable more sophisticated bandwidth management at the network layer rather than purely at the scheduler level.

• Cache-Aware Request Placement
• Selective Offloading
• Compute-Optimal Allocation