Attention compression refers to a class of techniques designed to reduce the computational and memory overhead of attention mechanisms in large language models by condensing historical context into compact summaries rather than maintaining complete token-level information across entire sequences. This approach addresses a fundamental scalability challenge in transformer architectures: the quadratic complexity of standard attention mechanisms, which becomes prohibitively expensive for long-context applications.

Transformer-based language models rely on attention mechanisms to process dependencies between tokens, enabling the model to weight relevant information when generating predictions. However, standard scaled dot-product attention computes interaction matrices between all pairs of tokens, resulting in O(n²) computational complexity and O(n²) memory usage relative to sequence length. For models processing extended contexts—spanning thousands or millions of tokens—this quadratic scaling becomes a critical bottleneck.

Attention compression addresses this limitation by selectively reducing the information density of historical context while preserving task-relevant signals. Rather than storing complete embeddings and attention values for all previous tokens, compression techniques create lossy or lossless summaries that capture essential information at reduced computational cost. This trade-off between compression ratio and information preservation forms the core technical challenge in designing effective attention compression schemes ¹⁾

Several architectural patterns have emerged for implementing attention compression:

Hierarchical Compression: Models organize attention into blocks or hierarchies, where recent tokens maintain full-resolution attention while older tokens are progressively compressed into summaries. This preserves fine-grained access to recent context while using cheaper operations for historical information.

Token Summarization: Specific tokens or token clusters are selected or created to represent compressed state. These summary tokens are computed through pooling, clustering, or learned selection mechanisms, reducing the effective sequence length for subsequent attention computations ²⁾

Hybrid Attention Blocks: As implemented in systems like DeepSeek-V4, hybrid approaches allocate different attention mechanisms to different layers. Some layers may use standard full attention on recent context, while other layers employ compressed attention blocks that operate on summarized historical states. This stratified approach enables selective computation based on the information richness required at each processing stage.

Sparse Attention Patterns: Rather than compressing information, some approaches maintain sparsity patterns that connect tokens to a subset of previous tokens selected through heuristics or learned criteria. Local attention windows, strided patterns, and learned sparsity mask different aspects of the attention landscape while reducing overall computation ³⁾

Attention compression has become increasingly important for real-world applications requiring extended context windows:

Long-Document Processing: Information retrieval and document analysis tasks benefit from attending to thousands of tokens without proportional increases in computational cost. Compression enables models to maintain broader contextual awareness while processing long documents efficiently.

Multi-Turn Conversation: In dialogue systems, conversation history can expand rapidly across multiple turns. Rather than maintaining full attention over all historical exchanges, compression summarizes earlier conversation segments, preserving key semantic content while reducing memory requirements for ongoing inference.

Retrieval-Augmented Generation: When augmenting language models with retrieved documents, attention compression allows efficient processing of multiple retrieved passages without quadratic scaling relative to document count ⁴⁾

Streaming and Inference: In production systems with continuous token generation, compression enables incremental processing where older tokens can be progressively discarded or summarized, maintaining constant memory overhead as generation continues.

Despite advantages, attention compression introduces several technical tradeoffs:

Information Loss: Compression necessarily discards detailed information about historical context. For tasks requiring precise recall of specific facts from distant context, over-aggressive compression may degrade performance. The optimal compression ratio varies significantly across applications and tasks.

Compression Overhead: Creating summaries requires additional computation. If summary generation costs approach savings from reduced attention complexity, overall efficiency gains may be marginal. This creates an efficiency frontier that varies with context length and model architecture.

Generalization: Models trained with specific compression schemes may not generalize to test scenarios with significantly longer contexts than training contexts, as the compression parameters and strategies may be optimized for specific length distributions.

Quality Degradation: Empirical studies show that attention compression typically produces measurable performance reductions compared to full attention, particularly for retrieval-focused tasks or when long-range dependencies prove critical to task performance.

Recent work explores learned compression strategies where models dynamically determine which information to compress based on task requirements. Research also investigates semantic-aware compression that preserves high-level meaning while discarding surface-level details, and adaptive approaches that adjust compression aggressiveness based on available computational budgets ⁵⁾

The evolution of attention compression reflects broader trends toward efficient transformers, as the field seeks to scale context length without proportional computational costs while maintaining strong task performance.

• Attention Kernel Optimization
• Block-Sparse Attention
• Hybrid Attention Architecture (CSA + HCA)
• Flash-Linear-Attention
• Multi-Head Latent Attention (MLA)