An autoregressive transformer is a neural network architecture that generates sequences by predicting one token or element at a time, conditioning each prediction on all previously generated tokens. This sequential generation approach has become foundational to modern language models, vision systems, and specialized applications across multiple domains in deep learning.

Autoregressive transformers extend the standard transformer architecture with a unidirectional attention mechanism that prevents the model from attending to future tokens during generation. This causal masking constraint ensures that predictions for position t depend only on tokens at positions 0 through t-1, making the model suitable for sequential generation tasks.

The architecture combines several key components: a token embedding layer that converts discrete inputs into continuous representations, positional encoding that encodes sequence position information, multiple stacked transformer blocks with causal self-attention, and a final output layer that produces probability distributions over the vocabulary or output space ¹⁾

The causal attention mask is implemented by setting attention weights to negative infinity for all future positions, ensuring that the softmax operation produces zero attention probabilities for inaccessible tokens. This constraint is mathematically enforced during the attention computation, where the attention weight for position i attending to position j is masked when j > i.

During inference, autoregressive transformers generate sequences through iterative prediction. At each step, the model takes the accumulated sequence of tokens generated so far and computes probability distributions over the next token. Sampling from this distribution produces the next token, which is then appended to the sequence. This process continues until a designated end-of-sequence token is generated or a maximum length is reached.

The computational cost of generation scales with sequence length, as each step requires a forward pass through the entire network. Techniques like key-value caching significantly reduce this cost by storing previously computed attention keys and values, avoiding redundant computation on earlier tokens. This optimization allows efficient generation of long sequences without recomputing attention for the entire prefix ²⁾

Decoding strategies influence the quality of generated sequences. Greedy decoding selects the highest-probability token at each step, while beam search explores multiple hypotheses in parallel to find higher-probability sequences. Nucleus sampling and temperature-based sampling introduce controlled randomness, allowing diverse but plausible outputs.

Autoregressive transformers have become the dominant architecture for large language models, including GPT-series models, which apply this architecture to natural language generation tasks ³⁾. The architecture extends beyond text to multimodal domains, including image generation, where tokens represent image patches or learned latent codes.

Specialized applications adapt the autoregressive framework to domain-specific problems. TARIO-2 applies autoregressive transformer principles to predict spatial gene expression maps from H&E histology images, where the model learns to sequentially generate expressions across spatial dimensions based on visual pathology features. This application demonstrates how the sequential generation paradigm can model spatial and biological relationships beyond traditional text processing.

The architecture also enables few-shot and in-context learning capabilities, where models condition generation on demonstrations or instructions provided in the prompt context. This emergent ability arises from training on diverse sequences and enables adaptation to new tasks without parameter updates ⁴⁾

Autoregressive generation exhibits quadratic complexity in sequence length due to the attention mechanism, limiting scalability to very long sequences. While key-value caching optimizes inference time, training still requires full attention computation across all positions.

The sequential nature of generation introduces exposure bias, where the model trains on teacher-forced sequences but generates autoregressively during inference, creating a distribution mismatch. This phenomenon can accumulate errors in longer sequences, particularly when the model encounters out-of-distribution states that differ from training data.

Decoding speed remains a practical constraint for real-time applications, as generating long sequences requires many sequential forward passes. Recent research explores speculative decoding and parallel decoding approaches to accelerate generation while maintaining quality ⁵⁾.

• Transformer Architecture
• Autoregressive (AR) Decoding
• Attention Mechanism
• Post-Transformer Architectures
• Recurrent-Depth Transformers