Long Short-Term Memory (LSTM) is a specialized recurrent neural network (RNN) architecture designed to effectively model sequential data and temporal dependencies. Introduced in the 1990s by Sepp Hochreiter and collaborators ¹⁾, LSTMs addressed a fundamental limitation of standard RNNs: the inability to learn long-range dependencies due to the vanishing gradient problem. By incorporating memory cells and sophisticated gating mechanisms, LSTMs became the dominant sequence modeling paradigm for nearly two decades, powering critical applications in machine translation, speech recognition, and early language modeling before the emergence of Transformer-based architectures in 2017.

Traditional RNNs struggled with learning long-range dependencies due to the vanishing gradient problem, where gradients exponentially decrease as they backpropagate through many time steps ²⁾. This limitation made it difficult for networks to capture relationships between distant elements in sequences. The LSTM architecture, building on earlier gated memory concepts, introduced a principled solution through the introduction of memory cells and gating functions that regulate information flow ³⁾.

The core innovation of LSTMs lies in their memory cell structure, which maintains information across many time steps through additive interactions rather than multiplicative weight multiplication ⁴⁾.

LSTMs maintain an internal cell state that flows unchanged through the network, allowing gradients to propagate without the severe degradation seen in vanilla RNNs. The architecture employs three primary gating mechanisms, each implemented as a sigmoid-activated neural network:

* Input Gate: Controls which new information enters the cell state * Forget Gate: Determines what information from the previous cell state is discarded * Output Gate: Regulates which parts of the cell state are exposed as the hidden state output

At each time step, the LSTM computes a candidate cell state update using a tanh activation function, which is then selectively added to the existing cell state according to the input gate. The gating mechanisms are mathematically expressed through sigmoid and tanh activation functions that produce continuous values between 0 and 1, enabling fine-grained control over information flow ⁵⁾.

The memory cell itself uses additive updates rather than multiplicative weight matrices, which preserves gradient magnitude across many sequential steps. This design choice directly addresses the vanishing gradient problem that plagued earlier RNN architectures, allowing effective training on sequences containing hundreds or thousands of time steps. The additive nature of cell state updates enables stable gradient flow during backpropagation through time, fundamentally addressing the vanishing gradient limitation ⁶⁾.

LSTMs achieved breakthrough performance across multiple sequential modeling domains and substantial success during the 2010s and early 2020s.

In machine translation, LSTM-based sequence-to-sequence models significantly outperformed earlier statistical approaches, enabling neural machine translation systems that could handle variable-length input and output sequences ⁷⁾, capturing long-range syntactic and semantic dependencies in natural language.

In speech recognition, LSTMs improved acoustic modeling by capturing long-range acoustic patterns and phonetic variations. Bidirectional LSTMs were employed to process audio signals, and systems utilized deep LSTM architectures to enhance accuracy in speaker-dependent and speaker-independent contexts ⁸⁾.

LSTMs achieved critical success across multiple domains and became the standard architecture for sequence modeling until the emergence of Transformer-based architectures, which offered superior parallelizability and computational efficiency.

• xLSTM Architecture
• LSTM vs Transformer
• Sequence Modeling