A hidden state is a fixed-size vector representation maintained internally by recurrent neural networks (RNNs) that encodes temporal and sequential information across multiple timesteps. This fundamental architectural component enables RNNs to process variable-length sequences while maintaining a compact, learnable representation of relevant historical context. The hidden state serves as the network's “memory,” allowing information from previous inputs to influence predictions on subsequent timesteps.¹⁾

The hidden state, often denoted as h_t, is computed at each timestep through a deterministic transformation of the current input and the previous hidden state. Formally, the recurrent update follows the pattern: h_t = f(x_t, h_t-1), where x_t represents the input at timestep t and f is a nonlinear activation function ²⁾.

The dimensionality of the hidden state—typically ranging from tens to thousands of units in modern architectures—determines the representational capacity of the network. Unlike convolutional neural networks that grow with input size, RNNs maintain O(1) inference memory complexity with respect to sequence length, as the hidden state remains fixed-size regardless of how long the input sequence becomes ³⁾.

Classical RNN variants, such as vanilla RNNs, suffered from vanishing and exploding gradient problems that limited the effective temporal dependencies the hidden state could capture. Long Short-Term Memory (LSTM) networks addressed these limitations by introducing a cell state alongside the hidden state, using gating mechanisms to control information flow ⁴⁾.

Modern RNN architectures feature significantly larger hidden state dimensions compared to their classical predecessors, enabling substantially improved performance on complex sequential tasks. Gated Recurrent Units (GRUs) and contemporary LSTM variants further refined the mechanisms by which hidden states accumulate and utilize information across timesteps. The expansion of hidden state size, combined with advances in training techniques and computational infrastructure, has enabled RNNs to achieve state-of-the-art results on sequence modeling, machine translation, and time series prediction tasks while preserving the memory-efficient inference properties that make them practical for deployment ⁵⁾.

Hidden states enable RNNs to model sequential dependencies in diverse domains including natural language processing, speech recognition, video analysis, and financial time series forecasting. The bidirectional processing of sequences using separate forward and backward hidden states—often concatenated to form richer representations—has become standard practice in many applications ⁶⁾.

However, hidden state architectures face inherent limitations. The bottleneck created by compressing entire sequences into a fixed-size vector becomes problematic for very long sequences, limiting the network's ability to maintain distant dependencies. Attention mechanisms and Transformer architectures emerged partially to address these constraints by allowing direct comparisons between all positions in a sequence, reducing reliance on the hidden state as the sole information carrier. Despite these alternatives, hidden states remain fundamental to understanding sequential neural computation and continue to inform modern architectural designs.

• Recurrent Neural Networks (RNNs)
• Long Short-Term Memory (LSTM)
• State Space Models (SSMs)