Sensory memory is the earliest stage of memory processing in cognitive architectures and AI agent systems, responsible for briefly retaining raw perceptual input from the environment. In the context of autonomous agents, sensory memory corresponds to the initial encoding of observations such as text, images, audio, or other modalities before they are processed into more structured representations. Drawing from cognitive science models, sensory memory in agents serves as a high-bandwidth buffer that captures far more information than can be consciously processed, with most data decaying rapidly unless attended to.

In human cognition, sensory memory comprises modality-specific stores: iconic memory for vision (lasting ~250ms), echoic memory for audition (~3-4 seconds), and haptic memory for touch. George Sperling's 1960 partial report experiments demonstrated that iconic memory holds far more information than can be reported, but it decays within milliseconds unless attention is directed to it. This finding directly inspires AI agent design, where raw input streams contain vastly more information than can fit in a context window, requiring selective attention mechanisms to filter what gets promoted to working memory.

Research by Cornell (2025) on brain-inspired AI models demonstrates that artificial systems can learn to sort “degraded” or noisy sensory input efficiently, mirroring how biological systems handle occluded or ambiguous stimuli. Mathematical models from ModernSciences (2025) suggest that seven sensory dimensions may be optimal for maximizing memory capacity in artificial agents, guiding the design of multimodal robot systems.

Modern multimodal LLM agents implement sensory memory through specialized encoder modules that convert raw input into token embeddings:

Vision Processing. Vision Transformers (ViT), introduced by Dosovitskiy et al., 2020, divide images into fixed-size patches, apply self-attention across patch embeddings, and produce representations fed into downstream LLMs.¹⁾ Models like GPT-4o (OpenAI, 2024) integrate native multimodal encoders for real-time vision-text alignment. Gemini 2.0 (Google, 2024) uses a unified architecture with Mixture-of-Experts attention for scalable visual processing across images and video.

Text Processing. Text input is tokenized via subword methods (BPE, SentencePiece) and embedded into dense vector representations. This is the most mature modality, requiring minimal sensory preprocessing beyond normalization and tokenization.

Audio Processing. Models like Whisper (Radford et al., 2022)²⁾ convert speech waveforms or spectrograms into embeddings via transformer encoders. GPT-4o and Gemini 2.0 process audio natively, with attention mechanisms capturing temporal dependencies in the signal.

Multimodal Fusion. CLIP (Radford et al., 2021)³⁾ introduced contrastive pretraining that aligns image and text embeddings into a shared space, enabling zero-shot visual grounding. Modern agents use cross-attention layers to fuse information across modalities early in processing, allowing vision, text, and audio to inform each other before reaching the reasoning stage.

Attention is the mechanism that bridges sensory memory and short-term memory, determining which raw inputs are promoted for further processing:

Multi-Head Self-Attention in transformers computes, for each head $h$:

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

where $Q = XW_Q$, $K = XW_K$, $V = XW_V$ are the query, key, and value projections, and $d_k$ is the key dimension. Multiple heads attend to different aspects of the input simultaneously, spatial relationships, semantic content, temporal patterns, effectively implementing parallel sensory filters. The outputs of all $H$ heads are concatenated:

$$\text{MultiHead}(X) = \text{Concat}(\text{head}_1, \ldots, \text{head}_H)W_O$$

Cross-Modal Attention in multimodal models (GPT-4o, Gemini, Claude 3.5) enables one modality to query another. For example, a text query can attend to relevant image regions via $Q_{\text{text}} K_{\text{image}}^\top$, or an audio signal can be grounded in visual context. This mirrors how human sensory memory integrates information across modalities before conscious processing.

Perceptual Grounding connects raw sensory representations to semantic meaning. CLIP-style contrastive learning minimizes:

$$\mathcal{L}_{\text{CLIP}} = -\frac{1}{N}\sum_{i=1}^{N}\left[\log\frac{\exp(\text{sim}(v_i, t_i)/\tau)}{\sum_{j=1}^{N}\exp(\text{sim}(v_i, t_j)/\tau)}\right]$$

where $v_i$ and $t_i$ are the image and text embeddings for the $i$-th pair, $\text{sim}$ is cosine similarity, and $\tau$ is a learned temperature. This trains encoders so that the embedding of an image and its textual description are nearby in vector space, a capability essential for embodied agents and vision-language tasks.

Research on short-term plasticity simulation (UChicago, 2025) shows that AI networks can replicate the brain's “silent” memory phases via synaptic-like plasticity, maintaining information without persistent neural activity during initial sensory processing.

Sensory memory feeds directly into short-term/working memory, which holds attended information for active manipulation. Information that is not attended to in sensory memory decays and is lost, analogous to how an agent discards unattended tokens or image regions. Through attention and encoding, selected sensory information can eventually be consolidated into long-term memory via mechanisms like retrieval-augmented generation or fine-tuning (implicit memory).

In hierarchical memory architectures, sensory memory occupies the lowest tier, processing the highest bandwidth of raw data but retaining it for the shortest duration. The Hierarchical Cognitive Agent architecture (2025) implements this as a reactive layer handling sensor-to-actuator reflexes before information reaches deliberative or meta-cognitive layers.

• How to Add Memory to an Agent
• Memory Retention
• Implicit Memory
• Sensory Edge Agents
• Agent Memory Frameworks