Embeddings

Embeddings are dense vector representations that capture semantic meaning of text, images, and other data in continuous high-dimensional space. They are the foundation of semantic search, retrieval-augmented generation, clustering, and classification in AI agent systems. Choosing the right embedding model directly impacts retrieval quality, agent accuracy, and operational costs.

Embedding models transform input data into fixed-size numerical vectors $\mathbf{x} \in \mathbb{R}^d$ where semantically similar items are positioned close together in vector space. The similarity between two embeddings is typically measured using cosine similarity:

$$\text{sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{||\mathbf{a}|| \cdot ||\mathbf{b}||}$$

This enables:

• Semantic search — Find relevant documents by meaning rather than keyword matching
• RAG retrieval — Power the retrieval stage of retrieval-augmented generation
• Clustering — Group similar items for analysis and organization
• Classification — Use vector proximity for categorization tasks
• Anomaly detection — Identify outliers in embedding space

import numpy as np
from openai import OpenAI
 
client = OpenAI()
 
def embed_texts(texts: list[str], model="text-embedding-3-large") -> np.ndarray:
    """Embed a batch of texts using OpenAI API."""
    response = client.embeddings.create(input=texts, model=model)
    return np.array([item.embedding for item in response.data])
 
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
 
# Embed documents and query
documents = [
    "RAG combines retrieval with generation for grounded responses",
    "Fine-tuning adapts model weights on domain-specific data",
    "Prompt engineering designs inputs to guide model behavior"
]
doc_embeddings = embed_texts(documents)
query_embedding = embed_texts(["How do I ground agent responses in facts?"])[0]
 
# Find most relevant document
similarities = [cosine_similarity(query_embedding, doc) for doc in doc_embeddings]
best_match = documents[np.argmax(similarities)]
print(f"Most relevant: {best_match}")  # RAG document

The number of dimensions $d$ in an embedding affects the trade-off between semantic precision and computational cost:

• Higher dimensions ($d = 2048$-$3072$) — Capture more nuanced semantic distinctions but require more storage, memory, and compute for similarity search
• Medium dimensions ($d = 768$-$1024$) — The sweet spot for most agent applications, balancing quality and efficiency
• Lower dimensions ($d = 256$-$512$) — Suitable for large-scale applications where speed and cost are prioritized over precision

Practical guidance: Start with medium dimensions (768-1024). Only scale up if retrieval quality benchmarks show meaningful improvement. Use dimensionality reduction (PCA, Matryoshka embeddings) to test whether lower dimensions maintain acceptable recall.

Multi-modal embedding models project different data types (text, images, audio) into a shared vector space $\mathbb{R}^d$, enabling cross-modal search:

• CLIP and variants — Align image and text embeddings for visual search
• ImageBind (Meta) — Unified embeddings across text, image, audio, video, depth, and thermal
• Jina CLIP v2 — Open-source text-image embeddings with multilingual support
• Cohere Embed v4 — Handles text, images, and mixed documents in a single model

Cross-modal search enables agents to find relevant images from text queries or match text descriptions to visual content.

The Massive Text Embedding Benchmark (MTEB) evaluates embedding models across 56+ tasks spanning retrieval, classification, clustering, reranking, and semantic similarity. Key findings for 2026:

• Top proprietary: OpenAI text-embedding-3-large (~70%), Voyage 3 (~69%)
• Top open-source: BGE-M3 (~68%), Jina v3 (~67%), Nomic (~66%)
• Real-world gap: Models that score well on MTEB may underperform on specific domains — always evaluate on your own data
• Non-English: Cohere and BGE excel on multilingual benchmarks

• Task fit — Retrieval-optimized models (BGE, Voyage) for RAG; general models (OpenAI) for broad applications
• Cost/latency — Open-source models for high-volume agents; proprietary for peak accuracy
• Domain/language — Multilingual models (Cohere, BGE-M3) for global agents; domain-specific fine-tuned models for specialized retrieval
• Infrastructure — Consider whether you need API-based (OpenAI, Cohere) or self-hosted (BGE, Nomic) models
• Dimensionality — Higher dimensions for precision-critical applications; lower for scale and speed

Embeddings are stored and queried in vector databases using approximate nearest neighbor (ANN) algorithms. The most common distance metrics are:

• Cosine similarity: $\text{sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{||\mathbf{a}|| \cdot ||\mathbf{b}||}$ — measures angular closeness, invariant to magnitude
• Euclidean distance (L2): $d(\mathbf{a}, \mathbf{b}) = ||\mathbf{a} - \mathbf{b}||_2 = \sqrt{\sum_{i=1}^{d}(a_i - b_i)^2}$ — measures absolute distance in vector space
• Dot product: $\langle \mathbf{a}, \mathbf{b} \rangle = \sum_{i=1}^{d} a_i b_i$ — used for MIPS when magnitudes carry meaning

Key ANN implementations include:

• HNSW (Hierarchical Navigable Small World) — Best recall/speed trade-off, used by most vector databases
• IVF (Inverted File Index) — Good for very large collections with acceptable recall trade-offs
• FAISS — Meta's library for efficient similarity search at scale, supports GPU acceleration

• MTEB Leaderboard
• OpenAI Embeddings Guide
• Best Embedding Models for RAG 2026

• Retrieval Augmented Generation — RAG systems powered by embeddings
• Knowledge Graphs — Graph representations complementing vector search
• Agent Memory Frameworks — Memory systems using embedding-based retrieval
• Fine-Tuning Agents — Fine-tuning embedding models for domain-specific tasks