How to Choose Chunk Size for RAG

Chunk size is the most underestimated hyperparameter in Retrieval-Augmented Generation. It silently determines what your LLM sees, how much retrieval costs, and how accurate answers are. This guide synthesizes published benchmark data and practical strategies for choosing optimal chunk sizes.¹⁾

• Too small — Loses context, fragments meaning, increases retrieval noise
• Too large — Dilutes relevance, wastes context window, reduces precision
• Just right — Balances semantic completeness with retrieval specificity

In NVIDIA's 2024 benchmark across 7 strategies and 5 datasets, wrong chunking strategy reduced recall by up to 9%.²⁾³⁾

Sources: Vecta 2026 (50 papers), NVIDIA 2024 benchmark, MDPI 2025 clinical study, AI21 Labs 2026⁴⁾

Overlap prevents information loss at chunk boundaries. The industry standard is 10-20% overlap.

Rule of thumb: Use 10% overlap for clean structured text, 20% for dense unstructured text, 0% for naturally bounded content (Q&A pairs, code functions).

Research from NVIDIA (2024) and AI21 Labs (2026) shows optimal chunk size depends on query type:

AI21 Labs demonstrated that multi-scale indexing (100, 200, 500 tokens with Reciprocal Rank Fusion) outperforms any single chunk size because different queries need different granularity.⁵⁾

from langchain.text_splitter import RecursiveCharacterTextSplitter
 
# Default: Recursive splitting at 512 tokens with 10% overlap
def create_splitter(content_type="general"):
    configs = {
        "general": {"chunk_size": 512, "chunk_overlap": 50},
        "code": {"chunk_size": 1000, "chunk_overlap": 0,
                 "separators": ["\nclass ", "\ndef ", "\n\n", "\n"]},
        "legal": {"chunk_size": 1024, "chunk_overlap": 200},
        "conversational": {"chunk_size": 256, "chunk_overlap": 50},
        "faq": {"chunk_size": 300, "chunk_overlap": 0,
                "separators": ["\n\n"]},
    }
    config = configs.get(content_type, configs["general"])
    return RecursiveCharacterTextSplitter(
        chunk_size=config["chunk_size"],
        chunk_overlap=config["chunk_overlap"],
        separators=config.get("separators", ["\n\n", "\n", ". ", " "]),
        length_function=len,
    )
 
# Multi-scale indexing for maximum accuracy
def multi_scale_index(documents, sizes=[100, 256, 512]):
    from collections import defaultdict
    all_chunks = defaultdict(list)
    for size in sizes:
        splitter = RecursiveCharacterTextSplitter(
            chunk_size=size, chunk_overlap=int(size * 0.1)
        )
        for doc in documents:
            chunks = splitter.split_text(doc)
            all_chunks[size].extend(chunks)
    return all_chunks  # Index each scale separately, fuse at query time
 
# Parent-child chunking
def parent_child_chunks(documents, parent_size=1024, child_size=256):
    parent_splitter = RecursiveCharacterTextSplitter(chunk_size=parent_size)
    child_splitter = RecursiveCharacterTextSplitter(chunk_size=child_size)
    index = []
    for doc in documents:
        parents = parent_splitter.split_text(doc)
        for parent in parents:
            children = child_splitter.split_text(parent)
            for child in children:
                index.append({"child": child, "parent": parent})
    return index  # Search on child, retrieve parent

Always benchmark chunk sizes on your own data. Key metrics:

• Retrieval Recall@k — Does the correct chunk appear in top k results?
• Answer Accuracy — Does the final LLM answer match ground truth?
• Latency — Time from query to retrieved chunks
• Cost — Embedding tokens + storage + retrieval compute

# Simple chunk size evaluation
def evaluate_chunk_sizes(documents, test_queries, ground_truth, sizes=[256, 512, 1024]):
    results = {}
    for size in sizes:
        splitter = RecursiveCharacterTextSplitter(
            chunk_size=size, chunk_overlap=int(size * 0.1)
        )
        chunks = splitter.split_documents(documents)
        vectorstore = build_vectorstore(chunks)
 
        correct = 0
        for query, expected in zip(test_queries, ground_truth):
            retrieved = vectorstore.similarity_search(query, k=5)
            if any(expected in r.page_content for r in retrieved):
                correct += 1
        results[size] = correct / len(test_queries)
    return results  # e.g. {256: 0.72, 512: 0.81, 1024: 0.76}

1. Start with 512 tokens, recursive splitting, 10% overlap. This is the best default backed by benchmarks.
2. Content type matters more than a magic number. Code, legal, and conversational content each need different strategies.
3. Multi-scale indexing gives the best accuracy (+7-13%) but at higher storage and complexity cost.
4. Parent-child chunking is the best balance of precision and context for complex documents.
5. Always benchmark on your data. Published numbers are starting points, not guarantees.
6. Query type affects optimal size: factoid queries want small chunks, analytical queries want large ones.

• When to Use RAG vs Fine-Tuning — Choose the right knowledge approach
• How to Structure System Prompts — Optimize prompts for RAG systems
• Single vs Multi-Agent Architectures — Agent design patterns