Document Chunking Strategies: Recursive vs Semantic vs Agentic in Practice

RAG's "upstream" determines the "downstream" ceiling. No matter how sophisticated the embedding and Reranker stack, if chunking destroys semantic boundaries, retrieval quality will never recover. This article provides a production-engineering comparison of three mainstream chunking strategies -- fixed-size, recursive, and semantic -- plus the emerging Agentic Chunking paradigm, with quantitative selection guidance.

Why Chunking Is RAG's Hidden Bottleneck

The standard RAG pipeline is: document -> chunk -> embed -> retrieve -> answer. Chunking looks like the simplest step, but it directly determines the input quality of every downstream component.

Most common failure modes:

• Critical information split across chunks: a table footer lands in chunk N while the header sits in chunk N-1, and the model never sees the table as a unit
• Semantic units broken mid-thought: a complete argument is sliced in half, and both halves lose context
• Noise pollution: a single chunk covers three unrelated topics, and embedding maps it to a fuzzy vector
• Over-chunking: a 1000-token document sliced into 20 pieces of 50 tokens each, so retrieval only matches half-sentences

Worse, these errors are silent. Your metrics may show RAG accuracy stuck at 60%, and you assume the embedding or LLM is at fault, spend weeks tuning embeddings, and only later discover the real culprit was the chunking strategy.

Strategy 1: Fixed-Size Chunking

The simplest approach: cut every N tokens, with N typically set to 256, 512, or 1024.

from langchain.text_splitter import CharacterTextSplitter

text_splitter = CharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separator="\n\n",
)

chunks = text_splitter.split_text(document)

Strengths:

• Simplest implementation, a few dozen lines
• Uniform chunk size keeps embedding quality stable
• Predictable performance: chunks per document = total length / chunk_size

Weaknesses:

• Hard cuts ignore semantic boundaries
• Table headers, code blocks, and list items often land in different chunks
• Conceptual continuity is broken: paragraphs three and four of an argument are split into separate chunks, both losing context

Best for:

• Internal knowledge bases with structured, uniform-format technical documents
• Minimal prototype phase; ship first, optimize later

Strategy 2: Recursive Chunking

LangChain's default-recommended strategy. First try paragraph-level splits; if a paragraph is too long, fall back to sentence-level; if a sentence is too long, fall back to word-level -- a cascading fallback.

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", "! ", "? ", " ", ""],
    length_function=len,
)

chunks = text_splitter.split_text(document)

Key design points:

• Separator priority: \n\n > \n > . > > "" -- prefer the largest semantic boundary
• Multilingual separators: the example above supports both English and Chinese punctuation
• chunk_size unit: defaults to character count, not tokens. For token-based sizing, customize length_function with tiktoken or similar

Strengths:

• Balances semantic integrity with size control
• Paragraph and sentence structure preserved
• Performance comparable to fixed-size, minimal overhead

Weaknesses:

• Cannot handle overly long paragraphs (the whole paragraph becomes one chunk)
• No special treatment for tables, code blocks, or list items
• Chunk boundaries are still heuristic

Best for:

• Most general-purpose documents (product docs, blog posts, FAQ)
• Projects that do not want to spend much effort tuning chunking

Strategy 3: Semantic Chunking

Dynamically cuts based on embedding similarity: when the similarity between adjacent sentences drops below a threshold, treat that as a "concept switch" point.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai.embeddings import OpenAIEmbeddings

text_splitter = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=80,
)

chunks = text_splitter.split_text(document)

Implementation:

1. Split the document by sentence
2. Compute cosine similarity of adjacent sentence embeddings
3. Find the "cliffs" in the similarity series (breakpoints)
4. Cut at each breakpoint

Strengths:

• Chunk boundaries align with semantic boundaries
• Conceptually continuous paragraphs merge into the same chunk
• Chapter changes and topic shifts are correctly identified

Weaknesses:

• High computational cost (every sentence goes through embedding)
• Threshold is hard to tune
• Short documents (<10 sentences) underperform
• Sensitive to embedding model quality

Best for:

• High-quality, deep technical documents
• Recall-sensitive RAG systems
• Projects that can absorb the extra embedding cost

Strategy 4: Agentic Chunking (Latest Paradigm)

A 2024-era method where the LLM decides how to chunk. An LLM reads the entire document and outputs "which paragraphs should merge into one chunk":

from openai import OpenAI

client = OpenAI()

def agentic_chunk(document: str, max_chunk_size: int = 800) -> list:
    prompt = f"""You are a document chunking specialist. Your job is to split the
following document into semantically coherent chunks for a RAG system.

Rules:
1. Each chunk should cover ONE concept or topic
2. Do not split mid-sentence
3. Do not split tables, code blocks, or list items
4. If a paragraph is self-contained, keep it as one chunk
5. Maximum chunk size: approximately {max_chunk_size} characters
6. Output the chunks as a JSON array of strings

Document:
{document}

Output (JSON array of strings):"""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )

    import json
    result = json.loads(response.choices[0].message.content)
    return result["chunks"]

Strengths:

• Truly "understands" document structure -- tables, code blocks, list items stay intact
• Combines context to judge boundaries
• Suited to complex structures (research papers with figures, legal contracts)

Weaknesses:

• Expensive (one LLM call per document)
• Slow (one LLM call is 2-5 seconds)
• Not reproducible (LLM output has randomness)
• Hard to parallelize (needs the full document context)

Best for:

• Small but high-value document collections (legal contracts, medical guidelines)
• Offline batch processing, not real-time
• Budget-permitting scenarios

Comparing the Four Strategies

Dimension	Fixed	Recursive	Semantic	Agentic
Speed	Fastest	Fast	Medium	Slow
Cost	Near zero	Near zero	Medium (per-sentence embedding)	High (per-document LLM)
Semantic boundaries	Poor	Medium	Good	Excellent
Table/code handling	Poor	Poor	Medium	Excellent
Tuning difficulty	Low	Low	Medium	High
Reproducibility	Full	Full	Full	Weak

Chonkie: The De Facto Chunking Library

If you do not want to build chunking yourself, Chonkie is currently the fastest option:

from chonkie import RecursiveChunker, SemanticChunker, TokenChunker

chunker = TokenChunker(chunk_size=512, chunk_overlap=50)
chunks = chunker.chunk(text)

chunker = RecursiveChunker(chunk_size=512)
chunks = chunker.chunk(text)

from sentence_transformers import SentenceTransformer
embedder = SentenceTransformer("BAAI/bge-m3")
chunker = SemanticChunker(embedder=embedder, chunk_size=512)
chunks = chunker.chunk(text)

for chunk in chunks:
    print(f"Text: {chunk.text[:80]}")
    print(f"Tokens: {chunk.token_count}")
    print(f"Start: {chunk.start_index}, End: {chunk.end_index}")
    print("---")

Chonkie's advantage over LangChain: 5-10x faster (Rust core), more lightweight, more modern API.

LlamaIndex SentenceSplitter

LlamaIndex provides fine-grained chunking control:

from llama_index.core.node_parser import SentenceSplitter, SemanticSplitterNodeParser

parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)
nodes = parser.get_nodes_from_documents(documents)

from llama_index.embeddings.openai import OpenAIEmbedding
embed_model = OpenAIEmbedding()
parser = SemanticSplitterNodeParser(
    embed_model=embed_model,
    breakpoint_percentile_threshold=80,
)
nodes = parser.get_nodes_from_documents(documents)

Selection Decision

Start with Recursive by default. Most projects do not need chunking perfection, and Recursive handles 80% of cases adequately. Spend your energy on embedding selection and retrieval strategy.

When to upgrade to Semantic:

• Recursive chunks frequently break concepts in half
• Recall@10 is clearly below expectations
• Documents switch topics often (technical manuals, policy docs)

When to upgrade to Agentic:

• Documents are highly structured (contracts, papers)
• Recall quality directly drives business outcomes (healthcare, legal)
• Volume is small (<1000 documents per day) and LLM cost is acceptable

When to stay with Fixed:

• Minimal prototype phase
• Document collection is highly homogeneous (every document has the same structure)

Offline Evaluation

Do not pick a chunking strategy by gut feel -- quantify with an evaluation set:

from langchain_community.embeddings import HuggingFaceEmbeddings
from qdrant_client import QdrantClient

embedder = HuggingFaceEmbeddings(model_name="BAAI/bge-m3")
client = QdrantClient(":memory:")

def evaluate_chunking(strategy, eval_set):
    metrics = {"recall_at_5": 0, "mrr": 0}
    for item in eval_set:
        chunks = strategy(item["document"])
        client.add("eval", chunks, item["document_id"])
        results = client.search(item["query"], k=5)
        result_ids = [r.id for r in results]
        if any(rid in item["relevant_doc_ids"] for rid in result_ids):
            metrics["recall_at_5"] += 1
        for i, rid in enumerate(result_ids):
            if rid in item["relevant_doc_ids"]:
                metrics["mrr"] += 1 / (i + 1)
                break
    return {k: v / len(eval_set) for k, v in metrics.items()}

strategies = {
    "fixed_500": lambda doc: fixed_chunk(doc, 500),
    "recursive_500": lambda doc: recursive_chunk(doc, 500),
    "semantic_500": lambda doc: semantic_chunk(doc, 500),
}

for name, fn in strategies.items():
    metrics = evaluate_chunking(fn, eval_set)
    print(f"{name}: Recall@5={metrics['recall_at_5']:.3f}, MRR={metrics['mrr']:.3f}")

Typical results:

• Fixed: Recall@5 = 0.55
• Recursive: Recall@5 = 0.68
• Semantic: Recall@5 = 0.78
• Agentic: Recall@5 = 0.85 (10x the cost)

Before each chunking upgrade, check the evaluation set for measurable improvement.

Implementation Path

Week 1: Start with Recursive chunking (chunk_size=512, overlap=50); ship a basic RAG pipeline. Week 2: Build a 100-query evaluation set; quantify baseline recall@10. Week 3: Try Semantic chunking; compare metrics. Week 4: If Semantic improves metrics by >10%, switch; otherwise optimize embedding and retrieval. Week 5: Apply Agentic chunking to high-value documents (policies, product manuals); build a "high-quality chunk" pipeline. Week 6: Add offline chunking quality evaluation to CI.

Summary

Chunking is the foundation of a RAG system. Picking the wrong strategy caps everything downstream. Start with Recursive, build your evaluation set, and let data drive upgrades to Semantic or Agentic. Do not tune chunk_size by feel -- treat every chunking parameter change as an A/B experiment and let evaluation data decide.

Reference tools: Chonkie (fastest Rust-based chunking library), LlamaIndex (fine-grained chunking node parsers), LangChain (Recursive / Semantic splitters), txtai (end-to-end RAG with built-in chunking), and EmbedAnything (multimodal embedding framework with custom chunking) cover the core nodes of the chunking toolchain.