Text chunking strategies

RAG Pipeline Overview

• RAG workflow: source document → chunk into pieces → user query → retrieve relevant chunks → add to prompt context
• Start simple, add complexity incrementally

Why Chunking Matters

• Chunking strategy directly impacts retrieval quality
• Poor chunking causes retrieval errors (e. g. , retrieving irrelevant sections due to keyword overlap)
• Example: word "bug" in medical section + "infection vectors" in software section = misleading matches

Three Chunking Strategies

Size-Based Chunking

• Divide into fixed-length character strings (easiest, most common in production)
• Problems: words cut mid-word, chunks lose context from headers/sections
• Solution: add overlap (include characters from neighboring chunks) to preserve context
• Trade-off: creates duplicate text but improves context per chunk

Structure-Based Chunking

• Split on document structure (headers, paragraphs, sections)
• Works well with markdown or clearly formatted documents
• Limitation: fails on unstructured documents (plain PDFs, unformatted text)
• Produces well-formed sections but requires structural guarantees

Semantic-Based Chunking

• Use NLP to measure relatedness between consecutive sentences/sections
• Group related sentences into chunks
• Most advanced; no fixed implementation; depends on use case

Practical Comparison

| Strategy | Chunk Size | Strengths | Weaknesses | |----------|-----------|----------|-----------| | Character (500 length, 150 overlap) | Small to medium | Universal fallback; works on code | Not semantically aware | | Sentence (5 sentences, 1 overlap) | Medium | Solid information per chunk; reliable | Regex fails on edge cases | | Section | Large | High-quality chunks; excellent for structured docs | First chunk often low-value; requires guaranteed structure |

Strategy Selection Guide

• Structured documents (markdown, clear sections) → chunk by section
• User-provided documents (unknown format) → chunk by sentence (most reliable)
• Code or unusual formats → chunk by character (fallback; sentence splitting fails on periods in code)

Key principle: Match strategy to document predictability and structure guarantees.

Takeaways

• Chunking strategy directly impacts retrieval quality; poor chunking causes irrelevant matches due to keyword overlap across unrelated sections
• Three main strategies exist: size-based (fixed character length with overlap), structure-based (split on headers/sections), and semantic-based (group by NLP relatedness)
• Match chunking strategy to document type: structured documents → section-based, unknown formats → sentence-based, code → character-based
• Size-based chunking with overlap preserves context but creates duplicate text; structure-based produces high-quality chunks but requires guaranteed formatting
• Start simple with character or sentence chunking, then add complexity (semantic chunking) only if retrieval quality demands it