Chunking
LLMs and embedding models have token limits. A 200-page manual can't be embedded or retrieved as one unit. Chunking is the highest-leverage RAG decision — bad chunking causes missed answers, fragmented context, and hallucinations even with perfect retrieval infrastructure.
Chunking is like cutting a textbook into flashcards. Cut too big and each card is overwhelming; cut too small and you lose the sentence that explains what 'it' refers to.
Visual Workflows
Start here — study each diagram, then use + / − to zoom if needed.
Scroll inside the frame · use + / − to zoom
Key Takeaways
- 1.Chunking splits documents into retrieval-sized segments that fit embedding models and LLM context windows while preserving enough semantic context for accurate answers. Chunking strategies: (1) Fixed-size — simple, fast, breaks mid-sentence.
- 2.(2) Recursive character — splits on paragraph → sentence → word boundaries. (3) Semantic — embed sentences, merge until similarity drops.
- 3.(4) Structure-aware — respect headers, tables, code blocks. Production parameters: chunk_size 256–1024 tokens, overlap 10–20% to preserve boundary context.
- 4.Parent-child chunking stores small chunks for retrieval but returns larger parent context to the LLM. Document-type matters: code uses AST-aware splitting, legal docs use section boundaries, chat logs use turn-based chunks.
Real Example
Scenario
An HR policy handbook uses header-based chunking: each section (e.g., 'Remote Work Policy §3.2') becomes one chunk with metadata {section_title, page}. Overlap of 50 tokens ensures questions spanning section boundaries still retrieve relevant context.
What you would do
In RAG Engineering, apply Chunking to this scenario: An HR policy handbook uses header-based chunking: each section (e. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.
Practice Task
Open the Code Walkthrough below and run it locally. Change one parameter related to Chunking (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.
Code Walkthrough
Highlighted lines show where Chunking happens in the code.
1from langchain_text_splitters import RecursiveCharacterTextSplitter # import dependencies2
3splitter = RecursiveCharacterTextSplitter( # key line for Chunking4 chunk_size=512, # key line for Chunking5 chunk_overlap=64, # key line for Chunking6 separators=["\n\n", "\n", ". ", " ", ""],7 length_function=len,8)9
10chunks = splitter.split_documents(documents) # key line for Chunking11
12# Parent-child pattern13parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200) # key line for Chunking14child_splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=50) # key line for Chunking15
16parents = parent_splitter.split_documents(documents) # key line for Chunking17children = []18for i, parent in enumerate(parents):19 for j, child in enumerate(child_splitter.split_documents([parent])): # key line for Chunking20 child.metadata["parent_id"] = f"parent_{i}"21 child.metadata["chunk_index"] = j # key line for Chunking22 children.append(child)Cheat Sheet
Quick recap
quick ref- •Default: 512 tokens, 64 overlap
- •RecursiveCharacterTextSplitter = go-to
- •Parent-child: retrieve small, generate large
- •Structure-aware for legal/technical docs
- •Benchmark before deploying
- •Bad chunking ≠ fixable by better embeddings
Common Mistakes
- ✕Using default 1000-char chunks without domain evaluation
- ✕Zero overlap — misses boundary-spanning queries
- ✕Splitting tables and code blocks mid-structure
- ✕Ignoring document hierarchy (headers, sections)
- ✕Same chunking strategy for all document types