Chunk boundaries decide what can be found
Documents are split into chunks before being embedded and indexed. The chunk is the unit of retrieval, so the split decides what can ever be returned together. A fact split across two chunks is a fact the retriever can only half find.
Why it is like this
Embeddings represent a fixed span of text as one vector. Too large and the vector means everything and matches nothing; too small and the context needed to understand the passage is in a different chunk.
Not to be confused with
A preprocessing detail. It is one of the highest-leverage decisions in the system, and the one most often made by accepting a library default.
In practice
A table split from its heading, or a clause split from the article number that gives it meaning, will retrieve badly no matter what you do downstream.
Source: Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401) — §2. Read on .
The OWASP Top 10 for LLM Applications is © the OWASP Foundation and published under a Creative Commons licence; NIST AI 600-1 is a US Government work in the public domain. These cards are independent commentary written in our own words. ytyk.org is not affiliated with or endorsed by the OWASP Foundation or NIST.
Odd one out Check yourself
Three of these are real consequences of chunk size. Which is not?
Show the answer
Chunk size determines the model's context window
The context window is a property of the model. Chunking decides what goes into it, not how big it is.