A practical guide to RAG that actually works in production

Most RAG tutorials stop at "embed your documents, retrieve the top-k, stuff them in the prompt." That version works in a demo and degrades quietly in production, usually in ways that don't show up until real users start asking real questions.

Chunking is the decision that matters most

Chunk size and overlap determine what the retriever is even capable of finding. Chunks that are too large dilute the embedding with irrelevant surrounding text, so a specific question retrieves a document that only tangentially contains the answer. Chunks that are too small lose the surrounding context a model needs to interpret them correctly. Splitting on semantic boundaries — sections, paragraphs — rather than a fixed token count, and keeping a modest overlap between adjacent chunks, consistently outperforms naive fixed-size splitting once documents get past a page or two.

Evaluate retrieval, not just generation

It's tempting to judge a RAG system entirely on whether the final answer sounds right. That hides the actual failure: if the retriever didn't surface the relevant chunk, the model is guessing or hallucinating regardless of how fluent the answer reads. Tracking retrieval metrics separately — did the correct chunk appear in the top-k at all — makes it possible to tell whether a bad answer is a retrieval problem or a generation problem, which need completely different fixes.

Failure modes that only appear at scale

Two problems consistently show up only after a knowledge base grows past what fit in early testing. First, near-duplicate documents (multiple versions of the same policy, for instance) start competing for the same top-k slots, crowding out genuinely different relevant content. Second, embedding drift: if the embedding model or chunking strategy changes after initial indexing, old and new chunks can end up in inconsistent regions of the vector space, quietly degrading retrieval quality until the whole index is rebuilt. Planning for periodic re-indexing, rather than treating the index as a one-time setup step, avoids both.