FajarA retrieval-augmented generation (RAG) demo takes a weekend. Load some PDFs into a vector database, connect a language model, ask a question, watch it answer. The pattern itself is well established: the original RAG paper dates from 2020. What the demo hides is how many quiet decisions sit between “it answered my test question” and “users trust it with real ones.” The usual story runs like this: the system impresses for two weeks, then people start reporting answers that are confidently wrong, slightly outdated, or missing, and nobody can say why.
The causes are rarely mysterious. They cluster in five places: chunking, retrieval, evaluation, answer guards, and cost. This guide works through each one, with the published research and vendor documentation behind the advice, and ends with the operational problems that tend to show up around month three.
Chunking Is Where Answers Go Wrong First
The tutorial default is to split every document into fixed 1,000-character pieces with some overlap. It is simple, and it does a lot of damage. Fixed-size splitting cuts through the middle of tables, separates headings from the paragraphs they describe, and parts a policy’s rule from its exceptions. The retriever then returns half a thought, and the model fills the gap by guessing. That guess is the hallucination your user sees, and it was created at ingestion time, long before anyone asked a question.
Split on Structure Before Size
Split on headings, sections and list boundaries first, and fall back to size limits only inside a long section. For HTML and Markdown this is nearly free. PDFs are harder, because text extraction is where garbage enters the pipeline: multi-column layouts, tables, headers and footers repeated on every page. Before you commit to a parser, run it on your ten ugliest documents and read the output yourself.
As a starting range, aim for roughly 300 to 800 tokens per chunk, and let structure win when the two conflict. A 150-token chunk that holds one complete, self-contained rule is more useful than an 800-token chunk that merges three unrelated ones.
Give Every Chunk Its Context
A chunk that reads “the limit is 50 requests per minute” is close to useless if nothing says which API, which plan or which version. Prepend a breadcrumb (document title and section path) to each chunk before embedding it. For difficult corpora, add a one-line, model-written description of where the chunk sits in the document.
Anthropic published results for this technique under the name contextual retrieval. In their tests, contextual embeddings alone reduced the top-20 retrieval failure rate by 35%. Combining contextual embeddings with contextual BM25 reduced it by 49%, and adding a reranker brought the reduction to 67%. Your corpus will differ, but the direction is consistent with the reasoning: a chunk that carries its own context is easier to find.
Retrieve Small, Read Big
Embed small chunks so that matching is precise, then hand the model the parent section around the hit. This separates what is findable from what is readable, and both improve. It costs one extra lookup per retrieved chunk and a parent ID stored in the chunk’s metadata.
Vector Search Alone Is Not Enough
Dense vector search struggles on exactly the queries many users care most about: part numbers, error codes, product names, acronyms. Embeddings capture meaning, and a string like ERR_CONN_RESET_4012 has identity rather than meaning. Keyword search has the opposite weakness: it misses paraphrase. A production system needs both.
Hybrid Search With Rank Fusion
Run BM25-style keyword retrieval and dense retrieval in parallel, then merge the two ranked lists. The standard merging method is reciprocal rank fusion, introduced by Cormack, Clarke and Büttcher in 2009. It needs no score calibration between the two systems, because it uses only each document’s rank:
def rrf(result_lists, k=60):
scores = {}
for results in result_lists:
for rank, doc_id in enumerate(results, start=1):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)You may not need a new datastore for any of this. The pgvector documentation describes combining vector search with Postgres full-text search for hybrid retrieval. One caveat: Postgres’s built-in ranking functions are not BM25, so treat them as a reasonable keyword signal and not an exact equivalent. We prefer to stay on Postgres until corpus size or query volume makes a dedicated search engine worth the extra operations work.
Rerank the Candidates
Retrieve 30 to 50 candidates cheaply, then rerank them with a cross-encoder or a hosted rerank API such as Cohere Rerank. A bi-encoder embeds the query and the document separately and compares the vectors. A reranker reads the two together, which lets it catch relevance that vector similarity misses. The typical win is the case where the right passage was retrieved but ranked twelfth: a reranker moves it into the set the model reads.
Filter on Metadata First
If the user is on the billing page, restrict the search to billing documents. If documents carry effective dates, exclude superseded versions at query time. A large share of what teams call hallucination is the retriever returning a true statement from a document that is no longer current. The model did its job. The index did not.
Rewrite Queries, Carefully
A small model pass that expands acronyms and resolves pronouns from the conversation (“does it support SSO?” becomes “does the Enterprise plan support single sign-on?”) is cheap and usually helps. More aggressive methods deserve more caution. HyDE, which embeds a model-written hypothetical answer in place of the query, improves results on some corpora and hurts on others. You cannot tell which group you are in without measuring, and that is the subject of the next section.
Without an Eval Set, You Are Guessing
This is the strongest opinion we hold about RAG. Every change (a new chunk size, a new embedding model, a reworded prompt) makes the system better or worse, and without an evaluation set the people who find out are your users.
A useful eval set is smaller than most teams expect. Between 50 and 200 questions is enough to start, provided they are real: taken from search logs, support tickets and chat transcripts. Questions invented by the team are too well phrased, and they use the same vocabulary as the documents.
Measure Retrieval on Its Own
For each question, label the chunks that contain the answer. That lets you score the retriever in isolation: did a correct chunk appear in the top k? Recall at k is the number to watch, because no prompt can recover from context that did not arrive. When an answer is wrong, this split also tells you where to look. If the right chunk was retrieved, the problem is in generation. If it was not, leave the prompt alone and fix retrieval.
Score Answers for Faithfulness
For end-to-end scoring, write reference answers and have a strong model grade against a rubric that separates faithfulness (is each claim supported by the retrieved context?) from helpfulness.
Model judges have documented weaknesses. The paper Judging LLM-as-a-Judge names position bias, verbosity bias and self-enhancement bias, while also finding that a strong judge can agree with human raters more than 80% of the time, about as often as humans agree with each other. The practical reading: use a judge, and check it. Grade a sample by hand, compare, tighten the rubric where they disagree, and repeat the check whenever you change the judge model.
Run It Like a Test Suite
Put the eval in CI and run it on each meaningful change to the pipeline, with results tracked over time. Retrieval regressions make no noise. Nothing crashes and no error is logged: answers simply get a little worse. The eval run is the only alarm you have.
Guards Against Unsupported Answers
Prompting alone will not bring unsupported answers to zero. What you can do is make them rare, detectable and cheap to handle. Four layers, in rough order of value:
- Make refusal a normal outcome. The prompt should tell the model what to do when the context does not contain the answer: say so, and route to a person or to plain search results. Then test that path with eval questions whose answers are deliberately absent from the corpus. A system that answers every question put to it is not impressive. It is inventing some of those answers.
- Require citations, then verify them. Have the model cite chunk IDs for its claims, and check in code that each cited ID was present in the context. Remove or flag sentences that cite something that was not there. Citations also give users a way to check the answer themselves.
- Check retrieval confidence before generating. If the top reranked score falls below a threshold tuned on your eval set, skip the fluent answer and return the best documents as links, with a plain “here is what we found.” The experience is slightly worse. A confident fabrication is much worse, and in regulated fields it is not acceptable.
- Add a verification pass where the stakes are high. A second, cheaper model call can check each claim in the draft against the retrieved context. It adds latency and cost, so it suits legal, medical and financial content better than internal wiki search.
For customer-facing assistants, these guards sit beside escalation rules and spending limits. Our AI chatbot launch checklist covers that side.
Where the Money Goes
Teams often expect embedding to be the expensive part. It is usually minor: a corpus is embedded once and then updated incrementally. The recurring cost is generation, and input tokens dominate it. Suppose each request carries eight chunks of 800 tokens plus a chat history that grows with every turn. By turn ten, most of what you are paying for is text the model has already seen.
Send Less, but Measure It
Good reranking lets you send fewer chunks for the same answer quality, which makes better retrieval a cost feature as well. Do not assume that fewer is better, though. In Anthropic’s contextual retrieval tests, passing 20 chunks outperformed 5 or 10. In the other direction, Lost in the Middle showed that models use information at the start and end of a long context better than information buried in the middle. Both findings point the same way: choose the chunk count on your own eval set, and place the strongest chunks first.
Cache the Static Prefix
Put the system prompt, instructions and examples at the start of the request, where provider-side prompt caching can reuse them. Anthropic’s prompt caching documentation prices cache reads at a fraction of normal input tokens, and other providers offer similar discounts. Caching matches on the prefix, so anything that changes per request (retrieved chunks, the user’s question) belongs after the stable part.
Tier the Models and Trim the History
Query rewriting, routing and metadata extraction run well on a small model. Only the final answer may need a larger one. Our guide to choosing between GPT, Claude and Gemini covers how to make that call. For chat history, keep the last few turns verbatim and summarize the rest. A follow-up question rarely needs turn 3 of 40 in full.
The Operational Problems That Show Up at Month Three
Some failures do not appear in tutorials because tutorials end before they happen.
Documents Change and the Index Does Not
Build incremental sync with change detection from the start: a content hash per document, re-embedding only what changed, and deletion of chunks whose source was removed. A weekly full rebuild means answers can be up to a week stale, and deleted documents keep being quoted until the next run.
Permissions Leak
If the source system has access controls, the retriever has to enforce them for each user on each query, as a filter applied before ranking. Asking the model to withhold restricted content does not work, because by then the content is already in the prompt. The OWASP Top 10 for LLM Applications lists vector and embedding weaknesses as a risk category of its own, and cross-user leakage through retrieval is one of the examples it describes.
Embedding Model Upgrades Mean Re-Embedding
Vectors from two different embedding models are not comparable, so changing the model means re-embedding the whole corpus. Version your index, store the model name alongside each vector, and plan the migration as a parallel build followed by a switch, with the eval set deciding whether the new index is better.
A Diagnostic Order for a System Users Have Stopped Trusting
If you have inherited a RAG system with a poor reputation, work through it in this order:
- Collect 50 real questions that failed, and label the chunks that should have answered them.
- Measure recall at k. If the right chunk is missing from the results, look at chunking, hybrid search and metadata filters.
- If the chunk was retrieved but ranked low, add or tune a reranker.
- If the chunk reached the model and the answer was still wrong, work on the prompt, citations and the refusal path.
- Check freshness and permissions separately. Those failures look like model errors and are not.
None of this is glamorous work. The distance between a RAG demo and a RAG product is rarely a smarter model. It is chunking that respects your documents, retrieval that is measured, refusals that are tested, and a bill that somebody watches. If you would like a second pair of eyes on a system like this, our AI and automation team builds and reviews retrieval systems, and you can request a call to talk it through.
