RAG in Production: What Actually Breaks (and How to Fix It)

A RAG demo takes a weekend. Load some PDFs into a vector database, wire up an LLM, ask it a question, watch it answer. Every team we work with has built that demo, and most of them come to us with the same complaint: it was impressive for two weeks, then users started reporting answers that were confidently wrong, subtly outdated, or just missing — and nobody could explain why. Having shipped retrieval-augmented systems over support knowledge bases, legal document sets, and internal engineering wikis, we can tell you the failure points are remarkably consistent. There are five of them.

1. Chunking Is Where Most RAG Systems Quietly Die

The default tutorial approach — split every document into fixed 1,000-character chunks with some overlap — is responsible for more bad RAG answers than any model limitation. Fixed-size chunking slices through the middle of tables, separates headings from the paragraphs they describe, and splits a policy's rule from its exceptions. The retriever then faithfully returns half a thought, and the LLM fills the gap by guessing. That guess is your hallucination, and it was manufactured at ingestion time, months before the user asked anything.

What we do instead:

• Structure-aware splitting first. Split on document structure — headings, sections, list boundaries — before falling back to size limits. For HTML and Markdown this is nearly free; for PDFs it is worth paying for a proper layout-aware parser, because PDF text extraction is where garbage enters the pipeline. If your source PDFs contain tables, test extraction on your ten ugliest documents before committing to any parser.
• Target 300-800 tokens per chunk, but let structure win. A 150-token chunk that is a complete, self-contained policy rule beats an 800-token chunk that merges three unrelated ones.
• Prepend context to every chunk. A chunk that reads "the limit is 50 requests per minute" is nearly useless without knowing which API, which tier, which version. We prepend a breadcrumb — document title, section path, and for tricky corpora a one-line LLM-generated summary of the parent section — to every chunk before embedding. This single change produced the largest retrieval-quality jump we have measured on real corpora.
• Retrieve small, feed big. Embed small chunks for retrieval precision, but hand the LLM the surrounding parent section. Small-to-big retrieval decouples "what is findable" from "what is readable," and both get better.

2. Retrieval Quality: Vector Search Alone Is Not Enough

Pure dense vector search fails on the exact queries your users care most about: part numbers, error codes, product names, acronyms, anything where the literal string matters. Embeddings capture meaning, and "ERR_CONN_RESET_4012" does not have meaning — it has identity. Meanwhile, keyword search fails on paraphrase. Production systems need both.

Our default retrieval stack, in order:

• Hybrid search: BM25 keyword retrieval and dense vector retrieval run in parallel, fused with reciprocal rank fusion. This is table stakes, not an optimisation. Postgres with pgvector plus its built-in full-text search covers this for most workloads without adding a dedicated vector database to your infrastructure — we resist introducing new datastores until the corpus or QPS demands it.
• Cross-encoder reranking: Retrieve 30-50 candidates cheaply, then rerank to a final 5-8 with a cross-encoder or a rerank API. Rerankers read the query and chunk together, which catches relevance that bi-encoder similarity fundamentally cannot. In our evals this typically converts several "right document was ranked 12th" failures into hits.
• Metadata filtering before semantics: If the user is on the billing page, filter to billing docs. If documents have effective dates, filter out superseded versions at query time. Half of what teams call hallucination is actually the retriever returning a true answer from an outdated document.
• Query rewriting, applied conservatively: A cheap LLM pass that expands acronyms and resolves pronouns from conversation history ("does it support SSO?" → "does Acme Enterprise support single sign-on?") pays for itself. We are more cautious with aggressive techniques like HyDE — they help on some corpora, hurt on others, and you cannot know which without evals, which brings us to the real point.

3. If You Do Not Have Evals, You Do Not Have a System

This is the strongest opinion we hold about RAG: a retrieval system without an evaluation set is not engineering, it is vibes. Every change — new chunk size, new embedding model, new prompt — either improves your system or degrades it, and without evals you find out from your users.

The good news is that a useful eval set is smaller than teams expect. Ours typically look like:

• 50-200 real questions, harvested from actual user queries and support tickets, not invented by the team. Invented questions are always too well-phrased.
• Labelled source chunks for each question, so you can measure retrieval in isolation: did the right chunk appear in the top-k? Recall@k on the retriever is the metric to watch, because no prompt engineering can recover from context that never arrived.
• Reference answers for end-to-end scoring, judged by an LLM with a rubric that scores faithfulness (is every claim supported by the retrieved context?) separately from helpfulness. LLM-as-judge has known biases — it rewards length and confidence — so we spot-audit 10% of judgments by hand until we trust the rubric, and re-audit whenever we change the judge model.
• A place in CI. The eval suite runs on every meaningful change to the pipeline, exactly like a test suite, with a tracked dashboard. Retrieval regressions are silent; this is the only alarm you get.

4. Hallucination Guards That Actually Work

You cannot prompt your way to zero hallucination, but you can engineer the system so that unsupported answers are rare, detectable, and cheap. The layers we ship, in rough order of value:

• Make refusal a first-class outcome. The prompt must give the model a dignified exit: if the retrieved context does not contain the answer, say so and route to a human or a search fallback. Then — critically — test the refusal path with eval questions whose answers are deliberately absent from the corpus. A system that answers 100% of questions is not impressive; it is broken.
• Require inline citations, then verify them. We have the model cite chunk IDs for each claim, and a post-processing step checks that every cited ID was actually in the context and drops any sentence citing a nonexistent source. Citations do double duty: they reduce fabrication during generation and give users a trust signal.
• Score retrieval confidence before generating. If the top reranked score is below a threshold you tuned on your eval set, do not generate a fluent guess — return the top documents as links with an honest "here is what I found." A slightly worse UX beats a confident fabrication in every domain we have worked in, and in regulated ones it is not negotiable.
• For high-stakes domains, add a verification pass. A second, cheap model call that checks each claim in the draft answer against the retrieved context catches a meaningful share of the remaining fabrications. It adds latency and cost, so we reserve it for legal, medical, and financial content — and skip it for internal wiki search.

5. Cost Control: Where the Money Actually Goes

Teams assume embedding costs will hurt them. They almost never do — embedding even a large corpus is usually a rounding error. The real spend is generation tokens, and it is dominated by context size: stuffing 8 chunks of 800 tokens into every request, times every message in a growing chat history, times thousands of users. We have audited RAG systems where 70% of the monthly bill was re-sending conversation history nobody needed.

What keeps bills sane:

• Rerank hard so you can send less. Five precisely reranked chunks outperform twelve mediocre ones and cost 60% less per request. Better retrieval is a cost feature, not just a quality feature.
• Use prompt caching for the static prefix. System prompt, instructions, and few-shot examples should be structured cache-first; on providers with prompt caching this cuts the effective cost of the static portion dramatically for chat workloads.
• Tier your models. Query rewriting, routing, and metadata extraction run on a small cheap model; only the final answer needs a frontier one. In most of our deployments the flagship model handles well under half of total LLM calls.
• Summarise or truncate chat history. Keep the last few turns verbatim, summarise the rest. Nobody's follow-up question needs turn 3 of 40 in full.

The Boring Parts That Bite You at Month Three

Finally, the operational failures that never appear in tutorials: documents change and your index does not (build incremental sync with change detection from day one, not a weekly full re-embed); permissions leak (if the source system has access controls, the retriever must enforce them per user, per query — retrieval that ignores ACLs is a data breach with extra steps); and embedding model upgrades silently require re-embedding the entire corpus, so version your index and budget for the migration.

None of this is glamorous. That is rather the point. The gap between a RAG demo and a RAG product is not a smarter model — it is chunking that respects your documents, retrieval that is measured, refusals that are tested, and a bill somebody actually watches. If you have a RAG system that users have stopped trusting, our AI engineering team can usually diagnose which of these five layers is failing within a week. Book a free consultation to talk it through.