RAG for enterprise knowledge base - what actually breaks at 500K records

RAG for enterprise knowledge base search fails in a predictable order

The standard demo is five steps: split the documents, embed the chunks, store the vectors, retrieve the closest matches, hand them to a model. On a small clean corpus that works, and the reason is not the pipeline. It is that a hundred documents rarely hold two things that answer the same question differently, so almost any retrieval finds the one passage that matters.

We run a system like this over a knowledge base of 500K+ records, and the useful part was never the happy path. It was the order things went wrong in as the corpus grew. Chunks lose their meaning first, then ranking degrades quietly, because it keeps returning something and nothing looks broken. Entity ambiguity and staleness come after that. Trust goes last, and it is the only item on the list that takes months to get back.

Not one of the fixes below is a better model. They are all changes to retrieval, to ingestion, and to the checking wrapped around both, which is the unglamorous half nobody puts in a demo. One question does sit earlier than all of this: whether the process was worth automating in the first place. This article assumes you already got a yes there and are now dealing with the engineering.

The chunk is the first thing to break

Chunking is where a document stops being a document. A paragraph that reads "the limit is 30 days from delivery" makes sense on its page and means nothing on its own: which product, which region, which version of the terms. On a small corpus the neighbouring chunks give it away. On a large one there are four hundred chunks about 30-day limits and nothing to tell them apart.

Anthropic published numbers on exactly this in their write-up on contextual retrieval. Prepending a short generated description of where each chunk sits in its document cut the top-20 retrieval failure rate from 5.7% to 3.7%. Running BM25 alongside the embeddings took it to 2.9%, and reranking the merged results took it to 1.9%. Those are their figures on their evaluation set rather than a law of nature, but the shape matches what we see: the cheap structural fixes move the needle before anything clever does.

Structured data breaks chunking harder than prose does. The base we work with holds item IDs, zone drop tables, respawn rates and patch history, each in its own schema, and splitting a drop table by character count destroys the row that made it useful. Chunk along the document's own structure - a table row, a clause, a section - and carry metadata with every chunk: source document, section, version, effective date. All four fields come back in the parts of this article that hurt more.

Semantic similarity is not the same as being useful

A naive embedding search returns the text most similar to the question, which is not the text that answers it. Ask our assistant for the best way to farm a particular cloth item and a plain vector search surfaces everything that mentions cloth farming, while missing the one drop table that decides the answer. The results are not wrong, exactly. They are on topic and useless, which is worse, because on-topic results look like a working system.

Xenoss makes the same point structurally in their piece on enterprise RAG architecture: vanilla RAG treats all documents equally, with no mechanism to prioritise a regulatory source over a supplementary one, and a fixed top-k strategy quietly equates semantic similarity with relevance. In a company base that means the onboarding deck and the signed policy compete on even terms, and the deck often wins, because it is written in the same conversational register as the question.

Dense vectors are also weakest exactly where business queries are strongest: exact identifiers. A part number, an error code, a contract reference, two product names that differ by one token - those are lexical problems, and BM25 handles them for almost nothing. So the retrieval in our base is hybrid: dense and BM25 combined, then reranked over the merged set, so an exact-name hit and a semantic hit both get a shot at the final context. A reranker is not free, since it puts an extra model call in front of every answer, and it is still the cheapest way we know to fix "on topic but useless".

Resolve the entity before you search for anything

The change that moved the most on our side was refusing to search first. Retrieval runs in two stages. Stage one parses the query for names - items, zones, mechanics - and maps them to canonical IDs through a structured lookup against real tables. Stage two runs semantic search over the chunked content, filtered by those IDs as metadata constraints.

A question about farming a specific item resolves the item first, then retrieves only the guides and drop tables that reference that exact item, instead of everything that happens to mention cloth. The search space collapses from the whole corpus to the handful of documents actually about the thing you asked about, and ranking gets easier because there is far less to rank.

Your entities are not items and zones. They are customers, SKUs, contract numbers, machines, policies, employees, and you already keep them in tables. That is the point: the resolution step is a database lookup rather than a model call, which makes it the cheapest accuracy in the pipeline. It also removes the near-miss class of failure, where a question about order 4471 gets answered from order 4417 because the two embed almost identically.

Sometimes no single passage holds the answer

Some questions have no passage that answers them. The example that shaped our design is mundane on the surface: the best way to farm a particular cloth requires cross-referencing item IDs, zone drop tables, mob respawn rates and patch history, all sitting in different schemas. No chunk contains the answer, because the answer does not exist until four sources are joined.

Company bases are full of the same shape. Can we ship this configuration to a customer in Germany under their current contract? That is a product spec, an export classification and a contract clause, in three documents that were written by three teams who never read each other's work. Top-k retrieval returns the closest one and the model writes a confident answer from a third of the picture.

Three moves help, and they get expensive in that order. Decompose the query into sub-questions and retrieve for each one, which is mostly a prompt change. Route the parts that are really table lookups to the table, instead of asking a vector index to remember what a JOIN is. Only then look at graph approaches, and go in knowing the bill: Xenoss reports an EU AI Act implementation where GraphRAG needed 20x more tokens than the vanilla equivalent, with processing time roughly 2x. Sometimes the answer is not in documents at all but in a system that knows the current state, and at that point retrieval stops being search and becomes tool calls, which is what connecting AI agents to live data means and brings a different set of problems.

Make the model prove every number it states

The failure that costs the most is a number the model produced from memory. Ask the assistant for a drop rate and an ungrounded version states a precise percentage, confidently and precisely wrong. Precision is what disarms the reader. Almost nobody re-checks a figure carrying two decimal places, which is how the wrong ones survive review and end up in a report.

So every numerical claim in the answer - a percentage, a timer, a required reputation level - is validated against the source chunk it supposedly came from before the response goes out. A claim with no retrievable source is omitted or flagged as unverified, never smoothed over into fluent prose. That last part is a product decision as much as an engineering one: the system is allowed to say it does not know, and that permission is what makes the rest of its answers worth reading.

Citations carry more than a link. Each fact cites the source page, the patch version and the timestamp of the data collection, so a reader can see how current an answer is without asking anyone. Translate that to a company base and you get document ID, version, effective date, retrieval time. Then check that the cited source exists at all - on a content pipeline we run the model invented plausible internal links to pages that were never there, so every link now gets a live request before publish.

The same rule turns up in a completely different domain. On the AI rector's assistant built for a university, the model is not allowed to answer statistics from memory at all: every figure is retrieved at the moment of the question, decisions are confirmed by a person, and each action is logged. The pattern under all of it is one sentence. If the model can state it, make it prove it - the same conclusion we reached taking apart five production failures.

An answer that was right last quarter is now wrong

A knowledge base is not a snapshot, and the systems that rot quietly are the ones treated like one. In the game base every patch reshuffles drop rates, zones and mechanics, so the ingestion pipeline processes source data continuously, versions chunks when a patch changes them, and flags outdated content before it ever reaches retrieval. That last step sounds like housekeeping and is not: it is the rule that stops an old answer resurfacing as a current one.

Company documents move slower and do more damage when they slip. Version 3 of a policy sits in the index next to version 7, the two embed almost identically, and retrieval has no opinion about which one is in force unless somebody gave it one. The resulting failure is not a hallucination, which is what makes it hard to argue with. It is an accurate quotation from a document that stopped applying two revisions ago.

In practice: version at ingest instead of appending, keep effective and expiry dates in chunk metadata, filter on them at query time, and delete-and-reindex when a document changes rather than laying the new one down beside the old. Ingestion stops being a script somebody runs after lunch and becomes a pipeline with state. Most teams underestimate that step, and it is the one that decides whether anyone still trusts the system in month six.

The complaint you hear is rarely the thing that is broken

Users do not report retrieval failures. They report that the assistant is unreliable, which is true and unactionable. The table below maps the sentences people actually say to the layer underneath, because the debugging order matters: fixing generation when retrieval is the problem produces a system that writes better wrong answers.

Read the fix column top to bottom and notice what is missing from it. No row says to buy a larger model or a newer embedding. The whole list is plumbing.

Five probes you can run against your own stack this week

You do not need a framework or an outside team to find out which of these failures you have. Five queries and an afternoon will tell you, and each probe targets one layer, so a failure points at a fix instead of a mood. Run them against the system you already have, not against a fresh index built for the test.

That last row is the one teams skip and the one that pays. Score retrieval separately from generation: the number to watch first is whether the right chunk reached the context at all, because a retrieval problem and a generation problem look identical from the outside and have nothing in common inside. Fifty questions is enough to start. It only has to be more evidence than an opinion.

We build these systems for a living, so weigh the bias accordingly - but the probes cost you an afternoon and no vendor, and they will tell you more than another round of prompt tuning. If you would rather have a second pair of eyes on where a specific base is likely to break, a short audit is a low-key way to get one.

FAQ