Seventy percent is where enterprise RAG systems stop. Nothing in the architecture produces that number and it is not a benchmark; it is shorthand for the plateau between a system that impresses stakeholders in a demo and one anyone is willing to put in front of a customer.
The first prototype answers seven of ten questions correctly, stakeholders are impressed, and a roadmap gets drawn. Six months later the system still answers seven of ten, leadership now wants nine, the model has been swapped twice, and no one can account for why the same three failures persist.
By then the model has become the obvious suspect, though it is rarely the constraint. Most systems at this stage are held back by the retrieval layer, the corpus, the query path, or the absence of an instrument that would tell anyone which of those is failing.
That last one is the expensive part. The plateau is usually not a capability problem: it is a diagnosis problem, and the organizations that break through are the ones that fund the diagnosis before the next upgrade.
The sections below walk from the layer everyone looks at down to the one that decides whether any of it improves.
1. The model is the visible layer, not usually the failing one
When a RAG system returns a wrong answer, the artifact everyone sees is the model output. The reasonable conclusion is that the model produced it, so the model is the problem. That is why remediation budget flows toward model swaps, larger context windows, fine-tuning, and waiting on the next frontier release.
In the production systems I encounter, most stubborn failures happen upstream of the model. The right document was never retrieved, the right passage inside it was never surfaced, or the retrieved context was noisy enough that no model could have produced a defensible answer from it. Generation does fail on good evidence sometimes, which is exactly why the two layers need to be measured separately.
There is a cheap test that settles it. Take the failing queries, manually inject the correct source passages into the prompt, and re-run them. If the answers come back correct, retrieval — not generation — is the bottleneck for those cases. In many engagements this single exercise explains a large share of failures without changing the model at all, and it costs an afternoon.
Run it before approving the next model migration. Until someone does, every model swap is an expensive guess dressed as a decision.
2. Chunking is a week-one decision that sets a year-two ceiling
Chunk size gets chosen in the first week of a project, usually by whoever wrote the ingestion script, before anyone has seen a real query distribution. That is the correct sequence: you cannot know the query mix before you have users. What is missing is anything downstream that triggers a revisit, so an early guess quietly becomes the system’s ceiling months later.
The failure modes it produces are specific:
- Fixed-size chunks cut across semantic boundaries. A definition lands in chunk 14 and the qualifier that inverts its meaning lands in chunk 15. The model sees only the definition and answers confidently and wrongly.
- Too-small chunks retrieve cleanly but lack the surrounding context needed to interpret them. Embeddings match; answers are still wrong.
- Too-large chunks dilute the embedding signal. A relevant sentence buried in 2,000 tokens of unrelated material does not retrieve.
- Tables, lists, and structured content get mangled by naive splitters. A pricing table broken across two chunks is worse than no pricing table at all.
- Headings and document hierarchy are usually stripped, discarding the strongest signal a human reviewer would have used to judge relevance.
The fix is not a better chunk size. It is a chunking strategy that respects document structure: splitting on semantic boundaries, preserving hierarchy as metadata, handling tables and code blocks distinctly, and attaching neighboring context at query time so the model sees the passage and what surrounds it.
Contextual retrieval is a further option: preserve or generate document-level context when chunks are embedded or indexed, so an otherwise ambiguous passage retains information about where it came from. Anthropic’s contextual retrieval approach and contextualized chunk embeddings implement it differently. Neither removes the need to test chunk boundaries, metadata, latency, and retrieval quality against your own corpus.
This work is unglamorous and hard to put in a steering committee update, which is most of why it goes unfunded. In stalled systems it is routinely a better use of a sprint than another model evaluation. The implementation tradeoffs belong in the component-by-component RAG guide; the executive point is that chunking and context representation are measured design choices with a budget line, not ingestion defaults.
3. Semantic similarity is not the same as answering the question
Dense vector search is the default retrieval mechanism in most implementations, and it carries a well-documented limitation: it retrieves documents similar to the query, and similarity is a weaker proxy for relevance than it appears.
Three query patterns expose this quickly:
- Exact-match queries. A user asks about “error code E-4471.” Embedding similarity surfaces documents about error handling in general, while the document naming E-4471 ranks tenth.
- Acronyms and proper nouns. Embeddings often treat domain-specific acronyms as low-information tokens. The document containing the acronym is missed; the document discussing the neighborhood topic is returned.
- Negation and constraints. “Which products are not covered under the standard warranty” retrieves documents about coverage rather than exclusions, because the embedding is dominated by topic, not constraint.
Two changes reliably earn their cost:
Hybrid search. Combine dense vector retrieval with lexical search (BM25 or similar) and merge results with reciprocal rank fusion or a learned ranker. It improves coverage across both semantic and exact-match traffic, though the size of the gain depends on your query distribution and corpus.
Reranking. Retrieve a wider candidate set and score each candidate against the query. Rerankers add latency and per-query cost, but they model the query-passage relationship more directly than vector proximity does. Where initial retrieval has adequate recall but poor ordering, this deserves evaluation before anyone reopens the question of the generation model — it is a smaller change with a larger expected effect.
4. Query understanding: what the user typed vs. what they meant
Enterprise users do not write queries the way retrieval systems expect. They write fragments, multi-part questions, and follow-ups that reference earlier turns implicitly. They assume domain knowledge the retrieval layer has no access to. And they ask questions answerable only by combining several documents, which a single retrieval pass cannot satisfy by construction.
The interventions that address this:
- Query rewriting. Use the model to expand, clarify, or reformulate before retrieval. A vague “what about Q3?” becomes “What were the Q3 2025 revenue figures and key variance drivers?”, with a far better chance of retrieving the right document.
- Multi-query retrieval. Generate several reformulations, retrieve against each, and union the results. This lifts recall for questions that can be phrased many ways.
- Decomposition for multi-hop questions. Some queries need two or three facts living in different documents. Detect them, decompose into sub-queries, retrieve for each, and synthesize.
- Conversation-aware retrieval. In multi-turn applications, resolve references against prior turns before retrieval. “Show me their renewal terms” is unanswerable without knowing who they refers to.
Most stalled systems pass the user’s query to retrieval verbatim. That is a framework default rather than a decision anyone made, and it underperforms on classes of traffic that matter commercially, exactly the queries a paying user asks when the easy questions are already answered.
Some systems now place an agent around retrieval, letting it search, inspect results, refine, follow references, and stop when it has enough evidence. That is genuinely useful for investigative and multi-hop work, but it changes what you are buying. The unit of evaluation stops being a single top-k call and becomes the whole retrieval trajectory, which has to be scored step by step, and that is a different and more expensive instrument than the one most teams have built. Budget for the shift before approving the architecture, not after the first invoice.
5. The corpus is dirtier than the business believes
RAG projects are scoped on an unstated assumption that the source corpus is reliable ground truth. That assumption belongs to the organization, not to the engineers implementing against it, and it is where a great many programs quietly lose their accuracy budget.
Real enterprise corpora contain:
- Multiple versions of the same document, with no signal indicating which is current.
- Drafts that look authoritative but were never approved.
- Contradictions across departments — sales collateral against policy documents against engineering specs.
- Stale content still sitting in the source system long after it was superseded.
- Charts, tables, diagrams, and scanned pages whose meaning is lost when ingestion extracts plain text.
- Documents written for an audience the user is not part of, carrying assumptions that lead the model astray.
A RAG system retrieves what its index scores as relevant, not what the organization considers authoritative. If three versions of a policy exist and two are obsolete, similarity alone cannot identify the governing one, and the answer will sound completely confident while citing the wrong version. That is a compliance exposure on top of a quality issue.
The remedy is corpus curation: an explicit pipeline that decides what gets ingested, deduplicates near-duplicates, attaches authority signals (date, owner, status), filters or down-weights drafts and superseded versions, and gives retrieval something to work with beyond treating every document as equally valid. Where answers live visually, that pipeline may also need OCR, layout-aware parsing, or multimodal retrieval that represents text and images together — evaluated by content type, since a text-only aggregate score hides failures on visual documents.
This work has no natural owner. It sits with the business functions that produced the documents, none of whom agreed to maintain a retrieval-quality dataset, and it does not get done because it was never assigned. The consequence of leaving it unassigned is a system that answers fluently from a polluted ground truth — worse than one that admits it does not know. The durable fix is to stop treating the corpus as a project input and run it as an operational service with owners and objectives, which every product team should inherit by default.
6. Without measurement, iteration is indistinguishable from guessing
The clearest predictor of whether a RAG program can improve deliberately is whether a real evaluation system exists, by which I mean an actual instrument and not a habit of spot-checking outputs or confirming that the demo questions still work.
What one is made of:
A golden set. A curated collection of representative queries with known correct answers and known relevant source documents, built to cover the long tail of question types where the obvious ones will already pass. Versioned and reviewed like any other asset.
Retrieval metrics separated from generation metrics. Measure whether the right documents were retrieved (recall at k, mean reciprocal rank) independently of whether the answer was correct. Without that separation, no one can tell a retrieval failure from a generation failure — which means no one can prioritize the fix.
LLM-as-judge with calibration. Use a strong model to grade against the golden set, and calibrate it against human judgment on a schedule. Uncalibrated judges drift, and a rising score against a drifting judge reports progress the system has not made.
Production telemetry, not only offline eval. What are users actually asking? What is the query mix? Which queries produce low-confidence answers, retries, or negative feedback? Offline eval measures performance on questions you anticipated; telemetry tells you which questions you did not.
Regression testing on every change. Prompt edits, model swaps, chunking adjustments, reranker updates — each runs against the golden set before deployment. Without it, gains in one area silently cost you another and nobody learns of it before a user does.
Absent this layer, a team can work hard for two quarters and be unable to demonstrate whether the system improved. That is not a failure of effort or rigor by the people iterating; it is an instrument that was never funded. Evaluation infrastructure is the least visible line item in a RAG program and the one that determines whether every other line item can be justified.
7. Ownership sits on a seam
RAG systems straddle an organizational boundary. Retrieval looks like a search problem and lands with a data or platform team. Generation looks like an ML problem and lands with the AI team. The corpus belongs to whichever business functions produced the documents, usually several, none of whom took on retrieval quality as a responsibility.
Each of those teams is doing its job competently. The AI team improves the model, and end-to-end accuracy does not move. The platform team optimizes retrieval latency and meets its SLO. The document owners keep writing for human readers, which is what their function asks of them. The seam between them is where user-visible answer quality lives, and it is unowned by construction.
Closing it takes an explicit owner of the system as a whole — one person accountable for answer quality, with authority across retrieval, generation, and corpus curation. That is an org design decision, not an engineering one, and it is made above the teams doing the work. Without it, every group optimizes its own surface and the plateau holds regardless of how much is spent. The same seam runs through production agent systems for the same structural reason, which is worth knowing before an organization concludes the problem is specific to retrieval.
What moving past the plateau looks like
Systems that reach production reliability share a profile.
They treat retrieval as a primary engineering surface with its own budget. They evaluate hybrid search, reranking, and contextual or multimodal retrieval where the corpus justifies it. They fund chunking and corpus curation before defaulting to fine-tuning. They rewrite, decompose, or iteratively investigate queries when one retrieval pass cannot answer them. They measure retrieval and generation separately, regression-test every change, and name one owner accountable for end-to-end answer quality. The controls behind the corpus half of that list are enumerated there in full.
None of it requires frontier model capability. It is engineering discipline applied to failure modes that are well understood and rarely addressed in the right order.
The reason the order goes wrong is structural. The model is the only component in the stack that has a vendor, a launch event, and an upgrade path — so it is the easiest thing to put on a roadmap and the easiest to get approved. Retrieval quality, corpus curation, and evaluation infrastructure have none of that, and they lose the argument for funding even when they are the binding constraint.
Moving past the plateau is mostly a matter of funding the right layer. The diagnosis costs an afternoon.
If your RAG system has plateaued and you need an honest read on where the bottleneck actually is, I work with engineering leadership to diagnose retrieval and corpus issues and build the measurement discipline that makes iteration defensible. Get in touch.