Why Bigger Models Will Not Fix Enterprise RAG

Most enterprise retrieval-augmented generation systems work well in demos but fail in production. The pilot answers ten curated questions correctly. Six weeks after rollout, an employee asks for the emergency procurement limit for a supplier in the EMEA region, and the system returns the US figure in the same tone it uses for every correct answer. Nothing flags the error, so the wrong answer reads exactly like the right one.

The reflexive explanation is that the model is too weak, and the reflexive fix is a bigger one. Both miss the cause. These systems fail not because the language models cannot reason, but because no one tested them against how corporate documents behave, and because the evaluation most teams run cannot see the failure coming. The binding constraint is the quality and realism of that evaluation, not model capability. It is also the more tractable problem, because a team controls its own evaluation and cannot control when the next model ships. We built and released a benchmark, the Enterprise RAG Gold Standard, to reproduce the specific ways corporate documents defeat retrieval, and its design runs through this argument as a worked example.

Enterprise RAG fails in production, and the usual diagnosis is wrong.

The scale is not in dispute. In 2024, Gartner forecast that at least 30 percent of generative-AI projects would be abandoned after the proof-of-concept stage by the end of 2025. The realized figure was worse: by the end of 2025, at least half had been abandoned after proof of concept, which Gartner attributes to poor data quality, inadequate risk controls, escalating costs, and unclear business value. The problem outran even that pessimistic forecast.

‍

Those drivers span budgets and governance, and no single technical cause explains them all. But for systems built to answer questions over a company’s own documents, the pattern is sharper. The failure usually happens in retrieval rather than generation: the system fetches the wrong passage, or an incomplete set of them, and the model reasons well over the wrong evidence. The gate is the moment when the system decides which evidence to present to the model.

This changes the remediation path. If the model is the weak point, upgrade it. If retrieval is the weak point, a bigger model just produces more fluent answers built on the same wrong context, which is a costlier way to be wrong. End-to-end answer scores stay green while the system degrades, because they do not isolate the layer that breaks. The stakes also differ from consumer search. When an employee acts on a system’s stated spending limit or safety threshold, a wrong answer becomes a compliance exposure. The failure that kills the project usually surfaces not as a retrieval bug but as lost trust, which is harder to rebuild than a pipeline.

Rather than publish another warning, we built the test that the category lacks. The Enterprise RAG Gold Standard is a synthetic corpus modeled on a multinational aerospace-logistics company, paired with 23 validated question-context-answer triplets and a runnable evaluator, released openly. Building it synthetically lets us publish tangled, contradictory, regulated-looking documents with no real company’s data exposed, while anchoring every fact to a known ground truth. The triplets are graded into three tiers of difficulty.

‍

This is a small version-one seed: ten documents, 23 curated entries, one primary PDF visual trap, and no testing yet of access-control enforcement or OCR on scanned files. It is not a production performance study, and the sample run should not be read as one. Its job is narrower, to make a set of enterprise failure modes measurable and repeatable, which is the prerequisite for fixing them.

Corporate documents break retrieval in three ways, public corpora never could

If the failure concentrates at retrieval, the next question is why retrieval struggles specifically inside enterprises when the same techniques perform well on public data. Corporate document estates have structural properties that public corpora lack and generic benchmarks never exercise. Our analysis identifies three. They are not exotic edge cases. They are the ordinary texture of regulated, multinational operations, and each maps to a tier of the benchmark.

‍

The first is cross-referenced dependency. In a well-governed organization, the answer to an operational question rarely resides in a single place. An incident-management procedure defines a financial threshold for a “critical exception,” but the person authorized to approve that exception is named in a separate governance matrix. Answering correctly means retrieving both and combining them. A system tuned for single-document lookup retrieves the first hop, finds something responsive, and stops, producing an answer that is plausible but incomplete. The multi-hop problem has its own public benchmark, MultiHop-RAG, built because combining evidence across documents is a distinct and hard capability.

The second is contradiction by design. A multinational keeps conflicting policies across regions because the law requires it: remote-work benefits or spending authority differ between its US and EMEA entities, and both versions are correct. A naive system retrieves both and blends them into an answer that fits neither jurisdiction, or picks one without saying so. Detecting that two authoritative sources conflict and reasoning about which governs is something most production systems are never tested for. The research community has started building benchmarks for exactly this, such as RAMDocs for conflicting-evidence retrieval.

The third is structured data trapped in presentation formats. The most load-bearing facts in an enterprise, including authority limits, compliance rosters, and equipment specifications, often live in tables, which are then rendered as PDFs. A chunking strategy that treats a PDF as a stream of text flattens the table and retrieves the right page with the wrong cell. Recent work on document chunking shows the same: conventional text chunking struggles with multi-page tables and content that spans page boundaries. These visual traps look trivial in a demo with clean markdown and fail in a real repository.

The gap is not theoretical. An IBM team reporting on its own enterprise RAG work found that product documentation differs from Wikipedia-style benchmark corpora, and that common benchmark techniques were not useful for the novel questions real users asked. A system can score well on a generic benchmark and still fail all three of these tests on the first day of deployment, because the generic benchmark was measuring a different kind of corpus.

The binding constraint is evaluation, and it has to measure retrieval on its own

The three features explain why retrieval fails. They also define what a credible evaluation entails, and this is where most teams fall short. The common practice scores the system end-to-end: ask a question, read the answer, and judge it. That single composite number conflates two failures that need opposite fixes. A retrieval miss and a reasoning miss look identical on an end-to-end scoreboard, and a faithfulness metric will certify an answer that is faithful to the wrong document.

Separating the two is not exotic; the established frameworks already do it. RAGAS, ARES, and diagnostic tools such as RagChecker score retrieval and context quality, apart from generation and faithfulness. The capability exists. What is missing in most deployments is the will to use it and a scenario set worth running it against. Our benchmark’s most useful feature is also its least glamorous: the runner can score retrieval on its own, with no language model in the loop, and answer the question teams skip, which is whether the system fetched the right evidence before anyone asks how well it reasoned.

The benchmark’s own sample shows why this matters. It is a retrieval-only run, so its headline composite score is low by construction: with no generated answer, the answer-relevance component is zero, pulling the average down. The number looks like a failure, but the retrieval-precision figures inside it are high. A single blended score hides more than it reveals.

The field does not lack benchmarks, and the picture has changed in the past year. Onyx’s EnterpriseRAG-Bench is a serious public effort built on a simulated company with more than 500,000 documents across sources like Slack, email, and internal tools. Enterprise benchmarks exist; they make different bets, and none replaces enterprise testing against its own document pathologies.

‍

Generic retrieval and embedding suites, such as BEIR and MTEB, assess whether a retriever is broadly competent, but they lack enterprise-level structural features and were not built to certify production readiness. EnterpriseRAG-Bench bets on breadth and source mix. Our seed bets on depth in two underserved corners, contradiction handling, and PDF visual traps, in a small, auditable form. NVIDIA, writing for enterprise buyers, reaches the practical conclusion: the best retrieval evaluation data is your own, and proxy datasets may not represent your workload. So screen with public benchmarks, then gate the go/no-go decision on a scenario harness that looks like your company.

A bigger model cannot retrieve a document that was never found.

The most persistent counterargument is that all of this is avoidable: route corporate data through a frontier cloud model and let scale absorb the mess. It is a serious position, and cloud is the right choice in many deployments. But it assumes model capability can substitute for evidence quality, and it cannot.

FinanceBench shows this directly. It asks straightforward questions about public financial filings. On its expert-reviewed sample, GPT-4-Turbo, used with a retrieval system, answered incorrectly or refused to answer 81 percent of them. These are 2023-era models, and capability has improved since, so the figure is a baseline rather than a current scoreboard. The structural lesson holds, and the authors drew it themselves: the same model with no retrieval answered only 11 percent correctly, and they judged the usual workaround of stuffing more context into a longer window unrealistic for enterprise use because of latency and document size. Retrieval set the ceiling. A better model produced more fluent answers without introducing missing evidence.

‍

LegalBench-RAG makes the same point in law: precise retrieval beats dumping broad context into a long window, because over-retrieval raises cost and latency and degrades the answer. A larger model can make weak evidence sound authoritative. It cannot recover a regional addendum the retriever never surfaced, repair a misparsed table, or decide which of two conflicting policies governs, unless the retrieval and evaluation harnesses surface those cases first.

For many enterprises, the cloud-only path is not just insufficient but constrained. Regulated sectors operate under HIPAA, GDPR rules on cross-border data transfers, the EU AI Act’s requirement for auditable evidence, and supervisory guidance, such as the New York Department of Financial Services’ expectations for third-party and AI provider risk. None of these bans cloud outright, and the argument here is not against the cloud. The narrower point is harder to dodge: model quality sits downstream of evidence quality, and no regulatory posture or model upgrade changes that order.

What to ask your team on Monday

The dataset is useful, but the discipline it encodes is the real deliverable, and it determines whether or not a team ever runs our corpus.

Start by distrusting any single end-to-end score. A number that fuses retrieval and generation cannot tell a context problem from a hallucination problem, and it sends remediation budget to the wrong place. The retriever needs its own metric, measured before any answer is generated. Then make the gold set look like the company that will use it. A test of single-document lookups will pass a system that collapses the moment an employee asks a question spanning two policies, or one that contradicts itself across regions, so the questions have to include multi-hop chains and real contradictions drawn from the highest-stakes documents, such as the procurement limits and the regional compliance rules. If those questions are hard to write, that difficulty is itself the finding: the document estate is more tangled than the current evaluation assumes. And the gold set is infrastructure, not a one-time exercise, because retrieval quality breaks on changes that look harmless, such as a new embedding model or a swapped reranker. Teams that hold quality steady rerun the same labeled set on every such change and refuse to ship a regression they cannot see.

One closing point is ours to assert rather than prove, and we hold it lightly because the evidence is still accumulating. A firm that helps build the instrument an industry uses to measure itself earns particular credibility, the kind that comes from showing, in runnable code, that it understands where deployments break. That is why we built this in the open and plan to expand it. The operational point holds no matter who wrote the benchmark. So ask whoever owns your AI platform one question this week: Does our evaluation set include multi-hop questions across conflicting documents, and does it score retrieval sufficiency separately from answer quality? If the answer is no, this is not a problem you can postpone. You are running a production system you cannot yet trust, and the model was never the reason.