Retrieval Augmented Generation for Enterprise Knowledge Bases

RAG stands for retrieval-augmented generation. It decides whether an AI system answers from your company's actual, current records or from whatever the model absorbed during training. For an enterprise, that gap is the whole point: a chatbot that guesses versus one you'd actually let a customer talk to. I've watched a dozen of these deployments up close, and the ones that work and the ones that don't split on exactly this line.

The mechanics aren't complicated. Someone asks a question, that kicks off a search over an external corpus (documents, tickets, wikis, whatever's sitting in storage), the system pulls back the chunks that look most relevant, and stuffs them into the model's context window. Only then does the model write an answer, grounded in what it just retrieved instead of whatever got baked into its weights months ago. Running this well, though, is another matter entirely.

A model's training data is frozen at whatever point it got scraped. It knows nothing about your Q3 pricing changes, your open incidents, or the contract your sales team signed last Tuesday. Your company's knowledge changes daily and sits scattered across a dozen systems that were never built to talk to each other. RAG is the bridge between a smart, generic model and how your business actually runs day to day, which is a lot messier than any product demo lets on.

In practice, "external corpus" means Confluence pages, Salesforce records, Zendesk tickets, engineering runbooks in GitHub, PDF contracts buried in SharePoint, Slack threads nobody archived. Some of it's structured. Most of it isn't, and almost none of it was built with AI retrieval in mind.

The retrieval layer is where most of the real engineering happens. Semantic search, done through vector embeddings, finds chunks that are conceptually close to a query without sharing a single word with it. Hybrid retrieval pairs that with older sparse methods like BM25, because pure vector search whiffs on exact matches, a SKU number, an account ID, the stuff keyword search has nailed for twenty years. Then there's chunking: slicing a 40-page policy doc into retrievable pieces. Too coarse, and you pull back noise. Too fine, and you lose the context that made the answer make sense. Mess up chunking and nothing downstream saves you.

Quick word on where RAG's boundaries sit, because I've watched this confusion cost teams real weeks. Fine-tuning retrains the model's weights; RAG never touches them. A cache sits still for speed; RAG's corpus shifts minute to minute. Enterprise search hands you a list of documents; RAG treats retrieval as raw material for an engine that writes an actual answer. Confuse these going into a deployment and you'll build the wrong thing, and you won't find out until it's already live.

The market's caught on anyway. RAG is projected to grow from $1.94 billion in 2025 to $9.86 billion by 2030, a 38.4% compound annual growth rate. I've sat with that number a while trying to figure out what's driving it, and it's not hype cycles: it's the number of enterprises that tried plain LLM chatbots, watched them confidently invent answers, and went looking for something grounded instead. RAG is the default architecture for enterprise AI now, past the point of being a handful of labs experimenting on the side.

Why most enterprise RAG pilots fail before they reach production

MIT's Project NANDA found that roughly 95% of organizations deploying generative AI saw zero measurable return. The cause traced back to data readiness and governance. Not whether the model underneath was any good.

The models work. The retrieval technique is proven. So what's actually breaking?

A few patterns keep showing up, across companies that have never talked to each other. Retrieval surfaces a document the user has no business seeing, and the system either leaks something it shouldn't, or gets so over-filtered trying to avoid that outcome that it stops being useful. Indexes go stale because nobody synced the retrieval corpus with the source systems, so the AI answers confidently and wrong. Five teams build five separate RAG pipelines over the same overlapping documents, and none of them know the other four exist. When something breaks, there's often no way to trace which document got pulled, by whom, or why.

I kept expecting, the more of these I looked at, that the common thread would be model quality. It never was. These are symptoms of an infrastructure and governance problem, plain and simple. The fix isn't a fancier embedding model; it's the plumbing around retrieval that holds up once the system is live, not just in a demo where everyone already knows the right answer going in.

The permissions problem that sits underneath every RAG deployment

Here's the default failure mode, and it's more common than most vendors want to admit. A RAG system ingests documents from across the company into one shared vector index, without inheriting who's actually allowed to see what. Ask it something, and it retrieves purely on relevance, with zero regard for whether you have clearance to see the result.

Think about what that means for an actual person on an actual team. A sales rep should pull contract details for their own accounts, not every account in the company. An engineer should retrieve runbooks for systems they own, not production secrets from a team three floors away. HR documents get scoped by role and by geography for a reason; they were never meant to surface for any employee who happens to be logged in that day.

The obvious fix, filtering results after the fact based on someone's role, breaks in a way that doesn't show up until it's too late. It means re-encoding every source system's access rules in a second, separate layer, one that drifts the moment nobody's watching. Permissions change constantly at the source, and your filter layer has no idea, unless someone remembers to go update it by hand, which someone eventually forgets to do.

The better path inherits permissions straight from the source and keeps them synced automatically. Lose access in SharePoint or Confluence, and that change shows up immediately in what the RAG system will and won't hand you. No manual step, no lag.

A company running 50 source systems across 10 teams cannot keep permission parity by hand. Add more systems and more teams and the math only gets worse, not better. Get it right, though, and something good happens on the other end: people trust the answers, because they know the system only shows them what they already had access to anyway.

How MCP changes the retrieval architecture for governed knowledge access

Before MCP, hooking AI up to enterprise data meant a custom connector for every pairing of AI app and data system: M applications times N systems, each with its own login flow and its own data format. Boston Consulting Group called MCP "a deceptively simple idea with outsized implications," and the math backs that up. Without a shared protocol, integration complexity grows quadratically as you add agents and data sources. With one, it grows linearly.

MCP, short for Model Context Protocol, is an open standard Anthropic put out in November 2024. It defines how AI systems talk to external tools and data through a common interface, built around three roles (host, client, server) and three primitives (tools, resources, prompts).

Here's what matters most for RAG specifically: authentication and authorization live at the MCP server, not inside the AI agent. The server checks what the actual person is allowed to see in the source system, and the agent only gets back what that person could've pulled themselves, logged in under their own name. A March 2025 update to the spec added an OAuth 2.1-based framework for HTTP transports, which locked this pattern in as a formal requirement instead of leaving it as a nice-to-have convention.

For RAG, that means retrieval can query enterprise knowledge through a standard interface while inheriting real access controls, rather than approximating them with a bolted-on filter after the fact. On the knowledge base side, an MCP server exposes resources (documents, database rows, file contents) and tools (search functions, structured queries), and those become the actual building blocks retrieval runs on, respecting the source system's rules the whole way through.

There's a practical side benefit too, around context window bloat. Instead of loading every tool definition an agent might conceivably need, MCP supports progressive discovery, where agents query for relevant categories of tools as the need arises. In a knowledge-dense deployment, that's the difference between a context window you can actually use and one that's full before the real question gets asked.

What governed RAG infrastructure looks like in practice across teams

The alternative to governed RAG isn't chaos. It's duplication, which in some ways is worse because it looks like progress. Five teams, five pipelines, built over the same document types, none sharing retrieval logic, none sharing a permission model, none aware the other four exist. Multiply that wasted engineering time across a company of any real size and it piles up fast.

The governed version looks different. One team builds a retrieval skill, publishes it once to a shared registry, and other teams use it without rebuilding it from scratch. A registry brings structure that a shared drive full of scripts just doesn't have: changes get versioned, so if a retrieval skill regresses, you can trace what changed and roll it back. Every skill has an owner, an actual accountable team rather than an orphaned file nobody remembers writing, and access can be scoped, published broadly or locked down to one department or a named group of users.

By department, this tends to look like: RevOps builds retrieval over CRM records, deal history, and sales playbooks, scoped to account ownership so a rep sees their own pipeline and nothing else. Engineering builds retrieval over runbooks, incident history, and architecture docs, scoped to system ownership. Support builds retrieval over product documentation and past ticket resolutions, scoped to whatever tier of customer data that agent is cleared to touch.

Each team's scoped context gets delivered through its own MCP server, which plugs into whatever surface that team already uses (Claude, Slack, a support tool) without anyone building a custom front end per deployment. Mid-market IT teams running MCP-connected knowledge base setups report 30 to 50% faster ticket resolution. That's a real outcome, tied directly to retrieval being accurate and scoped to the right person at the same time.

Auditability as a first-class requirement, not a compliance afterthought

Auditability means you can answer four questions for any response a RAG system gives. Which documents got retrieved. Which user asked, and what permissions they held at that moment. Which version of the retrieval skill was running. And what action, if any, got taken because of the answer.

This matters well beyond satisfying an auditor once a year. When a RAG system hands back a wrong answer, someone needs to know whether that was a retrieval error (wrong document pulled), a permission error (a document that shouldn't have been in scope at all), or a skill error (flawed logic baked into the prompt). Without a log, that failure is just a black box, and you can't fix what you can't see.

MCP's governance controls are built for exactly this: per-operation role-based and attribute-based authorization, OAuth 2.0 authentication with credentials kept entirely outside the AI's context, and attribution-level audit logging that records every tool call, its inputs, and its outcome.

None of this is hypothetical, either. The first large-scale security study of MCP servers found that a notable share carried vulnerabilities, and a similar share showed signs of tool poisoning. Those numbers stopped me the first time I read them, because they mean the failures aren't edge cases dreamed up for a conference talk, they're already sitting in production systems. Perimeter security won't catch that on its own; attribution logging will.

For anything consequential, a write, a deletion, an email sent on someone's behalf, human approval needs to sit in the loop, and the audit log has to capture not just what the agent did but who signed off on it. Governance, framed this way, is the actual condition that lets people trust an AI-generated answer enough to act on it: pass it to a colleague without hedging, hand it to a regulator without flinching.

What separates a governed RAG implementation from one that creates new silos

The pattern to avoid is easy to describe, mostly because it's everywhere. RAG built team by team, each with its own index, its own homemade guess at permissions, its own prompt logic. No registry, no shared visibility, no inherited access control. It works fine in a demo. It falls apart the second two teams need to share knowledge, or a regulator asks a question with teeth.

A governed implementation holds up under scale in a way ad hoc pipelines never do. A few traits tend to show up together.

Permissions inherit straight from the source systems, so there's no bolt-on layer quietly drifting out of sync. Retrieval skills get versioned and published to a shared registry, instead of buried in someone's notebook or hardcoded into a prompt nobody else can find. MCP servers are scoped per team, so knowledge moves without a custom integration for every new surface that shows up. Every retrieval, every user, every action gets logged with full attribution, so failures can be diagnosed and compliance shown on demand. And the whole setup stays model-agnostic, since permissions and guardrails travel with the organization rather than with one specific LLM, so swapping models doesn't mean rebuilding governance from scratch.

Credal is a working example of this. Domain experts publish retrieval skills to a governed registry, permissions inherit and stay synced across hundreds of connected sources, and each team gets its own scoped MCP server that plugs into Claude, ChatGPT, Cursor, or Slack, whatever surface they're already using, without rebuilding the permission model every time.

The payoff shows up quietly, mostly in what stops happening. Five teams stop rebuilding the same prompt from scratch. Duplicated effort turns into something shared, and answers start carrying the kind of paper trail an organization needs before it's willing to act on them.

For any answer your RAG system gives, you should be able to say who retrieved what, under which permissions, approved by whom. Short of that, what you've built is still a pilot.