Capturing Tacit Knowledge in RAG Pipeline Design

Tacit knowledge falls into three practical categories that RAG systems routinely miss, and naming them precisely is the prerequisite for encoding them. Most pipeline designers skip this step entirely, which is why the failures downstream feel inexplicable.

The first is situational context. Experts say "it depends" constantly and almost never write down what it depends on. The document states the rule. The expert carries the fifteen situations where the rule doesn't apply and the three where a completely different rule takes precedence. I've watched sales teams build RAG systems on top of their best playbooks and then wonder why the outputs felt off. They were off because the playbook captured the conclusion of a thousand conversations, not the conditions under which any given conclusion actually held.

The second is negative knowledge: what not to do, which approaches failed, which vendors disappointed on which specific criteria. Nobody writes the page titled "things we tried and abandoned and why." That document doesn't exist because documentation is written to be true, not complete. The knowledge lives in the people who ran the experiments. When they leave, it goes with them, and the organization is left with a corpus that confidently describes only the things that worked.

The third is relational and causal reasoning, the chain of inference connecting an observation to an action. Why a particular signal in a customer conversation predicts churn two quarters out. Why the first engineer assigned to a project matters more than the project plan. These chains are the product of accumulated experience, and they rarely get articulated because the expert doesn't experience them as chains. They experience them as intuition: fast, automatic, and invisible to anyone watching from the outside.

Documents don't carry this content because they were written to record conclusions after deliberation, not the deliberation itself. Research cited by enterprise AI analysts has found that organizations effectively capturing tacit knowledge are substantially more likely to outperform peers in market responsiveness. That finding makes sense to anyone who has watched an organization lose a critical team member and spend the next six months quietly relearning what that person knew.

There's a timing problem layered on top of all this. Tacit knowledge is most accessible during the act of doing. A surgeon knows something in the OR they cannot fully recall in a debrief. A rep knows something on a call they won't think to write down afterward. Capture-at-the-moment is a pipeline design requirement, not a documentation recommendation, and most teams treat it as the latter until they've already lost the knowledge they were trying to preserve.

The practical implication: your source corpus is not neutral. It reflects what someone chose to write, which systematically excludes the most context-dependent, situationally qualified, and causally rich knowledge the organization actually holds.

How Chunking Decisions Encode or Erase Expert Reasoning

Chunking gets treated as a preprocessing step. It's actually a theory about where meaning lives in a document, and most default strategies embed a wrong theory.

Fixed-size chunking severs reasoning. An expert's logic typically has a setup, the context in which a rule applies; the rule itself; and the qualifier, the exceptions the expert would name without being asked. Splitting at token boundaries routinely separates the condition from its conclusion. You retrieve the rule without the setup that tells you when to apply it. The chunk is internally coherent but epistemically incomplete, which is a distinction that matters enormously in practice and almost never gets surfaced during testing because the chunk reads fine in isolation.

Semantic chunking is an improvement, splitting at natural language boundaries rather than token counts. It still requires knowing what an argumentative unit is in the target domain. The right chunk boundary in a legal contract is not the right boundary in a clinical protocol or an engineering runbook. Generic splitters guess. Domain experts know.

The most direct way to find those boundaries is to interview experts not to ask "what do you know" but to ask "where does one idea end and the next begin" in their core documents. This produces domain-specific chunking heuristics that no off-the-shelf splitter will find independently. It takes a few hours. It changes retrieval precision substantially. Most teams skip it because it doesn't feel like an engineering task, which is exactly why it keeps producing the same retrieval failures.

Hierarchical chunking addresses a structural problem that even well-tuned semantic chunking fails to solve. When you preserve parent-child relationships between a section summary and its constituent paragraphs, retrieval can return the specific child chunk the query matched while the parent provides the frame: why this knowledge exists, what situation produced it. Flat chunking destroys this relationship. The child answers "what." The parent answers "why this answer exists." You need both for a response that reflects expert reasoning rather than just document content, and most pipelines only return the child.

Metadata as the Layer Where Situational Context Gets Encoded

A retrieved chunk answers "what." Metadata answers "when does this apply," "who validated this," and "what situation produced this knowledge." Most pipelines treat these as separate concerns. They're not, and conflating them is one of the more expensive mistakes a pipeline can make quietly.

Standard metadata fields, file name, date created, author, are bibliographic. They tell you where a document came from. They do not tell you when the knowledge in it should be trusted, which situations it was designed for, or when it quietly became outdated. Bibliographic metadata supports citation. It does not support retrieval judgment.

Epistemic metadata does the second job. These are the fields a domain expert would add if they were curating rather than just uploading. Applicability conditions: "applies to enterprise accounts above a certain revenue threshold," "relevant only when the prospect is in an active procurement cycle." Confidence and provenance: "validated across three separate field cases" versus "one engineer's hypothesis from a post-mortem." Temporal validity: "reflects vendor capability as of a specific quarter," which is especially critical in fast-moving markets where a six-month-old document can be actively misleading. Failure context: "this approach was tried and abandoned because," encoding the negative knowledge that almost never surfaces in the document itself.

Who generates epistemic metadata matters more than how it's structured. The person who created the document is rarely the right choice. Automated pipelines are even less reliable. The person who has used the knowledge and knows when it breaks down is the right annotator. The practical workflow is structured annotation sessions where domain experts review chunks and tag them with applicability conditions and confidence levels. Time-intensive up front; transformative for retrieval precision downstream. I've seen teams skip this step and spend months debugging retrieval failures that were never going to resolve through prompt tuning.

A well-tagged corpus lets the retrieval step approximate the expert's first instinct, "does this even apply to my situation," before the LLM sees the content. That filter eliminates the retrieval of technically relevant but situationally inappropriate chunks. It's the difference between a system that retrieves what matches and one that retrieves what helps.

In multi-team pipelines, permission inheritance is a metadata concern, not an afterthought. When chunks carry access scope as a metadata attribute inherited from source systems, retrieval respects organizational information boundaries automatically. Redefining permissions per deployment is inefficient and produces gaps that eventually cause compliance problems, usually at the worst possible moment.

Query Handling That Accounts for What Experts Leave Unsaid

Expert users ask compressed questions. They assume shared context the pipeline doesn't have, omit qualifiers they consider obvious, and use domain shorthand that doesn't match the vocabulary in the corpus. A senior sales engineer asking "why did we lose Acme" is asking a question whose full form would span three paragraphs. The query as typed retrieves almost nothing useful.

Novice users present the inverse problem. They don't know what they don't know, so the question they form is missing the contextual signal that would retrieve the right chunk. Grammatically complete, epistemically insufficient.

Both cases require query handling that adds what the user didn't include.

Hypothetical document embeddings address the vocabulary gap at the structural level. Instead of retrieving against the sparse query, the system generates a hypothetical answer first, then retrieves against the embedding of that answer. The embedding of an imagined full response is closer to the embedding of an actual relevant document than the embedding of a three-word query is. This corrects for the distance between how people ask questions and how knowledge gets written down.

Expert-informed vocabulary mapping handles domain shorthand. A controlled map linking internal terminology to the full language used in the corpus, built once by subject-matter experts and applied at query time, closes the gap between "what the rep says" and "what the document says." This is not a synonym list. It's a curated translation layer built by people who know both vocabularies and can distinguish the cases where the terms are interchangeable from the cases where they're subtly not.

Multi-query generation rewrites the user's question as several semantically related variants and retrieves against all of them, useful specifically when domain experts phrase things differently than documents do, which is most of the time. Contextual query enrichment pre-pends situational context for known user roles before retrieval runs: a support engineer's query gets different context injection than a sales engineer's identical-sounding query, because their situations differ and the pipeline should reflect that distinction without requiring the user to articulate it.

The hybrid retrieval argument is well-supported in practice. Dense retrieval captures expert intent even when vocabulary doesn't match. Sparse retrieval, using keyword-based methods like BM25, captures exact technical terms that embeddings sometimes blur together. Research shows dense methods outperforming sparse approaches on general knowledge queries, while sparse methods can edge ahead on highly specialized technical terminology. Running both and combining the results covers the vocabulary mismatches that tacit knowledge routinely produces.

The query itself is also a signal most pipelines ignore. What someone asks, and how they phrase it, reveals their mental model and their assumptions. Pipelines that log and analyze query patterns over time surface the tacit assumptions users bring to the system. Those patterns feed back into metadata design and corpus gap analysis. The query log is a knowledge capture mechanism, not just a debugging tool, and treating it as the latter is leaving real signal on the table.

Routing and Agent Orchestration as Expert Judgment Made Explicit

A domain expert doesn't consult one source. They have a hierarchy: check the live system first, then the runbook, then call the person who built it. The order is not arbitrary. It encodes years of experience about where answers actually live and how fresh they are. Static RAG pipelines retrieve from one corpus and have no equivalent of that consultation hierarchy, which is why they produce answers that are technically retrieved but practically wrong in ways that can be hard to diagnose.

Routing encodes the hierarchy. Query classification categorizes an incoming question by type: procedural, factual, relational, exception-handling. Each type routes to the corpus most likely to contain that kind of knowledge. A question about "what to do when" goes to runbooks; a question about "why we decided" goes to decision records; a question about "what is this customer's situation" goes to CRM-adjacent structured data. The routing logic is where expert judgment about source reliability gets operationalized. It's one of the highest-leverage places to invest design time, and it's consistently underbuilt.

Confidence-triggered escalation routes to a secondary source, or flags for human review, when top-k retrieval returns low-confidence chunks. This is the pipeline equivalent of an expert who says "I'm not sure, let me check with someone who knows this better." Generating from weak evidence without flagging it is one of the most consequential failure modes in deployed RAG systems. It's also surprisingly common, because it fails to surface in standard accuracy testing.

Specialist sub-agents decompose complex questions into sub-questions requiring different expertise, route each to its domain-specific index, and synthesize the results. This mirrors how experts actually delegate: not by finding one person who knows everything, but by knowing who to ask for which part of a problem. GraphRAG adds relational structure by connecting retrieval to a knowledge graph built from internal content, letting the pipeline trace causal and relational links rather than returning isolated chunks that each answer a piece of the question without connecting them. For knowledge that is inherently relational, isolated chunk retrieval is structurally inadequate.

One governance requirement at the routing layer that gets missed more often than it should: routing decisions that reach across data sources with different access controls require permission checks at each hop, not just at the entry point. Routing logic that omits this enforcement produces answers derived from data the querying user shouldn't have seen. The compliance problem isn't visible at the surface. It's inside the orchestration, and by the time it surfaces, it's already a problem.

MCP as the Infrastructure for Delivering Encoded Context Across Surfaces

All of the encoding work described above, chunking heuristics, epistemic metadata, query expansion, routing logic, doesn't help if it has to be rebuilt every time a new AI surface is added. That's the integration problem MCP solves, and most teams don't fully appreciate it until they're three deployments in and rebuilding the same logic for the fourth time.

MCP is an open-source standard for connecting AI assistants to external tools and data sources, defining a universal, stateful, bidirectional interface using JSON-RPC 2.0. Any compliant AI client can communicate with any MCP-compatible server. Before MCP, every AI-to-tool integration was built custom. A GitHub integration written for one AI client didn't transfer to another. A Slack integration for one platform didn't transfer to a competitor's. The result was N times M integration surface area for N agents and M tools. MCP collapses this to N plus M, a structural efficiency that becomes critical as enterprise agent counts grow past the point where custom integrations are manageable.

For tacit knowledge pipelines specifically, MCP means the encoded expertise travels with the organization rather than with any single tool. The metadata schema, routing rules, and context layers built through the work described above become durable infrastructure rather than one-off configurations that evaporate when a tool gets replaced.

Adoption has moved quickly. By March 2025, OpenAI had adopted MCP across its platform. Google and Microsoft followed in subsequent months. In December 2025, Anthropic donated MCP to the Linux Foundation's Agentic AI Foundation, removing single-vendor governance risk. According to the Stacklok 2026 software report, 41% of surveyed software organizations were running MCP servers in limited or broad production.

Department-specific MCP servers are the right knowledge delivery unit. A revenue operations team's encoded deal context shouldn't share a server with an engineering team's runbook knowledge. Scoped servers let each domain's expert context reach the agents and users who need it, under appropriate access controls, while keeping context injection focused rather than polluting the window with irrelevant domain knowledge.

Context window constraint at scale is a real architectural concern, not a theoretical one. Each MCP server injects tool definitions into the agent's context window, and past a certain threshold of connected servers, model performance degrades while inference cost increases. Dynamic tool loading via an MCP gateway exposes only the tools relevant to the current conversation rather than all available tools. The gateway functions as a routing layer at the infrastructure level, parallel to the semantic routing described in the previous section, and it's the architectural answer to context bloat before it becomes a performance problem.

Governance, Permissions, and the Registry Problem at Scale

Here is the success failure mode: once a well-designed, expert-encoded RAG pipeline proves its value in one team, adjacent teams rebuild something similar from scratch. The tacit knowledge encoding work gets repeated, inconsistently, with no shared baseline and no way to know what changed or why. I've watched this happen in organizations that were genuinely committed to getting this right, and it still happened, because the infrastructure for preventing it wasn't there.

The solution is a registry, not a shared drive. A shared drive is a storage location. A registry is version-controlled, ownership-attributed, and access-scoped. Every change to a context layer, metadata schema, or routing rule is versioned, so that when an agent's behavior changes unexpectedly, the change can be traced to a specific edit by a specific owner. Ownership sits at the domain level: the team whose expertise is encoded controls updates to that context, not a central IT function that lacks the subject-matter knowledge to validate whether a change is even correct.

This is a quality argument, not a bureaucratic one. Expert knowledge encoded without expert validation is not expert knowledge. It's an approximation that will drift from reality as the domain evolves, without anyone noticing until a consequential answer is wrong.

Permission inheritance is non-negotiable at scale. When a RAG pipeline connects to sources with existing access controls, those controls should propagate automatically to every new agent or surface added downstream. Requiring redefinition of permissions per deployment is inefficient and produces gaps. Someone connects a new agent to a sensitive data source, fails to scope the permissions correctly, and the pipeline surfaces information to users who shouldn't see it. The fix is architectural: access scope as a metadata attribute inherited from the source system, rather than a configuration task for each new integration.

The registry and the permission inheritance model together address the same underlying problem. As an organization's investment in tacit knowledge encoding compounds, the infrastructure holding that investment needs to be as carefully designed as the encoding work itself. Expert knowledge that is captured, structured, and then left unversioned in a folder will diverge, quietly, from what the domain experts actually believe, and nobody will know until it matters.