Enterprise RAG: Architecture, Building Blocks and Pitfalls

Retrieval-augmented generation has become the default way to point a language model at company documents. A demo takes an afternoon; a system people trust with real questions, real permissions and messy files takes considerably more. This guide maps every building block of an enterprise RAG architecture, shows which variant fits which situation and lists the pitfalls that sink most projects. Current as of October 2026.

What is RAG, and what do companies use it for?

RAG is a pattern in which a system first retrieves relevant passages from your own sources and then gives them to a language model as the basis for its answer. The model itself stays unchanged; new knowledge arrives through the search index. In companies, RAG mostly answers questions about manuals, policies, tickets and contracts, with citations.

The term comes from a 2020 paper by Lewis et al., which paired a language model whose knowledge lives in its weights with a searchable text index and a retriever that fetches matching passages. The core idea has barely changed since; the engineering around it has changed a lot. For a short definition, see the glossary entry on retrieval-augmented generation.

Common enterprise use cases:

• Internal help desk: travel policy, HR rules, quality handbook, IT how-tos.
• Field and technical service: fault descriptions in past tickets, maintenance procedures, spare-part lists.
• Sales and quotation: earlier proposals, scope descriptions, product data sheets.
• Procurement and legal: comparing clauses across framework agreements and terms of supply.

RAG is rarely the product. It is usually one component inside an assistant or an AI agent. For what that looks like in a business function, see AI agents for knowledge management.

How does a RAG pipeline work?

A RAG system has two paths. The indexing path ingests documents, splits them and stores them in a search index. The query path takes a question, retrieves and ranks passages and passes them to the model. Around both sit access control, evaluation and operations, and each component has failure modes that only show up when everything runs together.

Ingestion: connectors and parsing

First decide which sources you connect: SharePoint libraries, file shares, the ticketing system, a wiki, attachments in your ERP. Each source needs a connector that brings along content, metadata (author, date, version, location) and access rights.

Then files become text, and this is where many systems quietly fail. A PDF full of tables, a scanned delivery note and a spreadsheet price list each need a different parser. A table that turns into a jumble of numbers during conversion cannot be retrieved correctly later, however good your search is. Scanned material needs OCR, and its output deserves spot checks.

Chunking: splitting documents into passages

Whole documents are too long for precise retrieval, so they are split into chunks. Splitting along the document's own structure, at headings, paragraphs and tables, usually beats cutting at a fixed character count.

The catch is lost context. A chunk reading "The notice period is 14 days" means little without the heading and document title above it. Anthropic describes a remedy called contextual retrieval: before indexing, each chunk gets a short, model-generated note explaining where it sits in the document. In Anthropic's own tests, reported in the same post, contextual embeddings plus contextual BM25 reduced top-20 retrieval failures by 49%, and adding reranking brought the reduction to 67%. Those are vendor figures on vendor test data; measure on your own corpus.

Embeddings and the index

An embedding model turns each chunk into a vector that represents its meaning, so passages about the same thing sit close together even when they use different words. Alongside the vector index you want a conventional keyword index, because the two approaches cover each other's blind spots.

One operational detail people forget: switching embedding models means re-embedding the entire corpus. Pin the model and its version deliberately and store that information with the index.

Retrieval: hybrid beats vector-only

Vector search finds conceptually similar text but often misses exact terms. Microsoft's Azure AI Search documentation lists product codes, specialised jargon, dates and people's names as cases where keyword search does better. Hybrid search runs both queries in parallel and merges the result lists.

A widely used merge method is reciprocal rank fusion (RRF). Each document scores 1/(rank + k) in every list it appears in, and the scores are summed; Microsoft's RRF documentation notes that a small constant such as k = 60 works well. Documents that rank high in both lists rise to the top. If your users search for part numbers, standards or error codes, hybrid retrieval is close to mandatory.

Reranking and context assembly

A reranker takes the top candidates and scores each one against the question more carefully. It is slower than first-stage retrieval but only runs on a few dozen passages, and it often improves the ordering noticeably.

Then you assemble the context. More is not automatically better: Liu et al. found that models use information at the beginning and end of a long context better than information in the middle. A handful of well-ordered, cited passages usually helps more than twenty mediocre ones.

Generation with citations

The model receives the question, the passages and clear instructions: answer only from the sources, cite a source for every claim, and say plainly when the sources do not contain the answer. Citations are not decoration. They are the main way users can check what the system tells them.

Which RAG variant fits which situation?

There is no single RAG design. The range runs from a single search followed by an answer to agents that decide for themselves when and where to look. The right choice depends mainly on corpus size, question types, the permission model and how costly a wrong answer is. Use the map below as a starting point.

No RAG at all. In its contextual retrieval post, Anthropic suggests that a knowledge base under roughly 200,000 tokens can often simply be included in the prompt in full. That is a more honest design than a retrieval pipeline searching through a dozen files.

Classic RAG. One retrieval per question, fixed pipeline. Strong for look-up questions whose answer sits in one or two passages. Weak for questions such as "Which suppliers announced price increases over the past three years?", where the answer is scattered across many documents.

Agentic RAG. Retrieval becomes a tool the agent calls on its own. It rewrites queries, searches several sources and checks whether what it found is enough. That improves coverage and flexibility at the cost of latency and less predictable behaviour. A dedicated guide on agentic RAG covers when the extra effort pays off.

Permission-aware RAG. As soon as not everyone may read everything, retrieval has to respect the asking user's rights before the model sees anything, not afterwards. A separate guide covers how to carry SharePoint and file-share permissions through to the index.

Fine-tuning instead of, or alongside, RAG. Fine-tuning shapes how a model writes and behaves. It does not reliably teach the model what is in this week's documents. A separate comparison weighs the two in detail.

What do you need before building enterprise RAG?

The most important prerequisite is not technology but a document base you can rely on: current, free of conflicting versions and with clear ownership. Beyond that you need access to source systems including permission data, a place to run the index and the model, and subject-matter experts who supply test questions and judge answers.

Data

• A bounded starting corpus, such as the maintenance manuals for one product line, rather than "the whole file share".
• Clarity on which version is valid. With three versions of the same policy in the index, the system may well quote the wrong one.
• Metadata such as effective date, document type and owner that retrieval can filter on.

Systems

• Read access to source systems through APIs that also expose permissions.
• A search engine that combines keyword and vector search and supports metadata filters.
• A decision on where the model runs: a public API, a cloud region of your choice, or your own servers. Our page on sovereign AI explains what running on your own infrastructure involves.

Organisation

• Domain experts who provide realistic questions and correct answers.
• A content owner who retires outdated documents.
• Early involvement of works councils or employee representatives wherever usage data is logged.

How should you build a RAG system, step by step?

Work incrementally: define questions and success criteria first, process a small corpus properly, measure, improve, and only then expand. Teams that start with the entire file share can rarely trace a bad answer back to its cause.

1. Collect questions. Ask domain experts for 50 to 100 real questions, each with the expected answer and the passage it comes from. This test set is your most valuable asset.
2. Pick a starting corpus. One area, one source, clear versions.
3. Build the indexing path. Parsers per file type, structure-aware chunking, metadata and permissions carried along.
4. Set up hybrid retrieval. Keyword plus vectors, merged for example with RRF, optionally followed by a reranker.
5. Measure retrieval and answers separately. First check whether the right passage appears among the results, then judge the answer. Frameworks such as Ragas provide metrics including context recall, context precision and faithfulness, the last of which scores how well an answer's claims are supported by the retrieved context.
6. Fix the right layer. Missing passages point to parsing, chunking or retrieval; wrong answers despite the right passage point to prompting or the model.
7. Pilot with real users. A feedback button, a log of unanswered questions, a weekly review.
8. Expand. Add new sources only once the test set passes consistently for the existing ones.

In our own operations, step 5 has made the biggest difference. Without separating retrieval from generation, teams end up debating models when the real problem is a PDF converter mangling tables.

What are the limits and risks of RAG?

RAG makes answers easier to verify, not error-free. The usual problems are poor document processing, outdated or conflicting content, missing access control and manipulated content. RAG also needs upkeep: an index that does not keep pace with its sources serves stale knowledge within months.

Wrong or missing passages. If retrieval misses the right chunk, the model answers from whatever it received, or from general knowledge, and the answer still sounds plausible. Countermeasures: instruct the model to decline when the sources are insufficient, and require a citation for every statement.

Stale and contradictory content. RAG exposes the state of your document management without mercy. Without version and validity metadata, the system cannot tell which document applies.

Access control. The OWASP entry LLM08:2025 "Vector and Embedding Weaknesses" names unauthorised access, leakage between tenants or user groups, and poisoned knowledge sources as key risks, and recommends a permission-aware vector store with fine-grained access controls. One shared index for everyone, with filtering only in the user interface, is not enough.

Indirect prompt injection. Documents can contain instructions the model then follows, for example hidden text in a supplier's PDF. OWASP LLM01:2025 Prompt Injection describes exactly this route via retrieved content. It becomes dangerous when the agent can take actions rather than just answer.

Cost and latency. Contextualisation, reranking and multi-step agentic retrieval improve results but add compute and model calls. Measure which component actually helps on your data before keeping it.

Legal obligations. Where personal data ends up in the index, data protection rules apply as they do to any other processing. A RAG system can be operated in line with the GDPR once storage location, deletion and access are settled. This is general orientation, not legal advice.

How do you get started with enterprise RAG?

Start with one well-defined question from one business function, a bounded set of documents and a test set of real questions. Expand sources, user groups or agent capabilities only once retrieval and answers are measurably stable on that first corpus.

In practice, the first weeks look like this:

• Choose a use case where a wrong answer is annoying but not dangerous, such as internal policy questions rather than safety instructions.
• Map your sources and their permission model before writing any code.
• Decide whether data may leave the company, because that drives the choice of model and infrastructure.
• Build the test set and re-run it after every change.

To see how RAG fits into agents for service, sales or quotation, read the overview of AI agents for companies. More background articles are collected in the guides section and the glossary.