RAG Architecture Explained: Building AI Apps With Your Own Data

Large language models (LLMs) are powerful, but their knowledge is generic and frozen in time. They know nothing about your company's internal documentation, your product specifications, or your customer support history. Retrieval-Augmented Generation (RAG) is the dominant architectural pattern for solving this problem, allowing you to build applications that reason over your private data with the power of an LLM.

However, moving from a simple proof-of-concept to a production-grade RAG system reveals significant complexity. Naive implementations often suffer from irrelevant search results, unhelpful answers, and an inability to handle user queries that fall outside the knowledge base. For business leaders and CTOs, understanding the components and trade-offs of a robust RAG architecture is essential for budgeting, planning, and ultimately delivering value.

This guide explains the end-to-end RAG pipeline, from data preparation to evaluation, based on our experience building and scaling these systems for clients. We will cover the specific engineering choices that separate a fragile demo from a reliable production service. While RAG is often presented as an alternative to fine-tuning, the two can be complementary; we explore this choice in more detail in our guide, Fine-Tuning vs RAG: Which AI Approach Should Your Company Use?.

What RAG Solves—And What It Does Not

At its core, RAG addresses the "knowledge gap" of general-purpose LLMs. It works by retrieving relevant snippets of information from your own data source and providing them as context to the LLM along with the user's query. The LLM is then instructed to synthesise an answer based only on the provided information.

This approach offers several powerful advantages over other methods of customising LLM behaviour:

1. Reduces Hallucinations: By grounding the model in specific, verifiable documents, RAG drastically reduces the model's tendency to invent facts. The answer is constrained by the evidence provided.
2. Enables Citations: Because you know exactly which documents were used to generate an answer, you can include citations. This is non-negotiable for enterprise use cases where trust and verifiability are paramount.
3. Lower Cost and Faster Iteration: Compared to fine-tuning a model, updating a RAG system's knowledge is as simple as updating the document index. This is orders of magnitude cheaper and faster than retraining, allowing your AI application's knowledge to stay current.
4. Uses Existing Data: RAG works with your existing documents, wikis, and databases. It doesn't require curated datasets of thousands of prompt-completion pairs, which are a prerequisite for fine-tuning.

However, RAG is not a magic bullet. Its effectiveness is entirely dependent on the quality of the retrieval step. If you can't find the right information, the LLM can't answer the question correctly. RAG does not solve:

• Poor Data Quality: If your source documents are inaccurate, contradictory, or poorly structured, the RAG system will inherit these flaws. The "garbage in, garbage out" principle applies with force.
• Reasoning Beyond the Data: RAG is designed to synthesise information, not to perform complex, multi-step reasoning that isn't explicitly laid out in the text.
• Learning a New 'Style' or 'Persona': RAG provides facts, not style. If you need the model to adopt a specific, complex voice or format consistently, fine-tuning might be more appropriate.

Understanding these boundaries is the first step in designing a successful RAG architecture.

The RAG Pipeline, Stage by Stage

A production RAG system is best understood as two distinct processes: an offline Ingestion Pipeline that prepares the data, and an online Inference Pipeline that answers user queries. Many early-stage projects conflate the two, leading to performance bottlenecks and maintenance headaches.

========================================================================================
| Ingestion Pipeline (Offline)                                                         |
========================================================================================

 [Source Docs] -> Load -> Split (Chunking) -> Embed -> [Vector & Keyword Index]
   (PDF, HTML,                                            (e.g., PostgreSQL w/pgvector,
    Markdown...)                                             Elasticsearch, Pinecone)

========================================================================================
| Inference Pipeline (Online)                                                          |
========================================================================================

                /------------------ [Keyword Index] ---\
               /                                         \
[User Query] -> --- Hybrid Search -> Rerank -> Augment -> Generate -> [Answer + Citations]
               \                                         /           (LLM Call)
                \------------------ [Vector Index] ----/

========================================================================================

Ingestion Pipeline

This process runs asynchronously whenever your source knowledge base is updated.

1. Load: Your raw documents (e.g., PDFs, Confluence pages, Markdown files, database records) are loaded. This step often involves connectors and extractors that can handle various formats, pulling out the core text and preserving crucial metadata like titles, authors, and creation dates.
2. Split (Chunking): The extracted documents are broken down into smaller, manageable pieces, or "chunks." This is one of the most critical steps for retrieval quality, and we dedicate a full section to it below.
3. Embed: Each chunk is passed through an embedding model (e.g., text-embedding-3-large or an open-source model like E5-large-v2). This model converts the text into a numerical vector that captures its semantic meaning.
4. Store/Index: The chunks and their corresponding vectors are stored in a vector database. Crucially, we also index the raw text of the chunks in a traditional full-text search index (like BM25). This dual-index approach is the foundation of hybrid search. This is a classic data engineering challenge, requiring a robust and scalable architecture.

Inference Pipeline

This process runs in real-time for every user query.

1. Query: The user submits a natural language query.
2. Hybrid Search: The query is used to search both the vector index (for semantic similarity) and the keyword index (for lexical matches). The results from both searches are combined.
3. Rerank: The top N results from the hybrid search (e.g., top 20-50 documents) are passed to a more powerful, but slower, reranking model. This model re-evaluates the relevance of each chunk to the specific query and outputs a new, more accurate ranking.
4. Augment: The top K reranked chunks (e.g., top 3-5) are selected. Their text is formatted and concatenated into a single block of context, which is then inserted into a prompt template along with the original user query.
5. Generate: The final prompt is sent to an LLM (e.g., GPT-4o, Claude 3 Opus). The model is instructed to generate an answer based only on the provided context and to cite its sources.

This multi-stage process, particularly the inclusion of hybrid search and a reranker, is a hallmark of a mature RAG system. It systematically funnels a large corpus of documents down to the few most relevant passages required for a high-quality answer.

Chunking Strategies Compared

How you split your documents into chunks has an outsized impact on retrieval performance. If a chunk is too small, it lacks context. If it's too large, it adds noise and increases LLM processing costs. There is no single "best" strategy; the choice depends on your document structure and content.

In our engagements, we typically start with recursive character splitting and iterate towards more semantic methods as we evaluate performance.

For a European insurer, we started with a recursive chunking strategy for their 10,000+ policy documents. This gave us a reasonable baseline. However, we found that retrieval for complex liability clauses was poor. We then moved to a semantic chunking approach, which grouped related sentences about specific exclusions and conditions together, even if they were separated by other text. This improved retrieval precision for complex queries by over 20% in our offline evaluations, justifying the increased ingestion complexity.

Hybrid Search and Reranking

A common mistake in early RAG projects is relying solely on vector search. Vector search is excellent at finding documents that are semantically similar to a query, but it can struggle with queries that hinge on specific keywords, product codes, or acronyms.

For example, a user might search for "project GOLUX-2024-Q3". A pure vector search might return documents about "Q3 planning" or "2024 project goals" because they are semantically related. However, it might miss the one document with the exact identifier "GOLUX-2024-Q3".

This is where hybrid search comes in. It combines the results from two different search algorithms:

1. Vector Search: Finds semantically related documents.
2. Keyword Search: Typically using an algorithm like BM25, this finds documents with exact keyword matches.

The results from both are combined using a fusion algorithm (like Reciprocal Rank Fusion) to produce a single, more robust list of candidate documents.

But we can do better. This initial list of candidates (e.g., the top 50 documents) is optimised for recall—we want to be sure the correct answer is somewhere in the list. The next step is to optimise for precision using a reranker.

A reranker is typically a smaller, specialised model (a cross-encoder) that takes the query and a single candidate chunk as input and outputs a relevance score. By individually scoring each of the top 50 candidate chunks against the query, it produces a much more accurate final ranking than the initial search.

Why not use the expensive reranker on the whole database? Latency and cost. Reranking is computationally intensive. The two-stage process—a fast but "coarse" retrieval followed by a slow but "fine" reranking—provides the best balance of accuracy, cost, and speed. Our experience shows that adding a reranker is one of the highest-leverage improvements you can make to a RAG system, often boosting top-5 retrieval accuracy by 10-30 percentage points.

Grounding, Citations, and Refusal Behaviour

For an enterprise AI application, being correct is only half the battle. It must also be trustworthy and safe.

Grounding refers to instructing the LLM to base its answer exclusively on the context provided. This is achieved through system prompts. A typical grounding instruction looks like this:

"You are a helpful assistant. Answer the user's question based only on the provided context. If the information is not in the context, say 'I do not have enough information to answer this question.' Do not use any of your prior knowledge."

This instruction is the primary defence against hallucination.

Citations are the mechanism for proving that the model has followed the grounding instruction. By mapping sentences in the generated answer back to the source chunks, you can provide users with clickable links to the original documents. This builds trust and allows for easy verification. Implementing robust citations is a non-trivial engineering task, often involving string matching or asking the LLM to output structured data linking its claims to source IDs.

Refusal Behaviour is the explicit implementation of the "I don't know" response. Without it, models have a tendency to "try their best," often by twisting the provided context into a weak or incorrect answer. Engineering proper refusal is critical. We often test this by asking questions we know are not in the knowledge base. A good system refuses cleanly; a bad one speculates. For internal helpdesks, like those we've helped build for clients, clear refusal is vital for preventing the spread of misinformation. You can read more about our experiences in How Companies Build Internal AI Assistants That People Actually Use.

Evaluation: Retrieval Metrics vs. Answer Metrics

"It seems to work well" is not an acceptable evaluation strategy. A mature RAG system requires a quantitative, automated evaluation framework. We separate evaluation into two categories: retrieval quality and generation quality.

You need a "golden set" of evaluation questions—typically 50-100 real-world questions with hand-annotated ideal answers and source documents. This set is used to score the system's performance every time you make a change to the pipeline.

Building this evaluation framework is a key part of any serious AI engineering project. Without it, you are flying blind, unable to know if changes to chunking, embedding models, or prompts are making the system better or worse.

Cost, Latency, and Caching

A production RAG system must meet business requirements for cost and performance. Let's model a hypothetical scenario for a system handling 100,000 queries per month.

Worked Example: Cost per 1,000 Queries (2026 Estimate)

Assume we're building a system for a Series A logistics platform to query their internal operational manuals.

• User Base: 200 support agents.
• Query Volume: ~5,000 queries per day, or ~100,000 per month.
• Technology Stack:
  • Embedding Model: text-embedding-3-small
  • Reranker: Cohere Rerank
  • LLM: gpt-4o-mini
  • Vector DB: Managed PostgreSQL with pgvector

Here's a breakdown of the inference cost per 1,000 queries:

1. Query Embedding: (1,000 queries) * (€0.00002 / 1k tokens) * (avg 30 tokens/query) = €0.0006 (negligible)
2. Vector DB: Cost is primarily fixed based on instance size. A suitable instance might be ~€300/month. For 100k queries, this is €3.00 per 1,000 queries.
3. Reranking: (1,000 queries) * (50 candidates/query) * (€1.00 / 1M units) * (1 unit/document) = €0.05 (very cheap)
4. LLM Generation: This is the dominant cost.
   • Input (Context): 1,000 queries * 4 chunks * 400 tokens/chunk * (€0.15 / 1M tokens) = €0.24
   • Output (Answer): 1,000 queries * 200 tokens/answer * (€0.60 / 1M tokens) = €0.12
   • Total LLM Cost = €0.36

Total cost per 1,000 queries ≈ €3.41 Total monthly cost ≈ €341

These costs are falling, but the key takeaway is that infrastructure (the vector DB) and the final LLM call are the main drivers. The "smart" components like reranking are surprisingly inexpensive.

Latency is the other side of the coin. A user expects a response in 2-4 seconds.

• Query Embedding: ~100ms
• Hybrid Search (DB): ~150ms
• Reranker API call: ~400ms
• LLM API call (streaming):
  • Time to First Token: ~500ms
  • Total Time: ~2,000ms

Total latency ≈ 3.15 seconds. This is acceptable but on the edge.

Caching is essential for reducing both cost and latency. We recommend a multi-layer caching strategy:

1. Exact Match Cache: Cache the final answer for identical user queries. High hit rate for common questions ("what are our opening hours?").
2. Retrieval Cache: Cache the retrieved and reranked document list for a given query. If a similar query arrives, you can skip the expensive search and rerank steps and go straight to the LLM.

Seven RAG Mistakes We See in Production

Transitioning a RAG prototype to a robust service is where most teams stumble. We frequently encounter the same set of anti-patterns.

1. Naive Chunking and Forgetting Metadata: Using a simple fixed-size chunker and discarding document titles, authors, or dates. This loses valuable context that could be used for filtering or improving relevance.
2. Relying Only on Vector Search: As discussed, this fails on keyword-sensitive queries and is a marker of an immature architecture.
3. No Reranker: Shipping a system without a reranking step leaves significant performance on the table for a relatively low cost.
4. No Automated Evaluation: Teams manually test a few queries and deploy. This makes it impossible to measure regression or improvement over time.
5. Forgetting to Engineer Refusal: The system tries to answer everything, leading to confident but wrong answers on out-of-domain questions and eroding user trust.
6. Ignoring the Data Ingestion Pipeline: Treating ingestion as a one-off script. A production system needs a robust, observable, and maintainable pipeline to keep the knowledge base current, which is a core data engineering discipline.
7. Underestimating the "Last Mile" Problem: The final output quality is heavily dependent on prompt engineering, citation logic, and user interface design. Simply showing a wall of text is not enough. For more on building helpful AI tools, see our analysis of successful internal AI assistants.

Frequently asked questions

What is RAG?

Retrieval-augmented generation, or RAG, is an architectural pattern for AI applications. It connects a large language model to your company's own data sources. The process works by first retrieving relevant passages from your content and then providing them to the language model as context. This grounds the model's response in your specific data, ensuring the answers it generates are based on facts from your documents rather than the model's generic, pre-trained knowledge.

Does RAG stop hallucinations?

It reduces them substantially but does not eliminate them entirely. When the retrieval system finds the correct information and the model is strictly instructed to refuse to answer without evidence, hallucinations become rare. However, if the retrieval step fails or the source documents themselves are ambiguous, the model can still misinterpret the context or "fill in the blanks". This is precisely why a robust evaluation framework, faithfulness metrics, and user-facing citations are mandatory components of any production RAG system.

Key takeaways

• RAG is the dominant architecture for building LLM applications that reason over private, up-to-date data. It reduces hallucinations and enables citations.
• A production-grade RAG system is a multi-stage pipeline including ingestion (load, chunk, embed, index) and inference (search, rerank, augment, generate).
• Chunking strategy is critical. While recursive character splitting is a good start, semantic or agentic chunking can yield significant performance gains for complex documents.
• Do not rely on vector search alone. A hybrid search approach combined with a reranker is essential for state-of-the-art retrieval accuracy.
• Trust is built through engineering. Grounding prompts, citation generation, and robust "I don't know" refusal behaviour are non-negotiable features.
• You cannot improve what you cannot measure. A quantitative evaluation framework covering both retrieval and generation quality is mandatory for iterating on your system.

Building a RAG system that is reliable, accurate, and cost-effective requires careful architectural choices and a deep understanding of the trade-offs at each stage of the pipeline. It is a complex software system, not just a simple API call.

If your team is planning to build on this architecture and you need a second opinion on your design, our senior engineers can provide a thorough assessment to help you avoid common pitfalls and accelerate your path to production.