Designing a Production RAG System: Full Architecture Walkthrough

Document ingestion pipeline, retrieval layer, reranker, answer policy, eval loop, and monitoring — all the decisions you need to make before you ship.

Building a RAG demo takes a weekend. Building a RAG system that works in production — one that handles messy documents, ambiguous queries, evolving knowledge bases, and real users who break things — takes months of iteration. This is the full architecture walkthrough: every component, every decision, every failure mode.

The hardest part of RAG is not the vector search. It's everything around it: document pipelines, index freshness, eval, observability. The retrieval itself is almost the easy part.

System overview

A production RAG system has four major subsystems: the data pipeline (ingestion and indexing), the retrieval pipeline (query-time), the generation pipeline (LLM call), and the operations layer (observability, evals, freshness). Most tutorials only show the middle two.

The data pipeline — where most teams underinvest

Document ingestion

Documents arrive in every format: PDFs with tables, HTML pages with nav menus, Word docs with headers, Confluence pages with embedded images. Each format needs a dedicated parser. A bad parser corrupts the chunk before it ever reaches the model.

• PDFs: use PyMuPDF or pdfplumber — not PyPDF2 (poor table handling). For scanned PDFs, you need OCR (Tesseract, AWS Textract, or Document AI).
• HTML: strip navigation, footers, ads before chunking. Beautiful Soup works; Trafilatura extracts article content better.
• Word/Excel: use python-docx and openpyxl. Preserve table structure — tables flattened to prose lose most of their meaning.
• Images and diagrams: either skip them (document clearly), extract captions, or use a vision model to generate descriptions.

Chunking strategy selection

Embedding model choice

Use MTEB leaderboard as a starting point but benchmark on your domain. Production choices in 2025: OpenAI text-embedding-3-large (3072 dims, best general-purpose), Cohere embed-v3-english (1024 dims, strong on long documents), nomic-embed-text (768 dims, runs locally, surprisingly competitive). For multilingual: multilingual-e5-large or Cohere embed-v3-multilingual.

Index freshness — the silent killer

Knowledge goes stale. A policy updated six months ago is still in your index as the canonical version unless you actively expire it. Every production RAG system needs an index freshness strategy:

• Track last-modified date for every source document in chunk metadata
• Run a freshness monitor daily: flag chunks whose source has changed since indexing
• Set TTLs on chunks from volatile sources (news, pricing pages, live docs)
• When a document is updated, delete all chunks with that document ID before re-indexing

The retrieval pipeline — advanced patterns

Query transformation

Raw user queries are often poor retrieval inputs. Users phrase things colloquially, with typos, with missing context. Three transformations that consistently improve retrieval:

• Multi-query expansion: generate 3 paraphrases of the query, retrieve for all, deduplicate. Adds one LLM call, lifts recall by 15–30%.
• HyDE (Hypothetical Document Embeddings): generate a hypothetical answer to the query, embed that instead of the query. The hypothetical answer's vocabulary matches document vocabulary better than a short question.
• Step-back prompting: rewrite a specific query to its more general form before retrieval. 'What's the max refund for order #A1234?' → 'What is the refund policy?' — much better retrieval.

Reranking

First-stage retrieval (vector search) optimises for speed, not precision. A cross-encoder reranker scores each query-chunk pair together — much more accurate but O(n) forward passes. The production pattern: retrieve top-20 cheaply, rerank to top-5 accurately, send top-3 to the model. Cohere Rerank, BGE-reranker-large, and Jina Reranker are the common choices.

import cohere

co = cohere.Client()

def rerank(query: str, chunks: list[str], top_n: int = 5):
    results = co.rerank(
        query=query,
        documents=chunks,
        top_n=top_n,
        model="rerank-english-v3.0",
    )
    return [chunks[r.index] for r in results.results]

# Usage: first retrieve 20, then rerank to 5
initial_chunks = vector_search(query, top_k=20)
precise_chunks = rerank(query, initial_chunks, top_n=5)

Context assembly

The context you inject into the LLM prompt is not just a paste of the top-K chunks. For production quality: order chunks by relevance (most relevant first AND last — lost-in-middle mitigation), add source attribution metadata, compress chunks that exceed a token budget using LLMLingua or selective sentence removal, and always include a 'no relevant information found' fallback instruction.

The generation pipeline

The LLM is the last component, but prompt design here matters more than model choice for most applications. Three things that dramatically improve generation quality:

• Citation instruction: 'When answering, cite the source by referring to its position: [Source 1], [Source 2]'. Forces the model to ground claims.
• Uncertainty instruction: 'If the provided context does not contain enough information to answer the question, say so explicitly. Do not extrapolate.' Reduces hallucinations on missing-context queries.
• Conflict instruction: 'If sources disagree, present both perspectives and note the disagreement.' Prevents silent resolution of conflicts.

The operations layer

Observability — what to instrument

Evaluation pipeline

A RAG eval pipeline has three test sets: a golden set (hand-annotated query/answer pairs), a regression set (past failures — every incident adds an example), and a synthetic set (LLM-generated Q&A over your documents, cheap to create at scale). Run all three on every significant change to chunking, embedding model, retrieval config, or prompt.

Architecture sizing guide

Build a production RAG system in RAG Lab →: Configure every component of the RAG pipeline and run eval comparisons between configurations.