Building a Production RAG Pipeline — The Decisions Inside Each Component

This is a guide to assembling a RAG pipeline — the components, the decisions inside each component, and the parts of the system that exist outside the happy path.

The audience here is engineers who are going to build and operate this system. The framing is opinionated; so pick the parts that match your constraints.

The pipeline at a glance

A production RAG system is really two pipelines plus a harness around them.

The indexing pipeline runs offline and on a schedule. It ingests source documents, parses them, splits them into retrievable units, generates embeddings, and writes the result to a vector store and a lexical index.

The query pipeline runs online per request. It takes a user query, rewrites or decomposes it, retrieves candidate passages from both indices, reranks them, assembles a prompt, generates a response, and emits telemetry.

The evaluation harness wraps both. It is a golden set, a regression runner, an LLM-as-judge with calibration, and the observability layer that tells you which queries are failing in production.

In outline:

INDEXING PIPELINE
Documents → Parsing → Chunking → Embeddings → Vector + Lexical Indices

QUERY PIPELINE
User Query → Rewrite / Decompose → Hybrid Retrieval → Reranking → Generation

MEASUREMENT HARNESS
Evaluation ↔ Observability ↔ Feedback Loop

Part 1: The indexing pipeline

1. Ingestion and parsing

Ingestion is the most underestimated stage. Whatever quality problems exist in your source corpus get baked into the index, and “garbage in, garbage out” is not a metaphor here: it is a precise description of how retrieval will behave.

Concretely, you need a parser per source type. PDFs are the hardest case and deserve serious attention:

Text-native PDFs: a layout-aware parser (Unstructured, PyMuPDF with layout heuristics, or a commercial extraction service) that preserves reading order, headings, lists, and tables.
Scanned PDFs: OCR quality must be evaluated against the actual scans, languages, handwriting, tables, and accuracy requirements. Tesseract can be adequate for clean, predictable documents; difficult layouts usually justify a vision-language model or dedicated document-understanding service.
HTML: strip navigation, headers, footers, and sidebars before extracting body content. Readability-style extractors get you most of the way; site-specific selectors handle the rest.
Office documents: use the structured representation directly (docx, xlsx) rather than converting through PDF and re-parsing.
Tables: preserve them as structured data where possible. Flattening a table into prose loses the column-row relationships that make the data useful.

Attach metadata at ingestion time: source URI, document title, section path, author or owner, published date, last-modified date, version, status (current, draft, superseded), and any authority signals available. Every one of these is a filter or a ranking input later. You cannot retrofit metadata you did not capture.

Treat modality as an architectural choice, not only a parsing problem. A chart, diagram, slide, or visually structured table can lose its meaning when reduced to extracted text. Current multimodal embedding models can place text and document images in the same retrieval space; Cohere Embed 4, for example, accepts text, images, and mixed text-image inputs. For visually rich corpora, evaluate text extraction, image retrieval, and a combined approach separately rather than assuming OCR alone preserves the answer.

Freshness, authorization, and deletion belong in the indexing design from the start. Every chunk should inherit an access-control scope and a source version. Deleting or superseding a source must remove it from every dense, lexical, cache, and derived-summary surface that can return it. These are the same lifecycle controls a platform should hand every team by default, enumerated on the context lifecycle checklist.

2. Chunking

Chunking is where a surprising number of production accuracy issues originate.

The defaults you see in tutorials — 1000-token fixed-size chunks with 200-token overlap — are a starting point, not an answer. The right strategy depends on document structure:

Structured documents with clear hierarchy (manuals, policies, wikis): split on headings first, then sub-split sections that exceed a max token budget. Carry the heading path as metadata so the retrieved chunk knows where it came from.
Long-form prose (articles, reports): semantic chunking — group sentences that cluster together in embedding space — outperforms fixed-size splits, at the cost of more compute at index time.
Code and config: split on syntactic boundaries (functions, classes, top-level blocks), not lines or tokens.
Conversations and transcripts: split on speaker turns or topic shifts, with a window of preceding context attached.
Tables: represent each row as a chunk with the column headers prepended, plus a separate chunk for the table-level summary.

Below are some patterns that are worth evaluating across chunking strategies:

Parent-child chunking. Index small chunks (200–400 tokens) for retrieval precision, but return the larger parent chunk (1500–3000 tokens) to the model for generation context. The model sees enough surrounding text to interpret the passage correctly.

Contextual chunk headers. Prepend each chunk with a short generated description of what document and section it is from, so the embedding captures topical context the bare chunk would not. This is one part of the approach Anthropic published as Contextual Retrieval; test its impact on your own corpus.

Contextualized chunk embeddings. A newer option is to embed each chunk while the embedding model can see the surrounding document. This preserves global context without generating and storing a separate header for every chunk. Voyage’s contextualized embedding API is one implementation. It is worth evaluating for contracts, manuals, transcripts, and other documents where local passages depend heavily on definitions or context elsewhere. It does not make chunking universally irrelevant: retrieval units, citation boundaries, permissions, updates, and generation context still need deliberate design.

3. Embeddings

Embedding model selection is an empirical decision. Results vary with domain, language, query style, document length, modality, and whether the workload is asymmetric search, clustering, or something else.

Current managed options include OpenAI text-embedding-3-large, the Voyage 4 family, and Cohere embed-v4.0, alongside a changing set of open models. Do not select among them from a public leaderboard alone. Build a corpus-specific retrieval set, measure the quality-cost-latency envelope, and include language, data residency, deployment, and modality requirements in the decision.

Embedding models types:

Dimensionality and representation. Higher-dimensional floating-point embeddings cost more in storage, memory bandwidth, and search time. Some current models support multiple output dimensions or quantized representations. Test the reduction on your retrieval set; do not assume the quality loss is negligible.
Domain. For specialized domains — legal, biomedical, code — a domain-tuned embedding model can outperform a generalist by a meaningful margin. Evaluate before assuming.
Symmetric vs. asymmetric. Some embedding models distinguish between query-side and passage-side encoding. Use the right side at the right stage; mixing them degrades retrieval.
Versioning. Pin the embedding model version and assume an upgrade requires re-indexing unless the provider explicitly guarantees a compatible shared embedding space. Mixing incompatible embedding versions in one index corrupts retrieval.

4. Vector store and lexical index

Run two indices.

A vector index for dense semantic retrieval. Options worth considering:

pgvector on Postgres — best choice when your team already runs Postgres and the corpus fits comfortably (millions of vectors, not hundreds of millions). Transactional, joinable, operationally familiar.
Qdrant, Weaviate, Milvus — purpose-built vector databases. Use when scale or feature requirements (named vectors, payload filtering, hybrid search built in) exceed what pgvector handles cleanly.
OpenSearch / Elasticsearch with vector support — strong choice when you already run one and want a single system for lexical and vector retrieval.
FAISS — embed it as a library when you need maximum control and minimal operational surface. Not a database; you handle persistence and updates yourself.

A lexical index for BM25 or similar. OpenSearch, Elasticsearch, or a Postgres full-text index will all do. Skipping lexical retrieval leaves exact identifiers, codes, names, and specialized terminology unnecessarily dependent on embedding behavior.

Index design choices that matter in production:

Metadata filters as first-class. Most queries should be filtered before similarity search — by tenant, document type, date range, status. Pre-filtering is faster and more accurate than post-filtering a similarity search result set.
Index parameters. HNSW with sensible ef_construction and M values is the default. Tune ef_search per query for the recall/latency tradeoff you want.
Reindex strategy. Have one. Either a blue-green index swap on full reindex, or an incremental update path with deletion handling. Discovering mid-incident that you cannot reindex without downtime is expensive.

Part 2: The query pipeline

5. Query processing

Treat the raw user query and a rewritten query as two candidates. Conversational fragments often need context and expansion; exact identifiers, quoted phrases, error messages, and carefully formed technical queries may be damaged by rewriting. Evaluate when to preserve, supplement, or replace the original.

Three transformations are worth evaluating, sometimes in combination:

Query rewriting. A small, fast model (Haiku-class) expands the query to be more retrieval-friendly — resolving pronouns, expanding acronyms, adding implicit context.

system: Rewrite the user query so it is self-contained and retrieval-ready.
       Preserve entities and intent. Output the rewritten query only.
user:  what about Q3?
       [conversation context: ... discussing 2025 revenue ...]
output: What were the Q3 2025 revenue figures and key drivers?

Multi-query generation. Generate three to five reformulations of the query, retrieve against each, and union the results before reranking. Improves recall for queries that can be phrased many ways.

Decomposition. For multi-hop questions, generate sub-queries, retrieve for each, and pass all results into reranking. A question like “How does our Q3 revenue compare to the guidance we gave in Q2?” decomposes into “Q3 revenue actuals” and “Q2 forward guidance.”

These transformations add a model call, latency, cost, and another place to distort intent. Their value can be large on ambiguous or multi-hop traffic and negative on precise queries. Run the raw query and transformed variants through the same evaluation set, and route only the query classes that benefit.

6. Iterative retrieval for agents

A fixed RAG pipeline performs retrieval once, assembles context, and generates an answer. An agent can use retrieval as a tool: search, inspect the results, identify what is missing, reformulate, and search again. This is useful for investigative and multi-hop work where the next query depends on evidence found in the previous result.

The flexibility creates a new failure surface. The agent can search the wrong branch repeatedly, broaden beyond the user’s authorization, accumulate contradictory context, or spend far more than the task warrants. Bound the loop with a maximum number of searches, latency and cost budgets, authorization filters enforced on every call, and an explicit stop or escalation condition. The general case — why these limits have to be enforced by the runtime and not requested of the model — applies here without modification.

Evaluation changes too. Record and score the retrieval trajectory: the queries issued, sources inspected, evidence retained or discarded, and whether each step moved the task closer to a supported answer. A correct final response reached through accidental or unauthorized retrieval is not a successful run.

7. Hybrid retrieval

Run dense and lexical retrieval in parallel, then fuse the results.

The standard fusion technique is Reciprocal Rank Fusion (RRF):

def rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: -x[1])

RRF is rank-based, not score-based, which sidesteps the calibration problem between dense and lexical scores. It is a strong, simple baseline before investing in a learned fusion model.

Retrieve broadly enough to give the reranker useful recall, but treat candidate count as a tuned parameter. Fifty to one hundred candidates per retriever is a reasonable experiment, not a universal production setting. Corpus size, filter selectivity, reranker limits, latency targets, and query difficulty should determine the final number.

8. Reranking

Reranking is often one of the highest-leverage experiments in a retrieval system, especially when first-stage recall is healthy and precision is poor. It will not rescue missing documents, incorrect permissions, broken parsing, or a first-stage retriever that never finds the answer.

A cross-encoder or model-based reranker takes the query and each candidate passage together and produces a relevance score based on their relationship, not only vector proximity. Managed options include Cohere Rerank and Voyage Rerank; open rerankers are also available. The latency and precision tradeoff varies by model, candidate length, batch size, hosting, and traffic. Benchmark it inside the complete query path.

Practical guidance:

Start by retrieving a wider hybrid candidate set and keeping a small, high-confidence generation set, then tune both values against recall, answer quality, latency, and cost.
Apply a relevance score threshold — if no candidate clears it, the right answer is probably “I do not have information on that,” not whatever the model would confabulate from low-relevance context.
Reranker latency stacks. Budget it deliberately and consider running it asynchronously where the UX allows.

9. Prompt assembly and generation

The prompt you send to the generation model is the contract between the retrieval pipeline and the user-visible answer. Treat it as code.

A defensible structure:

system: You answer questions strictly from the provided sources.
        Cite sources inline as [1], [2], etc.
        If the sources do not contain the answer, say so explicitly.
        Do not use prior knowledge beyond the sources.

user:   Question: {user_query}

        Sources:
        [1] {source_1_title} ({source_1_date})
        {source_1_passage}

        [2] {source_2_title} ({source_2_date})
        {source_2_passage}
        ...

Things worth getting right:

Citation as a first-class output. Require the model to cite sources by index, then map back to URIs in the response. Without citations, you have no grounding signal for users or for eval.
Source metadata in the prompt. Title, date, section — gives the model a way to prefer current sources and acknowledge conflicts.
Refusal as a valid output. The system prompt should explicitly authorize “I do not have information on that.” A model that always answers is a model that hallucinates when retrieval misses.
Context budget management. Large context windows increase the available design space; they do not make every additional passage useful. Test context size and ordering directly. A smaller set of well-ranked, non-duplicative evidence often beats a larger set of marginal passages.
Prompt caching. The system prompt and any static instructions should be cached. On Anthropic and OpenAI APIs, this is a meaningful cost reduction at production scale.

Part 3: The harness around the pipeline

10. Evaluation

Without evaluation, every change is a guess. The eval system has three layers.

Golden set. A versioned collection of queries with expected answers and expected source documents. Cover the long tail intentionally: ambiguous queries, multi-hop questions, queries with no good answer, queries where the corpus contains conflicts. A hundred well-chosen examples is more useful than a thousand cherry-picked ones.

Separated metrics. Measure retrieval and generation independently:

Retrieval: recall@k, mean reciprocal rank, and a hit-rate on the expected source documents. If retrieval fails, generation cannot succeed.
Generation: faithfulness (does the answer follow from the cited sources), correctness (is the answer right), and citation accuracy (do the citations actually support the claims). LLM-as-judge with periodic human calibration is the standard approach.

Regression runs on every change. The eval harness should run on every prompt change, model swap, chunking adjustment, or reranker update. CI-style integration is the right pattern. A change that improves the overall score but regresses a specific query class is worth knowing about before deployment. Funding this harness before the sophisticated retrieval technique is the argument in the executive companion to this post, and it is the single most common thing missing from a stalled system.

11. Observability and feedback

Production telemetry is the part of the system that tells you which questions you failed to anticipate.

The minimum useful instrumentation per query:

The raw query, the rewritten query, and any sub-queries
For iterative retrieval, every search step, inspected source, and stop or escalation decision
The retrieved candidates with scores from each retriever
The reranked top-k passed to generation
The final prompt, the generated answer, and the citations
Latency per stage
User feedback signals — thumbs, follow-up queries, abandonment

Sample and review systematically. The queries that produce low-confidence answers, low reranker scores, or thumbs-down feedback are where the next round of golden-set additions and corpus improvements come from. A RAG system without this feedback loop drifts.

12. Cost, latency, and caching

A few patterns to evaluate:

Embedding cache. Cache embeddings by content hash so unchanged chunks do not need to be re-embedded during reindexing.
Query embedding cache. For high-traffic applications, cache query embeddings keyed by normalized query string when measurement shows enough repetition to justify the complexity.
Result cache. For deterministic queries (FAQs, common lookups), cache the full response with a short TTL. Invalidate on corpus updates.
Model tiering. Use a smaller model for query rewriting and decomposition; reserve the largest model for generation. The cost difference compounds at scale.
Streaming. Stream tokens when the response format and UX allow it. This improves time to first visible output even though total generation latency is unchanged.

What to build vs. what to buy

Frameworks, managed retrieval services, and custom components are all viable production choices. The right boundary depends on where the application’s behavior is differentiated and how much control the organization needs over quality, security, residency, observability, and operations.

Buy or use a managed service when its quality is measurable, its security and residency model fits, it exposes the telemetry and controls you need, and operating the underlying retrieval stack is not a competitive advantage.
Use libraries for mature integrations such as file parsing, model SDKs, vector-store clients, and reranker APIs when their behavior is observable and replaceable.
Own the behavior that differentiates the system: corpus policy, access enforcement, evaluation data, quality thresholds, failure handling, and the orchestration decisions that users experience.
Own more of the stack when managed abstractions prevent necessary tuning, create unacceptable lock-in, obscure failure diagnosis, or cannot meet deployment and compliance constraints.

Do not confuse abstraction with immaturity. A managed system can support a consequential workload when its controls and evidence are adequate; a custom stack can be unsafe and unreliable when the team cannot operate it. Evaluate the operating model, not the category label.

Closing

A RAG pipeline that works in production is a dozen unremarkable components, each tuned to its job, wrapped in a measurement harness that reports when one of them regresses. Very little of that tuning transfers between corpora, which is why nearly every recommendation above ends by telling you to test it against your own.

Therefore the order that matters most is the order of investment. The golden set and the regression runner earn their keep before contextual embeddings or a learned fusion model do, since they are what tell you whether the sophisticated component helped at all. Teams that defer the harness still make every decision in this post; they simply make them without evidence and learn which ones were wrong from their users.

Every component here is well documented. What separates the systems built from them is the evidence behind each choice.

If you are building or operating a production RAG pipeline and want a technical review or hands-on help with retrieval, evaluation, or the surrounding infrastructure, get in touch.