RAG Tutorial: Build a Production-Ready Retrieval-Augmented Generation App

Overview

Retrieval-augmented generation, or RAG, combines a search system with a language model. Instead of asking a model to answer from its general training alone, the application retrieves relevant passages from a managed knowledge source and includes them in the model request. The model then uses that context to produce an answer.

A typical RAG application has six stages:

1. Ingestion: collect documents and record their source, ownership, timestamps, and access rules.
2. Parsing: extract usable text while preserving headings, lists, tables, and other meaningful structure where possible.
3. Chunking: divide documents into passages that are large enough to carry meaning but focused enough to retrieve accurately.
4. Embedding and indexing: represent chunks for semantic search and store them in a vector index, often alongside keyword-search fields and metadata.
5. Retrieval: use the user’s question to select and rank candidate passages, optionally applying filters or a reranking step.
6. Generation: provide the selected context to the model with instructions about evidence, citations, uncertainty, and output format.

These stages should be observable independently. A fluent answer does not prove that the right source was retrieved, and a good search result does not guarantee that the model used it correctly. For a broader implementation walkthrough, see How to Build a Reliable Retrieval-Augmented Generation Application.

Start with a narrow use case and a defined answer contract. For example, a support assistant might be required to answer only from approved product documentation, cite the relevant sections, and say that it cannot find an answer when the evidence is insufficient. This is more testable than a general instruction to “be helpful.”

Maintenance cycle

RAG quality depends on both application code and the knowledge collection. Treat the system as a maintained product rather than a one-time integration. A simple maintenance cycle can be organized into four recurring activities.

1. Inspect the knowledge source

Check whether documents are current, duplicated, incomplete, or incorrectly marked as available. Preserve metadata such as document ID, version, effective date, department, language, and permissions. When a source changes, reprocess only the affected documents if your pipeline supports incremental indexing. This reduces unnecessary work and makes changes easier to audit.

2. Review retrieval quality

Maintain a small evaluation set containing realistic user questions and the passages that should answer them. Include direct lookups, ambiguous questions, multi-step questions, and questions with no supported answer. For each case, examine whether the expected passage appears in the retrieved set, whether irrelevant passages crowd it out, and whether metadata filters behave correctly.

Chunk size and overlap should be treated as adjustable design choices, not permanent settings. If answers regularly contain half a procedure or combine unrelated sections, revisit the segmentation strategy. The guide RAG Chunking Strategies Compared provides a useful framework for considering those trade-offs.

3. Review generation behavior

Test whether the model follows the evidence boundary. It should distinguish between information present in the retrieved context and information that is merely plausible. Use a structured response when downstream systems need predictable fields, such as answer, citations, confidence_note, and needs_human_review. Keep the system prompt explicit about what to do when sources conflict or provide no answer.

4. Record operational changes

Version the prompt, retrieval settings, embedding configuration, index schema, and model choice together. Log a trace ID, query, retrieved document identifiers, scores or ranks, model request metadata, latency, token usage, and validation results while excluding sensitive content where appropriate. This makes it possible to compare a regression with the change that caused it. For related release checks, use the Prompt Engineering Checklist Before Shipping an AI Feature.

Signals that require updates

Do not wait for a major failure before reviewing a RAG application. The following signals usually justify investigation:

• Users report outdated answers: verify ingestion schedules, document timestamps, deletion handling, and index freshness.
• Correct passages are retrieved but ignored: inspect context ordering, prompt instructions, passage length, and the model’s output validation.
• Search works for exact terms but not paraphrases: consider hybrid retrieval, improved metadata, query rewriting, or a different embedding configuration.
• Answers become less specific: check whether chunks are too broad, retrieval returns too many candidates, or reranking is missing.
• Citations do not support claims: require citations at the claim or section level and test citations against the source text rather than checking only that a link exists.
• Latency or cost increases: measure ingestion, embedding, search, reranking, and generation separately. A change in one stage may be responsible for the overall regression. See How to Benchmark LLM Latency and AI Cost Monitoring for Developers.
• Users ask new types of questions: expand the evaluation set before changing the system. New intent may require additional metadata, query decomposition, or a different answer workflow.

Search intent can shift even when the underlying documents do not. A system designed for simple fact lookup may need new tests when users begin asking for comparisons, troubleshooting steps, or summaries across several sources.

Common issues

Retrieving by similarity alone

Semantic similarity is useful, but it does not always respect product versions, permissions, dates, or exact identifiers. Combine semantic retrieval with metadata filters and, where useful, keyword search. Never rely on the model to enforce access control after restricted documents have already entered its context.

Using context as a substitute for validation

Retrieved text can be incomplete, contradictory, or maliciously written. Treat documents as data, not as instructions that can override the application’s system rules. Add input and output checks, source allowlists, and clear handling for conflicting documents. The practical guidance in How to Add Guardrails to LLM Apps can help balance safety with useful responses.

Measuring only final-answer quality

A single thumbs-up metric hides where the pipeline failed. Track retrieval recall on labeled examples, citation support, abstention behavior, answer completeness, latency, and cost. A change that improves fluency while reducing evidence quality should not be considered an improvement.

Overloading the context window

Adding more passages does not automatically improve an answer. It can introduce distractions, contradictions, and higher processing cost. Rank candidates, remove duplicates, preserve the most relevant headings, and keep the final context within a deliberate budget. Review How to Fit More Useful Information into Prompts when context limits become a design constraint.

Changing several variables at once

Do not change chunking, retrieval, prompt wording, and model selection in one untracked release. Establish a baseline, change one major variable where practical, and compare results on the same evaluation set. This is slower than guesswork at first but makes the system easier to improve.

When to revisit

Schedule a lightweight review on a regular cycle appropriate to the application’s change rate. At minimum, revisit the pipeline when its source documents, retrieval configuration, model, embedding setup, prompt, or user workflow changes. Also review it after incidents involving stale information, unsupported claims, permission mistakes, or unexplained cost and latency increases.

A practical review checklist is:

1. Run the current evaluation set and save the baseline results.
2. Add recent production questions, including failed and unanswered examples.
3. Check document freshness, duplicates, deletions, metadata, and access boundaries.
4. Inspect retrieved passages before judging the generated answer.
5. Verify that citations support the claims they accompany.
6. Compare latency, token usage, retrieval volume, and error rates with the previous version.
7. Document any changed prompts, models, index settings, or source-processing rules.

Keep a small change log beside the application and expand the test set whenever a new failure mode appears. If the application serves several audiences or content domains, evaluate each separately rather than relying on one blended score. For prompt and model portability, compare behavior across the providers your application supports using Best Practices for Multi-Model Prompt Design.

The goal of a production RAG system is not to answer every question. It is to retrieve defensible evidence, communicate its limits, and make failures visible enough to correct. A recurring maintenance cycle turns those principles into an operating practice, helping an LLM application remain useful as its knowledge and workload evolve.