Explaining RAG Systems: Architecture, Retrieval, Evaluation, and Production Best Practices


Retrieval-Augmented Generation (RAG) connects large language models (LLMs) to an external knowledge base so an application can retrieve relevant evidence before generating an answer. This approach helps AI applications work with private, specialized, and frequently updated information without retraining a model every time a document changes. But useful RAG is much more than placing a vector database beside a chatbot. It is an end-to-end information system: documents must be ingested, interpreted, indexed, retrieved under the right permissions, ranked, supplied to a model, cited, evaluated, and maintained.
This guide explains how RAG systems work, how their components fit together, which design choices matter, and how to move from a small prototype to a dependable production service. It also covers common failure modes, security controls, evaluation methods, cost trade-offs, and the situations where RAG may not be the right solution.
Retrieval-Augmented Generation is an architecture that combines information retrieval with language-model generation. When a user asks a question, the application searches a collection of external sources for relevant passages. It then provides those passages, together with the question and clear instructions, to a language model. The model uses the supplied material to formulate a response.
The term describes two complementary operations:
A conventional language model generates from patterns and knowledge encoded during training, plus the context supplied in the current request. It does not automatically know the latest version of an internal policy, the contents of a newly uploaded report, or the details of a private company knowledge base. RAG gives the application a way to provide relevant external information at query time.
RAG does not guarantee that an answer is correct, and it does not eliminate hallucinations. Retrieval can return irrelevant, incomplete, outdated, or misleading material. A model can also misinterpret accurate passages or make claims that the passages do not support. The objective is to improve grounding, freshness, traceability, and control—not to promise perfect accuracy.
Organizations often want to use language models with information that is too specific, sensitive, or changeable to be reliably contained in a model's general training data. Examples include product documentation, support procedures, engineering runbooks, legal research collections, research papers, policy manuals, and internal project records.
RAG is useful because the knowledge source can be maintained separately from the model. When a policy changes, an ingestion process can update the relevant documents and index. The application does not necessarily need a new model training run. This separation can shorten update cycles and make it easier to inspect which source material was available when an answer was produced.
RAG can also improve the transparency of an AI application. If the retrieval layer preserves source identifiers, titles, document versions, and page or section references, the application can show users where supporting information came from. That makes it easier to verify important claims and investigate incorrect responses.
The architecture is not automatically cheaper than every alternative. Parsing, embedding, indexing, search infrastructure, reranking, model calls, and evaluation all have costs. RAG is most valuable when access to external knowledge, source traceability, or frequent content updates justify those additional components.
A typical RAG application has an offline or asynchronous preparation path and an online question-answering path. Separating these paths avoids repeatedly processing an entire document collection whenever someone asks a question.
The preparation path usually includes:
The online path usually includes:
A production design also needs monitoring, access control, data-retention policies, error handling, and an evaluation process. These are core system components rather than optional finishing touches.
Retrieval quality starts with the source data. A sophisticated embedding model cannot recover information that was never extracted correctly, and a search engine cannot distinguish a current policy from an obsolete copy unless the application supplies useful metadata.
An ingestion pipeline should identify supported source formats, extract their content, normalize encoding, remove irrelevant boilerplate, and preserve important structure. PDFs, spreadsheets, slide decks, HTML pages, source code, and scanned documents require different parsing strategies. Scanned pages may need optical character recognition (OCR), while tables may need specialized extraction so that row and column relationships are not lost.
Keep a stable identifier for each source document. Record where it came from, when it was ingested, when it was last updated, and which parser or transformation version processed it. These details make it possible to trace a retrieved passage back to its origin and reprocess documents when the extraction logic changes.
Ingestion should be repeatable and idempotent. Processing the same unchanged source twice should not create uncontrolled duplicate records. When a document changes, the pipeline should detect the change, replace or version the affected chunks, regenerate embeddings where required, and remove obsolete index entries. Deletion must also propagate: removing a source document should not leave its text available indefinitely in a vector index or cache.
Validate the pipeline with representative documents before processing a large collection. Inspect extracted text, tables, headings, page references, and metadata. A small set of carefully reviewed examples often reveals parsing problems that would otherwise become retrieval problems later.
Chunking breaks long source documents into smaller units that can be searched and placed in a model's context window. If a whole book or policy manual is stored as one record, a query may retrieve too much unrelated material. If the text is divided into fragments that are too small, the meaning of a passage may depend on context that has been separated from it.
There is no universally correct chunk size. The right size depends on document structure, query patterns, embedding model, retrieval method, and the context budget of the generation model. A practical starting point is to test several sizes—for example, a few hundred tokens per chunk—and compare retrieval and answer quality on representative questions. Treat any default as an experiment, not a rule.
Common chunking strategies include:
Store enough information to reconstruct context. A chunk should normally carry a document identifier, a heading or section path, and a source location such as page number or URL. Consider adding a short contextual description when a fragment would otherwise be ambiguous outside its original document.
Evaluate chunking with actual queries. Check whether the retrieved passage contains the answer, whether necessary qualifications are nearby, and whether the same information appears in many near-duplicate chunks. Do not optimize chunk size using storage efficiency alone.
An embedding model converts text into a numerical vector. The vector represents aspects of the text's meaning in a form that makes similarity calculations possible. A document passage and a query with related meanings may have vectors that are close under a selected distance or similarity measure, even when they use different words.
For example, a user might ask about “resetting account access” while a help article is titled “recover a forgotten password.” Semantic retrieval may connect the two because their meanings are related. This is one reason embeddings are useful for natural-language search.
Embedding models differ in supported languages, input limits, vector dimensions, domain performance, latency, and price. Some models are designed for general-purpose semantic search, while others are optimized for specialized domains or multilingual content. Select a model using representative queries and documents rather than relying only on benchmark claims.
Use compatible embedding configurations for indexed documents and query vectors. If you change the embedding model or its output dimensions, existing vectors may need to be regenerated and the index rebuilt. Record the model name and version alongside index metadata so that migrations can be managed safely.
Embeddings are not a complete representation of truth or relevance. Similarity indicates that two pieces of text are related according to the model; it does not prove that one passage answers the question. Exact identifiers, product codes, dates, names, and technical terms may require lexical search or structured filters in addition to semantic matching.
A vector database or search engine stores embeddings and supports nearest-neighbor queries. Many systems also store the original text, document identifiers, metadata, and ordinary searchable fields. The application can then retrieve passages by semantic similarity and filter them using structured conditions.
Common technology choices include dedicated vector databases, search platforms with vector capabilities, and relational databases extended with vector indexing. Examples include Pinecone, Qdrant, Weaviate, Chroma, Elasticsearch, Azure AI Search, and PostgreSQL with the pgvector extension. These products have different operational models and feature sets; the right choice depends on scale, hosting requirements, existing infrastructure, filtering needs, and team experience.
Approximate nearest-neighbor indexes can make similarity search practical for large collections. Algorithms such as Hierarchical Navigable Small World (HNSW) and inverted-file approaches trade memory, indexing time, and search accuracy in different ways. A system should be tuned against its own workload, not selected solely because a particular index type is popular.
Before choosing a database, test the requirements that affect the whole application:
The vector store is only one part of the retrieval system. A strong database cannot compensate for poor parsing, weak metadata, incorrect permissions, or an unsuitable retrieval strategy.
Vector search retrieves passages by semantic similarity. It is useful when a question and its answer use different wording or when the user describes a concept rather than repeating the exact terms in a document.
Keyword search looks for literal terms and lexical matches. Techniques such as BM25 rank documents using term frequency, document length, and inverse document frequency. Keyword search is particularly useful for exact names, error codes, identifiers, product numbers, version strings, and specialized terminology.
Hybrid search combines vector and lexical retrieval. The system can run both searches, merge their candidate lists, and rank the combined results. Reciprocal Rank Fusion (RRF) is one common way to combine rankings without assuming that the raw scores from different retrieval methods are directly comparable. Some platforms also support weighted combinations or other fusion methods.
Hybrid retrieval is often a strong baseline for business and technical collections because it can capture both semantic similarity and exact-term relevance. However, it is not automatically best for every dataset. A small collection with predictable queries may perform well with simple keyword search, while a specialized semantic corpus may benefit from vector retrieval. Compare approaches using the same evaluation set.
Retrieval should also apply filters before or during search. Examples include restricting results to the current user's tenant, an approved document classification, a language, or a valid document version. Applying permissions only after the model has seen the retrieved content is too late.
Users rarely phrase questions in the same way as the documents they need. They may use abbreviations, omit context, ask a multi-part question, or refer to something mentioned earlier in a conversation. Query processing helps translate that request into a form the retrieval system can handle.
Common techniques include:
Hypothetical Document Embeddings (HyDE) is a technique in which a model drafts a hypothetical passage that could answer the question, then embeds that passage for retrieval. It can help with some semantic gaps, but the hypothetical text is not evidence and must never be presented as a verified source.
Query transformation can increase recall, but it can also introduce assumptions or drift away from the user's intent. Preserve the original question, log transformations, and evaluate whether the extra retrieval step improves end-to-end results. For straightforward questions, a direct search may be faster, cheaper, and easier to debug.
Initial retrieval is usually designed to find a broad set of potentially relevant candidates. The top results are not necessarily the best passages to send to the language model. A reranker evaluates each candidate in relation to the actual query and produces a more useful ordering.
A common two-stage pattern is:
A cross-encoder reranker typically evaluates a query and candidate passage together, allowing it to consider their relationship more directly than a standalone embedding similarity score. A language model can also be used to assess relevance, although that may have different latency and cost characteristics.
Reranking can improve precision, but it adds processing time and another model or service dependency. Test several candidate-set sizes and final passage counts. Measure whether the additional cost produces a meaningful improvement in answer quality.
Do not select passages based only on their score. Check source quality, freshness, duplicate content, document authority, and permissions. A slightly lower-scoring current policy may be more appropriate than a highly similar obsolete draft.
After retrieval and reranking, the application constructs the context sent to the language model. This step is sometimes called prompt assembly or context packing. It should include the original question, relevant passages, source identifiers, and explicit instructions describing how the model should use the evidence.
A useful instruction set should make clear that:
Context size is limited by the model's context window, latency budget, and cost constraints. More retrieved text does not always produce a better answer. Irrelevant passages can distract the model, repeat the same facts, or obscure important qualifications. Deduplicate, order, and limit the context based on evaluation.
Keep citation metadata separate from the text where possible. Each passage can carry a stable source ID, document title, page or section, and link. The model can refer to those IDs, while the application resolves them into trusted citation objects. This reduces the risk of fabricated links and makes it easier to validate that every displayed citation corresponds to a retrieved record.
Grounded generation should not silently turn missing evidence into a confident answer. If the user asks for a fact that the collection does not contain, a useful system may say that it could not find supporting information and suggest an approved next step.
A simple retrieve-then-generate pipeline is often the best place to start. More advanced patterns can help when evaluation reveals specific limitations.
Multi-query retrieval creates several alternative search queries and merges their results. It can improve recall when one wording misses relevant passages, but duplicate results and query drift need to be controlled.
Parent-document retrieval indexes small chunks but retrieves a larger parent section after a match. This can restore context around a precise passage without making every indexed unit excessively large.
Contextual retrieval enriches chunks with information about the document or section they came from. This can help when a fragment contains pronouns, abbreviations, or references that make sense only in its wider context.
Corrective or self-checking RAG evaluates whether retrieved evidence is relevant or sufficient and may trigger another search, a different source, or an abstention. The check must be measured; a model's confidence statement alone is not a reliable quality guarantee.
Graph RAG represents entities and relationships as a graph, sometimes alongside ordinary text chunks. It can help with questions that require connecting multiple entities or understanding relationships across a large corpus. Graph construction and maintenance add complexity, so use it when relationship-oriented queries justify that cost.
Agentic RAG gives an agent the ability to choose retrieval tools, sources, or search steps dynamically. It can support complex questions and multi-source workflows, but introduces additional planning, authorization, latency, and observability requirements. An agent should only have access to tools and data that the current user is allowed to use.
These approaches are not mutually exclusive. A system may combine hybrid search, contextual chunks, reranking, and a limited retrieval loop. Add one capability at a time and retain it only when measured results justify the added complexity.
A RAG system should be evaluated as separate stages and as a complete user experience. If an answer is poor, the cause may be parsing, chunking, embedding, retrieval, ranking, prompt construction, generation, or citation handling. A single overall score will not identify which component needs attention.
Build a representative evaluation dataset before tuning the system. Include common questions, difficult questions, exact-term queries, ambiguous questions, multi-hop questions, questions with no answer in the collection, and questions that should be blocked by access permissions. Have knowledgeable reviewers identify the relevant source passages and acceptable answer characteristics.
Useful retrieval metrics include:
End-to-end answer evaluation should examine faithfulness or groundedness, answer relevance, completeness, correctness, citation accuracy, and appropriate abstention. A fluent answer is not necessarily a correct answer. Automated model-based judges can help scale evaluation, but should be calibrated against human review and domain-specific acceptance criteria.
Evaluate changes independently where possible. Compare chunk sizes, embedding models, search methods, filters, rerankers, context limits, and prompts against the same test set. Track latency and cost alongside quality so that a small relevance improvement is not mistaken for an operationally worthwhile change.
For production, combine offline regression tests with carefully monitored user feedback. Review failure examples, update the evaluation set, and rerun it when documents, models, parsers, prompts, or retrieval configuration change.
RAG creates a data access path between source systems and a language model. The fact that a document is stored in a vector index does not make it safe to expose to every user. Permissions must be enforced throughout ingestion, retrieval, generation, caching, and citation display.
Apply authorization before retrieved content is sent to the model. Store tenant, user-group, classification, and document-level access metadata with each indexed passage where appropriate. Derive filters from trusted identity and authorization services, not from user-supplied prompt text. Test for cross-tenant retrieval and verify that revoked access takes effect in indexes and caches.
Treat source documents and retrieved passages as untrusted input. A document may contain accidental or malicious instructions such as requests to ignore system rules, reveal secrets, or call tools. Retrieved text should be clearly delimited and treated as data. Do not allow the contents of a document to override system policy or grant new tool permissions.
Additional safeguards include:
RAG security is broader than prompt injection. It includes document poisoning, data leakage, insecure connectors, index tampering, cache isolation, stale permissions, and unsafe downstream tool use. The OWASP RAG Security Cheat Sheet provides a practical checklist for securing the full pipeline.
A RAG application depends on several services that can fail independently: source connectors, parsers, embedding endpoints, databases, search indexes, rerankers, and language-model APIs. Design for partial failure rather than assuming every stage will always respond.
Track operational signals such as ingestion success rate, documents awaiting processing, indexing lag, embedding failures, retrieval latency, reranking latency, generation latency, token usage, timeouts, and error rates. Track quality signals such as empty retrievals, low-relevance results, citation failures, user corrections, and evaluation scores. Monitoring only whether an API returns HTTP 200 does not show whether the answer is useful.
Make ingestion jobs resumable and retry transient failures safely. Use dead-letter handling or an equivalent review path for documents that repeatedly fail parsing. Version indexes and ingestion pipelines so that a new parser or embedding model can be tested before replacing the active index. Keep a rollback plan for changes that reduce retrieval quality.
Caching can reduce repeated work, but cache keys must include the relevant identity and authorization scope. A cached response or retrieved passage must never be served to a user who is not allowed to see it. Invalidate caches when source documents or permissions change.
Provide a fallback behavior for unavailable services. Depending on the application, that may mean retrying, returning a clear temporary error, switching to a permitted search path, or declining to answer. Do not silently present an answer from stale or unauthorized data as current.
The total cost of a RAG request includes more than the language-model response. It may include query embeddings, vector and keyword searches, reranking, query rewriting, prompt tokens, generation tokens, document parsing, storage, index maintenance, and observability.
Start by measuring the cost and latency of each stage. A slow response may be caused by a remote embedding call, an inefficient filter, a large reranking candidate set, or an unnecessarily long generation prompt. Optimize the measured bottleneck rather than changing every component at once.
Practical techniques include:
Do not optimize for the cheapest individual request at the expense of poor answers or unsafe retrieval. The useful objective is an acceptable balance among quality, latency, reliability, privacy, and total operating cost.
Imagine a company wants an assistant that answers employee questions using approved HR policies and IT support documentation. A basic design could work as follows.
First, an administrator connects the approved document repositories. The ingestion service extracts text from policy documents, FAQs, and support guides. It records the source ID, document title, version, effective date, department, and access classification. Documents that fail extraction are sent for review rather than silently indexed as empty content.
Next, the pipeline divides the text into meaningful sections, attaches metadata, generates embeddings, and writes the content to a search index. It also maintains keyword-searchable text. When a policy is updated, the system reprocesses the changed document and removes obsolete chunks.
When an employee asks, “How do I request parental leave?”, the application authenticates the employee and determines which policy collections they can access. It searches the authorized collection using a combination of semantic and keyword retrieval. A reranker orders the candidate passages, and the application selects a small set of current policy sections.
The language model receives the question, the retrieved sections, their source IDs, and instructions to answer only from the evidence. If the documents do not specify an eligibility condition, the assistant must not invent one. The application checks the citations and displays the policy title and section so the employee can verify the answer.
Finally, the team evaluates the system using approved test questions, including questions about outdated policies, restricted documents, ambiguous requests, and topics not covered by the collection. It monitors retrieval failures, answer quality, access-control events, and latency. This makes the assistant a maintained knowledge service rather than a one-off chatbot demo.
RAG, fine-tuning, and long-context prompting solve different problems. They can be combined, but they should not be treated as interchangeable techniques.
| Approach | Primary purpose | Strengths | Limitations |
|---|---|---|---|
| RAG | Retrieve external evidence at query time | Fresh or private knowledge, source references, selective context | Requires ingestion, indexing, retrieval, and permission management |
| Fine-tuning | Change model behavior or specialize task patterns | Consistent formats, domain-specific behavior, task adaptation | Does not automatically provide a current searchable knowledge base |
| Long-context prompting | Supply a large amount of source material directly | Simple for small, bounded collections or one-off analysis | Context can be expensive, noisy, and constrained by model limits |
Use RAG when the application needs to search a changing or sizable knowledge collection and show supporting sources. Consider fine-tuning when the main requirement is a repeatable behavior, style, classification, or output format rather than access to frequently changing facts. Use a long context when the source material is small enough to include directly and retrieval would add unnecessary complexity.
A hybrid design may use fine-tuning for behavior and RAG for external knowledge. The correct choice should come from the task's requirements and evaluation results.
Many disappointing RAG applications are not caused by a weak language model. They result from problems in the surrounding data and retrieval pipeline.
Treating vector similarity as proof of relevance: Similar passages may discuss the same topic without answering the actual question. Evaluate retrieval and consider keyword search, filters, and reranking.
Using arbitrary chunk sizes: Fixed fragments can separate definitions from qualifications or split tables. Test structure-aware and domain-specific strategies.
Ignoring metadata: Without source versions, dates, ownership, and permissions, the system cannot reliably filter obsolete or unauthorized material.
Indexing low-quality content: Duplicates, navigation text, broken OCR, and obsolete documents can dominate results. Clean and validate data before indexing.
Retrieving too many passages: More context can increase cost and distract the model. Measure the right candidate count and final context size.
Trusting citations without validation: A model can produce plausible but nonexistent references. Resolve citations against actual retrieved source IDs in application code.
Skipping no-answer tests: A system that always generates an answer may invent details when the knowledge base is silent. Include unanswerable questions in evaluation.
Ignoring permissions in caches: Even a correctly filtered search can leak data if cached results are shared across users or tenants.
Deploying without regression tests: A new parser, model, prompt, or index setting can make previously correct answers worse. Maintain a stable evaluation set and rerun it after meaningful changes.
A measured implementation is easier to debug than a large system built all at once.
For a practical architecture overview, Microsoft's RAG solution design and evaluation guide and information retrieval guidance explain the decisions involved in preparing, searching, and evaluating a knowledge collection. The Azure AI Search RAG overview also compares classic and more dynamic retrieval approaches.
RAG continues to evolve as models, search systems, and data platforms improve. Current design directions include more capable hybrid retrieval, contextualized chunks, graph-based knowledge representations, multimodal indexing, adaptive retrieval, and agent-directed search. These techniques aim to improve how systems interpret complex questions and find evidence across different types of data.
Multimodal RAG extends retrieval beyond plain text. A system may index information from images, diagrams, audio transcripts, video frames, or structured tables. This creates additional challenges: extracting reliable content, preserving relationships between modalities, indexing metadata, and presenting citations that let users inspect the original source.
Agentic retrieval can dynamically select data sources or break a question into multiple searches. It is useful when a request requires several steps, but it also increases the importance of tool authorization, execution limits, audit logs, and predictable failure behavior. More autonomy should not mean unrestricted access.
The most important direction is not simply adding more complex components. It is making retrieval systems measurable, secure, and maintainable. Better evaluation, clear source provenance, reliable updates, and permission-aware retrieval are foundational whether the application uses a simple pipeline or an advanced agentic architecture.
Retrieval-Augmented Generation gives language-model applications a practical way to work with external knowledge. By retrieving relevant passages before generation, a system can provide answers grounded in documents that are private, specialized, or updated more frequently than model training data.
Successful RAG depends on the entire pipeline. High-quality source extraction, meaningful chunking, useful metadata, compatible embeddings, well-configured indexes, effective retrieval, reranking, grounded prompts, validated citations, and strong access controls all contribute to the result. Evaluation and monitoring then show whether the system continues to work as documents, users, and models change.
Start with the simplest architecture that meets the use case. Build a representative evaluation set, measure retrieval before tuning generation, and add complexity only when the evidence shows it is needed. Most importantly, design the system to recognize when its sources do not support an answer. A RAG application becomes trustworthy not by claiming it never makes mistakes, but by making its evidence, limitations, and controls visible and testable.

Learn how to deploy machine learning models with batch, real-time, streaming, and edge inference. Explore model serving, safe releases, monitoring, MLOps, security, and production best practices.
Read
Understand model drift in machine learning, including data and concept drift, detection metrics, production monitoring, and safe model retraining.
Read
Learn how data engineering pipelines collect, transform, govern, and deliver reliable data for analytics, AI, and machine learning. Explore architecture, tools, data quality, and practical workflows.
Read