How RAG Works

Retrieval-augmented generation is the standard pattern for grounding AI answers in trusted source material.

A base language model knows only what it was trained on. Retrieval-augmented generation, or RAG, fixes that limitation by fetching relevant documents at query time and injecting them into the prompt before the model answers. RAG does not change the model's weights. It changes what the model sees at inference time, which is why it has become the standard pattern for grounding AI answers in trusted source material.

The system has two halves. An offline ingestion pipeline splits documents into chunks, converts each chunk into an embedding, and stores the embeddings in a vector database. An online retrieval pipeline embeds the user's query, finds the most similar chunks, re-ranks them, and passes the best ones to the model alongside the question. The model then generates a grounded answer with citations the reader can verify.

After this module you will know where quality is won and lost. Chunking is the single most consequential step: overly large chunks dilute relevance, and overly small chunks lose context. Hybrid search combines keyword matching with semantic vector search and merges results through reciprocal rank fusion, outperforming either approach alone. Re-ranking with a cross-encoder scores the query and document together and catches relevance signals that faster bi-encoders miss. Agentic RAG adds a reasoning layer that can rewrite queries, chain retrievals, or skip retrieval when the model already has the answer. You will also know the division of labor: RAG for hallucination mitigation, stale knowledge, and access-controlled enterprise data; fine-tuning for style, format, and behavior.

The failure modes are consistent. Teams treat a larger context window as a substitute for retrieval. They index sensitive documents without metadata filters for access control, so answers leak material the user should never see. They deploy without re-ranking and then blame the model for irrelevant answers. Production RAG needs continuous evaluation of faithfulness and answer relevance, and a re-ingestion trigger so the knowledge base stays current.

In one paragraph

A base language model knows only what it was trained on. Retrieval-augmented generation solves that limitation by fetching relevant documents at query time and injecting them into the prompt before the model generates its answer. The system has two halves: an offline ingestion pipeline that chunks documents, embeds them, and stores them in a vector database, and an online retrieval pipeline that embeds the query, finds the most similar chunks, re-ranks them, and passes them to the model alongside the user's question. The model then generates a grounded answer with verifiable citations.

This module is part of the Context and Grounding track in the free AI Learning Hub. Source material: How RAG Works.