What Is RAG? Retrieval-Augmented Generation Explained

RAG — retrieval-augmented generation — is the pattern of retrieving relevant documents and feeding them to an LLM before it answers. It grounds model output in real data, which reduces hallucination and keeps answers current.

How RAG works

1. Split your documents into chunks and embed them into vectors.
2. Store vectors in a vector database with the source text.
3. At query time, embed the user question and search for similar chunks.
4. Send the retrieved chunks plus the question to the LLM.
5. The model answers grounded in the provided context.

When to use RAG

• Chatting over your own documents or a knowledge base.
• Facts that change often — docs, support articles, internal policies.
• Source citation is required: RAG can point at which document informed the answer.
• You cannot retrain a model but can index new content.

When RAG is not the answer

• Small static corpora that fit in context — a simple paste may be fine.
• Tasks needing deep reasoning about relationships — consider graph or agent approaches.
• High-latency real-time requirements where retrieval adds too much delay.

RAG vs long-context models

Modern models accept huge context windows, so why retrieve at all? Cost and accuracy: stuffing 2M tokens of irrelevant docs is expensive, slows responses, and can degrade attention on what matters. RAG retrieves the relevant slice, keeping cost and latency down.

The hidden costs of RAG

• Embedding costs for the corpus and every query.
• Vector storage and index maintenance.
• Chunking quality — bad splits mean bad retrieval.
• Tokens per query grow with context length.

FAQ