If you have used a chatbot that answers questions about a company’s help center, or uploaded files and asked an assistant about them, you have probably used retrieval-augmented generation, or RAG. It is one of the most important techniques in practical AI, and the idea behind it is simple. The term comes from a 2020 research paper that paired a language model with a searchable index of Wikipedia, and noted that the model’s knowledge could then be updated without retraining.
Updated September 2026: we added links to the original RAG research paper and to OpenAI’s guidance on where RAG systems go wrong.
The problem RAG solves
A large language model learns from its training data, then stops learning. As we explain in our LLM explainer, it does not know about events after its knowledge cutoff, it has never seen your company’s internal documents, and when it is unsure it may make things up.
Retraining a model every time your documents change would be slow and expensive. RAG takes a different approach: look up the relevant information first, then hand it to the model along with the question.
How RAG works, step by step
- Prepare the documents. Your content, such as manuals, policies, articles or tickets, is split into smaller chunks.
- Index them. Each chunk is converted into a list of numbers called an embedding, which captures its meaning. These are stored in a searchable index, often a vector database.
- Retrieve. When a user asks a question, the system finds the chunks whose meaning is closest to the question. Many systems combine this with traditional keyword search.
- Generate. The retrieved chunks are added to the prompt, and the model is instructed to answer using that material, ideally citing it.
Analogy Without RAG, the model is answering an exam from memory. With RAG, it is an open-book exam: someone quickly finds the right pages and puts them on the desk before the question is answered.
Why it is so widely used
- More accurate answers grounded in real sources
- Up-to-date information without retraining, since you just update the documents
- Citations, so users can check where an answer came from
- Access control, because the system can retrieve only documents a user is allowed to see
- Lower cost than training a custom model
Where you will find it
- Customer support assistants built on help centers, as we discuss in our chatbot guide
- Internal company assistants that search wikis and shared drives
- “Chat with your PDF” features in consumer assistants
- AI search engines that retrieve web pages before answering
Its limitations
RAG reduces errors but does not eliminate them. OpenAI’s accuracy guide notes that a RAG system can break down in two places: retrieval, when it finds the wrong or irrelevant context, and the model, when it gets the right context but uses it poorly.
- Bad retrieval means bad answers. If the search step finds the wrong chunk, the model may answer confidently from irrelevant text.
- Outdated or conflicting documents produce outdated or conflicting answers.
- Chunking matters. Splitting a table or a policy in the wrong place can separate a rule from its exceptions.
- The model can still ignore or misread the context, especially with long or ambiguous material.
Good RAG systems are evaluated continually with real questions, and their source content is maintained like any other product.
RAG vs fine-tuning
Fine-tuning adjusts a model’s behavior by training it further on examples. It is useful for teaching a style or format. RAG is better for supplying facts that change. Many production systems use both, but for most “answer questions about our documents” projects, RAG is the right starting point.
The question to ask any AI product
RAG is how AI assistants learn to say “according to this document” instead of guessing. If you are evaluating an AI product that claims to know your business, ask how it retrieves information, how it cites sources and how the underlying content is kept current.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv, 2020
- Optimizing LLM Accuracy, OpenAI API documentation
