Retrieval-Augmented Generation for Knowledge-Intensive NLP
An AI-generated review of retrieval-augmented generation (RAG), a framework that grounds language models in external knowledge to improve factual accuracy.
Primary source: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401)
Large language models can produce fluent text, but they store knowledge implicitly in their parameters. That makes them prone to hallucination, especially on questions that require specific facts, dates, or domain details — and the knowledge is frozen at training time, with no way to point at where an answer came from.
The core idea
Retrieval-Augmented Generation, or RAG, splits the problem in two:
- Retrieve: given a query, find relevant documents from an external corpus.
- Generate: condition the language model on both the query and the retrieved documents to produce an answer.
By giving the model access to explicit source documents, RAG grounds generation in real text rather than parametric memory alone. The paper frames this as two memories working together: a parametric memory (the language model's weights) and a non-parametric memory (a searchable index of documents).
How the original system works
The original RAG model combines two pre-trained components:
- Retriever: a dense passage retriever (DPR) with two BERT-based encoders, one for the query and one for documents. Retrieval is a maximum inner-product search over pre-computed document vectors.
- Generator: BART, a sequence-to-sequence Transformer that writes the answer.
- Corpus: a December 2018 Wikipedia dump split into roughly 21 million passages of about 100 words each.
At inference time:
- The query is encoded into a dense vector.
- The index returns the top-k most relevant passages (the paper uses 5 or 10).
- The generator produces output while attending to the query and the retrieved passages.
Training is end-to-end, so the retriever and generator learn to cooperate. The query encoder and the generator are fine-tuned, while the document encoder and its index are kept fixed, which avoids having to re-embed millions of passages as training proceeds.
Two ways to use the retrieved text
The paper proposes two formulations that differ in how the retrieved passages are combined:
- RAG-Sequence uses the same retrieved passage for the whole output sequence, then averages over passages.
- RAG-Token can draw on a different passage for each generated token, so one answer can combine facts from several documents.
What the paper reported
RAG set state-of-the-art results on several open-domain question-answering benchmarks (Natural Questions, TriviaQA, WebQuestions and CuratedTrec), outperforming both purely parametric sequence-to-sequence models and task-specific pipelines that retrieve and then extract an answer span. On generation tasks the authors found that RAG produced more specific, diverse and factual text than a comparable model with no retrieval, and it was also competitive on fact verification. The authors also showed that swapping the document index changes what the model knows, without any retraining.
Why it matters
- Factual accuracy: answers can cite or be traced back to retrieved passages.
- Up-to-date knowledge: the external corpus can be updated without retraining the language model.
- Domain adaptation: swapping the corpus lets the same model answer questions in medicine, law, or finance.
What has changed since
The 2020 system was a research model; the pattern it named is now everywhere. Modern RAG applications typically pair a general-purpose large language model with a vector index, and much of the engineering is in the parts the paper treated lightly: how documents are split into chunks, which embedding model is used, whether a second ranking stage reorders the results, and how citations are shown to the user. Those practices come from the wider ecosystem rather than from this paper.
Limitations
- RAG is only as good as the retriever. If the retriever misses the relevant document, the generator has no factual basis. Dense retrieval also struggles with rare entities and out-of-domain queries.
- Retrieval reduces, but does not eliminate, hallucination. A model can misread a correct passage or ignore it in favour of its own memory, and a wrong passage can mislead it.
- Cost and latency. Every query now involves a search as well as a generation.
- Hard to evaluate. A good answer depends on both components, so failures can be hard to attribute.
Bottom line
RAG is one of the most practical ways to make language models more factual and transparent. It replaces the dream of an all-knowing parametric model with a system that knows where to look.
Source
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. arXiv:2005.11401
- Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, 2020. arXiv:2004.04906
- Lewis et al., BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, 2019. arXiv:1910.13461
Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.
Cite this report
@misc{knowyourml-rag-knowledge-intensive-nlp
title = {Retrieval-Augmented Generation for Knowledge-Intensive NLP},
author = {{Know Your ML}},
year = {2020},
howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=rag-knowledge-intensive-nlp}},
note = {AI-generated review of arXiv:2005.11401. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit
Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.
References
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401) — https://arxiv.org/abs/2005.11401
- Dense Passage Retrieval for Open-Domain Question Answering (arXiv:2004.04906) — https://arxiv.org/abs/2004.04906
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (arXiv:1910.13461) — https://arxiv.org/abs/1910.13461
Frequently asked questions
What is retrieval-augmented generation?
A design in which a language model is given relevant documents, fetched from an external collection at question time, in addition to the question itself. The model answers from the question plus the retrieved text instead of relying on its stored weights alone.
Is RAG better than fine-tuning for adding knowledge?
They solve different problems. Retrieval lets you add or change knowledge by editing the document index and gives you passages to cite, with no retraining. Fine-tuning changes how the model behaves or writes. For facts that change or need sources, retrieval is usually the simpler tool.
Does RAG stop a model from hallucinating?
No. It gives the model better evidence, which reduces unsupported answers, but the model can still misread a correct passage, ignore it, or be misled by an irrelevant one. If the retriever fails to find the right document, the generator has nothing reliable to work from.
Beyond the paper
How to Remain Valuable When Intelligence Becomes Cheap
This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?