AI-generated review 4 min read Intermediate 839 words

Retrieval-Augmented Generation for Knowledge-Intensive NLP

An AI-generated review of retrieval-augmented generation (RAG), a framework that grounds language models in external knowledge to improve factual accuracy.

rag retrieval knowledge nlp hallucination retrieval-augmented-generation
Applications & prompting · 2020

Primary source: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401)

Large language models can produce fluent text, but they store knowledge implicitly in their parameters. That makes them prone to hallucination, especially on questions that require specific facts, dates, or domain details — and the knowledge is frozen at training time, with no way to point at where an answer came from.

The core idea

Retrieval-Augmented Generation, or RAG, splits the problem in two:

User query "Who built the Transformer?" Encoder Vector index d₁ Vaswani 2017… d₂ Attention paper d₃ BERT 2018 d₄ GPT-2 2019 d₅ Vision Transf… top-k passages Concatenate query + passages [Q; d₁; d₂; …] Generator BART / T5 / LLM Grounded answer "Vaswani et al., 2017"
RAG: a query is encoded, matched against a vector index to retrieve top-k passages, then the generator conditions on both to produce a grounded answer.
  1. Retrieve: given a query, find relevant documents from an external corpus.
  2. Generate: condition the language model on both the query and the retrieved documents to produce an answer.

By giving the model access to explicit source documents, RAG grounds generation in real text rather than parametric memory alone. The paper frames this as two memories working together: a parametric memory (the language model's weights) and a non-parametric memory (a searchable index of documents).

How the original system works

The original RAG model combines two pre-trained components:

  • Retriever: a dense passage retriever (DPR) with two BERT-based encoders, one for the query and one for documents. Retrieval is a maximum inner-product search over pre-computed document vectors.
  • Generator: BART, a sequence-to-sequence Transformer that writes the answer.
  • Corpus: a December 2018 Wikipedia dump split into roughly 21 million passages of about 100 words each.

At inference time:

  1. The query is encoded into a dense vector.
  2. The index returns the top-k most relevant passages (the paper uses 5 or 10).
  3. The generator produces output while attending to the query and the retrieved passages.

Training is end-to-end, so the retriever and generator learn to cooperate. The query encoder and the generator are fine-tuned, while the document encoder and its index are kept fixed, which avoids having to re-embed millions of passages as training proceeds.

Two ways to use the retrieved text

The paper proposes two formulations that differ in how the retrieved passages are combined:

  • RAG-Sequence uses the same retrieved passage for the whole output sequence, then averages over passages.
  • RAG-Token can draw on a different passage for each generated token, so one answer can combine facts from several documents.

What the paper reported

RAG set state-of-the-art results on several open-domain question-answering benchmarks (Natural Questions, TriviaQA, WebQuestions and CuratedTrec), outperforming both purely parametric sequence-to-sequence models and task-specific pipelines that retrieve and then extract an answer span. On generation tasks the authors found that RAG produced more specific, diverse and factual text than a comparable model with no retrieval, and it was also competitive on fact verification. The authors also showed that swapping the document index changes what the model knows, without any retraining.

Why it matters

  • Factual accuracy: answers can cite or be traced back to retrieved passages.
  • Up-to-date knowledge: the external corpus can be updated without retraining the language model.
  • Domain adaptation: swapping the corpus lets the same model answer questions in medicine, law, or finance.

What has changed since

The 2020 system was a research model; the pattern it named is now everywhere. Modern RAG applications typically pair a general-purpose large language model with a vector index, and much of the engineering is in the parts the paper treated lightly: how documents are split into chunks, which embedding model is used, whether a second ranking stage reorders the results, and how citations are shown to the user. Those practices come from the wider ecosystem rather than from this paper.

Limitations

  • RAG is only as good as the retriever. If the retriever misses the relevant document, the generator has no factual basis. Dense retrieval also struggles with rare entities and out-of-domain queries.
  • Retrieval reduces, but does not eliminate, hallucination. A model can misread a correct passage or ignore it in favour of its own memory, and a wrong passage can mislead it.
  • Cost and latency. Every query now involves a search as well as a generation.
  • Hard to evaluate. A good answer depends on both components, so failures can be hard to attribute.

Bottom line

RAG is one of the most practical ways to make language models more factual and transparent. It replaces the dream of an all-knowing parametric model with a system that knows where to look.

Source

  • Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. arXiv:2005.11401
  • Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, 2020. arXiv:2004.04906
  • Lewis et al., BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, 2019. arXiv:1910.13461

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-rag-knowledge-intensive-nlp
  title        = {Retrieval-Augmented Generation for Knowledge-Intensive NLP},
  author       = {{Know Your ML}},
  year         = {2020},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=rag-knowledge-intensive-nlp}},
  note         = {AI-generated review of arXiv:2005.11401. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

Frequently asked questions

What is retrieval-augmented generation?

A design in which a language model is given relevant documents, fetched from an external collection at question time, in addition to the question itself. The model answers from the question plus the retrieved text instead of relying on its stored weights alone.

Is RAG better than fine-tuning for adding knowledge?

They solve different problems. Retrieval lets you add or change knowledge by editing the document index and gives you passages to cite, with no retraining. Fine-tuning changes how the model behaves or writes. For facts that change or need sources, retrieval is usually the simpler tool.

Does RAG stop a model from hallucinating?

No. It gives the model better evidence, which reduces unsupported answers, but the model can still misread a correct passage, ignore it, or be misled by an irrelevant one. If the retriever fails to find the right document, the generator has nothing reliable to work from.

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook