Retrieval-augmented generation
Retrieval-augmented generation (RAG) is a method where an AI model fetches relevant documents first, then writes its answer using them.
The model doesn’t rely only on what it learned in training. It pulls in current sources at the moment someone asks, which is how AI answers can cite web pages.
The method comes from a 2020 paper by researchers at Facebook AI Research, University College London and New York University: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis and colleagues, presented at NeurIPS 2020. Their system paired a language model with a searchable index of Wikipedia, and produced more specific and factual answers than the model alone.
RAG in AI search
AI search products are RAG systems built on web indexes. Google’s documentation on AI features says AI Overviews and AI Mode may use a “query fan-out” technique, running several related searches across subtopics before writing an answer. ChatGPT search draws on pages crawled by OpenAI’s OAI-SearchBot. In each case, a page can only be cited if the retrieval step finds it.
The pages AI answers cite are often not the pages that rank. Ahrefs found that only 12% of URLs cited by ChatGPT, Gemini and Copilot across 15,000 prompts ranked in Google’s top 10 for the same query. Even inside Google, the overlap is shrinking: an Ahrefs study of 863,000 keywords found the share of AI Overview citations that ranked in the top 10 as well fell from 76.1% in July 2025 to 38% in March 2026. Fan-out explains much of the gap. The AI is answering sub-questions you never targeted.
What makes a page retrievable
| Factor | Why it matters for RAG |
|---|---|
| Crawl access | The retrieval crawler must be allowed in robots.txt and your firewall |
| Server-rendered HTML | Most AI crawlers don’t run JavaScript |
| Passages that stand alone | Systems retrieve sections, not whole pages; Google’s passage ranking system works the same way |
| Sources and statistics | The GEO research paper (KDD 2024) found adding citations, quotations and statistics raised visibility in AI answers by up to 40% |
| Coverage of sub-questions | Fan-out searches pull from pages that answer the follow-up questions |
Common mistakes
-
Assuming a top-10 Google ranking means AI answers will cite you.
-
Answering only the head query and none of the follow-up questions.
-
Burying the answer in long introductions that don’t stand alone when extracted.
-
Blocking retrieval crawlers along with training crawlers.