Large language model
A large language model (LLM) is an AI system trained on very large amounts of text to read and generate language.
The GPT models behind ChatGPT, Google’s Gemini and Anthropic’s Claude are all LLMs. In search, LLMs write the answers you see in ChatGPT search, Google’s AI Overviews and AI Mode.
An LLM answers from two places: what it learned in training, and pages it fetches at the moment someone asks. Only the second can cite your site. Training data has a cut-off date and carries no links, so a model can describe your category from memory and still name a competitor whose page it retrieved that morning. The fetching step is called retrieval-augmented generation, and it is where SEO work pays off in AI answers.
Which crawlers feed which product
AI companies run separate crawlers for training and for search, and each one obeys its own rules in robots.txt.
| Crawler | Company | What it is for |
|---|---|---|
| Googlebot | Google Search, including AI Overviews and AI Mode | |
| Google-Extended | A control for Gemini training and grounding. Google says it doesn’t affect inclusion or ranking in Search. | |
| OAI-SearchBot | OpenAI | Showing sites in ChatGPT search results |
| GPTBot | OpenAI | Collecting content that may be used to train OpenAI’s models |
| ChatGPT-User | OpenAI | Fetching a page when a ChatGPT user asks for it. OpenAI says robots.txt rules may not apply to these requests. |
Sources: OpenAI’s crawler documentation and Google’s list of common crawlers. Blocking GPTBot keeps your content out of OpenAI’s training data. Blocking OAI-SearchBot keeps you out of ChatGPT’s search answers. Most B2B sites want the first choice to be theirs and the second one open.
Most AI crawlers don’t run JavaScript
A December 2024 study by Vercel and MERJ tracked AI crawler traffic across Vercel’s network. OpenAI’s GPTBot and Anthropic’s ClaudeBot both fetched JavaScript files but never executed them. A page that builds its main content in the browser can rank in Google, which renders JavaScript, and still look empty to the crawlers feeding AI answers. Put the content that matters in the HTML the server sends. The JavaScript SEO entry covers the details.
LLM visibility on a B2B site
For a Canadian taxation SaaS product, structured content and technical SEO work earned a steady run of citations in Microsoft Copilot, tracked in Bing Webmaster Tools. No special markup produced that. The pages answered specific questions clearly, in server-rendered HTML, and Bing could crawl and index them.
Common mistakes
-
Blocking every AI crawler at once and losing AI search visibility along with AI training.
-
Assuming the model knows about your company from training data.
-
Building main product pages in client-side JavaScript only.
-
Treating Google-Extended as a way to leave AI Overviews. It doesn’t do that.