Robots.txt

It sits at yourdomain.com/robots.txt and uses simple rules: a user agent line naming the crawler, then Allow and Disallow lines for paths.

Robots.txt controls crawling, not indexing. Google’s introduction to robots.txt says it is mainly for avoiding overloading your site with requests, and is not a way to keep a page out of Google. A blocked URL can still be indexed if other sites link to it, and then appears in results without a description. To keep a page out of Google, use a noindex tag or password protection. Don’t block the same page in robots.txt, or Google can’t crawl it to see the noindex.

Rules for AI crawlers

Each AI company names its crawlers, so you can treat AI search and AI training separately. This set allows Google Search and ChatGPT search, and opts out of training for Gemini and OpenAI’s models:

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Google says Google-Extended doesn’t affect a site’s inclusion or ranking in Google Search, and AI Overviews and AI Mode run on Googlebot’s crawl. Blocking Google-Extended won’t remove you from AI Overviews. Blocking Googlebot removes you from Google Search entirely. On OpenAI’s side, its documentation says sites that block OAI-SearchBot won’t appear in ChatGPT search answers, and that robots.txt changes take about 24 hours to apply.

Robots.txt isn’t the only gate

Robots.txt is a request, and crawlers choose whether to follow it. Google says Googlebot and other respectable crawlers obey it, but others might not. The opposite problem is common too: a CDN or firewall rule blocks a crawler your robots.txt allows. Check both. If an AI crawler is blocked at the firewall, your robots.txt settings make no difference.

Common mistakes

• Using Disallow to hide a page from search results.

• Blocking a page in robots.txt that carries a noindex tag Google then can’t read.

• Blocking CSS or JavaScript files Google needs to render the page.

• Blocking every AI crawler without separating search from training.

• Leaving a staging site’s Disallow rule in place after launch.