AI & LLM · ACR registry
Google-Extended — robots.txt guide
Google crawler for generative AI training (separate from Google Search indexing).
- robots.txt User-agent
- Google-Extended
- Company
- Official documentation
- https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
How to allow Google-Extended in robots.txt
Add an explicit allow rule (recommended when you use specific bot blocks elsewhere):
User-agent: Google-Extended Allow: /
How to block Google-Extended in robots.txt
Block this crawler from your entire site:
User-agent: Google-Extended Disallow: /
Free-check your live policy
Don't guess — run the free AI Bot Checker against all 241 ACR crawlers including Google-Extended.
Or open the shareable checker page.
Related AI bots
Part of the AI Crawler Registry (ACR) · Best GEO tools 2026