Check which AI and search crawlers your robots.txt allows or blocks. Free robots.txt policy audit powered by the ACR registry of 241 bots — GPTBot, ClaudeBot…

Features Pricing Free Tools Help Developers Success Stories Sign in Get started

Check 241 AI Bots Against Your robots.txt — Free AI Bot Checker

ChatGPT, Claude, Perplexity, and Google AI do not read your website like a human. They send documented crawlers — bots with names like GPTBot, ClaudeBot, and Google-Extended — and those bots obey your robots.txt file before they fetch pages.

If you block the wrong user-agent, you can disappear from AI answers while your traditional SEO looks fine. The fix starts with a simple question: which crawlers does my robots.txt actually allow?

241 crawlers, one open registry (ACR)

AigeoRadar maintains the AI Crawler Registry (ACR) — part of the AI Discovery Framework (ADF). The registry documents 241 web crawlers across AI & LLM bots, search engines, SEO tools, social link preview, RSS, ads, security, monitoring, and archive bots.

ACR gives you canonical User-agent tokens, categories, and capability flags — so you are not guessing whether Bytespider is an AI trainer or a social preview bot.

Free AI Bot Checker — check your site in seconds

Our AI Bot Checker is a free tool on the AigeoRadar tools page. Enter your domain and it will:

  1. Fetch your public robots.txt (or detect when it is missing)
  2. Match it against AI-relevant crawlers from the ACR registry
  3. Return a Crawler Access Score, grade, and lists of allowed vs blocked AI bots

No credit card. No signup required. Run it now: aigeoradar.com/tools → select AI Bot Checker.

What the checker does — and what it does not

Be precise about scope. The free AI Bot Checker is a robots.txt policy audit, not a live bot detector.

  • It does: parse User-agent, Allow, and Disallow rules and classify registry crawlers as allowed, blocked, or unknown.
  • It does not: ping bots, simulate visits, or prove that GPTBot crawled you yesterday.

If your site is reachable but has no robots.txt file (HTTP 404), the tool applies the common convention: implicit allow all for public pages, with a clear policy note in the report. That is honest policy analysis — not an error.

To see actual bot traffic, AigeoRadar Pro includes Bot Activity with server log import (and Cloudflare integrations on the roadmap).

Why this matters for GEO

GEO (Generative Engine Optimization) is more than llms.txt and FAQ schema. You need two layers:

  1. Discovery assetsllms.txt, entity.json, Organization schema, and AI Page Manifest (AIPM) sidecars so models understand your business.
  2. Crawler access — robots.txt rules that let the right bots in while keeping scrapers and trainers out if that is your policy.

Many sites publish great discovery files but accidentally block GPTBot or anthropic-ai with a blanket Disallow: / copied from an old SEO template. The AI Bot Checker surfaces those mismatches in one report.

High-impact bots to watch

These user-agents show up often in GEO conversations. Your report will flag each one against your live policy:

  • GPTBot — OpenAI training and browsing crawler
  • ClaudeBot / anthropic-ai — Anthropic crawlers
  • Google-Extended — Google generative AI use (separate from Googlebot search)
  • PerplexityBot — Perplexity answer engine
  • Bytespider — ByteDance / TikTok ecosystem
  • CCBot — Common Crawl (widely used in training corpora)
  • Googlebot / Bingbot — classic search (still relevant for AI Overviews and Copilot grounding)

The full ACR registry covers 241 agents; the free checker focuses on AI, search, training, and verified crawlers so the report stays actionable.

What to do if important bots are blocked

  1. Read the blocked list in your AI Bot Checker report — do not unblock everything blindly.
  2. Decide policy: allow AI search bots (Perplexity, Bing) while blocking trainers only, or allow all official AI crawlers for maximum visibility.
  3. Edit robots.txt with explicit User-agent blocks or allows. Test again with the free checker.
  4. Layer discovery files — run the llms.txt Generator and a full GEO scan on AigeoRadar so bots that are allowed find structured content.
  5. Monitor real visits — upgrade to Pro for Bot Activity and ongoing Crawler Access Score in your analysis panel.

FAQ

Is the AI Bot Checker really free?

Yes. Enter a domain on aigeoradar.com/tools, choose AI Bot Checker, and get an instant robots.txt policy report.

Does it check all 241 ACR crawlers?

The registry lists 241 documented crawlers. The free checker evaluates the AI-relevant subset (AI, search, training, and verified bots) so results stay focused on GEO impact. The full registry is public at /ai-crawlers.json.

Can I use ACR on my own site or product?

Yes. ACR is open data (CC BY 4.0). Link to the spec, import /ai-crawlers.json, or contribute via GitHub.

How is this different from Google Search Console?

Search Console shows Googlebot crawl stats. AI Bot Checker maps many vendors’ AI and search agents against your robots.txt in one pass — including bots GSC does not surface.

Run your free check

Stop guessing whether AI can reach your site. Run the AI Bot Checker — powered by ACR v1.0 and the 241-bot registry.

İlgili yazılar

Free AI Visibility Scan