Find out whether Google, ChatGPT, Claude and Perplexity can crawl your website. Enter a URL to test your robots.txt and each crawler's access.
A page is crawlable when a bot is allowed to request it and gets the content back. Three things usually get in the way: a robots.txt rule, a firewall that turns bots away, or a noindex tag.
This checker tests each of them for search engines and AI crawlers at once.
Run the checker above to see whether your robots.txt allows Googlebot and whether your server lets it through.
For Google's own answer, use URL Inspection in Google Search Console. It shows whether Googlebot could fetch the page, when it last crawled it and whether the page is indexed.
AI companies run two kinds of crawlers. Search bots like OAI-SearchBot and PerplexityBot fetch pages so AI answers can cite them. Training bots like GPTBot and ClaudeBot collect content to train models.
You can block training bots and still show up in AI answers. Google's AI Overviews and AI Mode use Googlebot, so any page Google can crawl can appear there.
Cloudflare blocks AI crawlers by default on many new sites, and its Block AI bots setting takes one click to turn on. Security plugins and hosting firewalls can block them too.
These blocks happen outside robots.txt, so most SEO tools won't flag them.