Check if ChatGPT, Claude, Perplexity, and other AI crawlers can access your website. We analyze your robots.txt to show exactly which bots are allowed or blocked.
Works with any website — checks GPTBot, ClaudeBot, PerplexityBot, and more
Guide
AI crawlers are web bots operated by AI companies to index and understand website content. Just as Googlebot crawls the web for Google Search, AI crawlers like GPTBot, ClaudeBot, and PerplexityBot crawl websites to power AI assistants and search engines.
When a user asks ChatGPT about a product, or Perplexity searches for a recommendation, these crawlers are what allow AI to access and reference your content. If your robots.txt blocks them, AI platforms simply can't see your site.
The good news: you have full control. Your robots.txt file determines which crawlers can access your site, and you can allow or block each one individually.
Crawlers
Powers ChatGPT's web browsing and knowledge. Blocking this means ChatGPT can't reference your content directly.
Enables Claude to access and cite your website. Used for real-time information retrieval.
Drives Perplexity's AI search engine. Critical for appearing in Perplexity search results.
Specifically for OpenAI's search features. Can be allowed separately from GPTBot for search-only access.
Google's primary crawler. Also feeds Google's AI features like AI Overviews and Gemini.
ByteDance's crawler powering TikTok's AI features and search capabilities.
FAQ
AI crawlers are web bots used by AI platforms to index and understand website content. GPTBot powers ChatGPT's browsing, ClaudeBot serves Anthropic's Claude, and PerplexityBot drives Perplexity's AI search. They work similarly to Googlebot but feed AI models instead of search indexes.
AI crawlers respect your robots.txt file. To allow them, make sure you don't have 'Disallow: /' rules for AI bot user-agents like GPTBot, ClaudeBot, or PerplexityBot. You can add explicit 'Allow: /' rules for each crawler you want to permit.
For most websites, allowing AI crawlers is beneficial — it helps AI assistants recommend your content and products to users. Block them only if you have specific concerns about AI training data usage or content licensing. Many platforms like OpenAI offer ways to allow search browsing while opting out of training.
GPTBot is OpenAI's general web crawler used for training and browsing. OAI-SearchBot is specifically for OpenAI's search features. You can allow OAI-SearchBot (for search visibility) while blocking GPTBot (to opt out of training) if you want granular control.