Skip to content

AI Crawler Audit

Audit your robots.txt for AI crawler access. Check GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and 7 more AI bots. Get an AI Visibility Score and generate optimized robots.txt.

FreeNo SignupNo Server UploadsZero Tracking

Robots.txt Input

How to Use AI Crawler Audit

  1. 1

    Paste your robots.txt

    Copy the contents of your robots.txt file and paste it into the input area. Or enter your site URL to try fetching it automatically.

  2. 2

    Analyze bot access

    Click Analyze to check which AI crawlers are allowed, blocked, or partially blocked by your current robots.txt rules.

  3. 3

    Review AI Visibility Score

    See your score from 0 to 100 based on how many AI crawlers can access your content. Higher scores mean more AI search visibility.

  4. 4

    Generate recommended rules

    Use the toggle controls to allow or block specific AI bots and generate a recommended robots.txt with your preferences.

Frequently Asked Questions

Key AI crawlers include GPTBot (OpenAI training), ChatGPT-User (ChatGPT browsing), ClaudeBot (Anthropic), Google-Extended (Google AI), PerplexityBot (Perplexity search), CCBot (Common Crawl), Bytespider (ByteDance), and others from Apple, Meta, and Amazon.

It depends on your goals. Blocking training crawlers (GPTBot, CCBot) prevents your content from being used to train AI models. However, blocking search crawlers (PerplexityBot, Google-Extended) may reduce your visibility in AI-powered search results.

llms.txt is a proposed standard for providing structured information about your site to LLMs. It helps AI systems understand your site's purpose, content, and preferred interaction. See llmstxt.org for details.

The AI Visibility Score (0-100) measures what percentage of major AI crawlers can access your site. A score of 100 means all AI bots can crawl your content; 0 means all are blocked.

No. All parsing and analysis happens entirely in your browser. If you use the URL fetch feature, the request goes directly from your browser to the target site (may fail due to CORS).