AI Crawler Access Checker

AI Crawler Access Checker & llms.txt Validator

This tool answers two questions that matter more every month: which AI crawlers can reach your site, and is your llms.txt file built correctly. Paste your robots.txt into the first tab to get a bot-by-bot matrix showing whether each major AI crawler is allowed or blocked, and which rule made the decision. Switch to the second tab to check an llms.txt file against the published specification. Both checks run entirely in your browser, so nothing you paste is ever uploaded.

The Four Kinds of AI Crawler

The single biggest mistake site owners make is treating every AI bot the same way. They are not the same, and blocking the wrong one quietly removes you from AI answers. There are four distinct categories, and the access checker labels each one:

• Training crawlers fetch pages to train large language models. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), Amazonbot, and Meta-ExternalAgent belong here. Blocking these is a bandwidth and content-rights decision; it does not affect whether AI assistants can cite you.
• AI search and retrieval crawlers build the indexes that let an assistant find and cite your pages in real time. OAI-SearchBot (ChatGPT search), Claude-SearchBot, and PerplexityBot are the important ones. If you care about being mentioned in AI answers, these are the bots to allow.
• User-triggered fetchers visit a page only when a human asks the assistant to. ChatGPT-User, Claude-User, and Perplexity-User fall here. Because a person initiated the request, some of these may not strictly follow robots.txt.
• Opt-out tokens are not crawlers at all. Google-Extended and Applebot-Extended are directives that control whether Google and Apple may use already-crawled content for generative-AI training. Blocking Google-Extended does not affect your normal Google Search ranking, and blocking Applebot-Extended does not affect Siri or Spotlight.

How to Read Your Results

The matrix gives each bot a verdict of Allowed, Blocked, or Allowed (partial), plus a note showing which rule governed it — a bot’s own named rule, the catch-all User-agent: * group, or the default that applies when nothing matches. Remember that robots.txt is permissive by default: if no rule matches a bot, it is allowed. “Allowed (partial)” means the bot can reach your homepage but some paths inside its group are disallowed.

The 2026 Consensus: Block Training, Allow Search

Across large samples of robots.txt files, a clear pattern has settled in: site owners increasingly block the training crawlers while allowing the search and retrieval ones. The logic is economic. Traditional search crawlers send traffic back in rough proportion to what they take, but training crawlers can fetch enormous volumes of pages while returning almost no direct visitors. Reported crawl-to-referral ratios for training-focused crawlers run into the tens of thousands of pages fetched per visitor referred, compared with a handful of pages per referral for a conventional search bot. That asymmetry is why “block training, allow search” has become the default recommendation — though the right answer depends on your own goals for content rights, bandwidth, and AI visibility.

One important exception for our audience: bingbot is not just a search crawler — it also feeds Microsoft Copilot. Blocking it removes you from Bing Search and from Copilot answers at the same time, so treat it with care.

Want the plain-language version of this decision? See AI crawlers, robots.txt, and llms.txt, which also explains why llms.txt is not a substitute for robots.txt.

What Is llms.txt?

llms.txt is a proposed standard for a Markdown file at /llms.txt that gives AI systems a clean, curated map of your most important content — without the navigation, ads, and scripts that clutter a normal HTML page. The specification is strict about structure: the file must begin with a single H1 naming the site or project (the only strictly required element), followed by a one-line blockquote summary, optional free-text context, and then H2 sections containing Markdown link lists. A special ## Optional section marks links that can be skipped when an assistant’s context window is tight. The validator in the second tab checks each of these rules and flags the common mistakes: a missing H1, a missing summary, relative URLs instead of absolute ones, and list items with no link.

Why This Tool Can’t Fetch Your robots.txt Automatically

Browsers block scripts from reading files on other domains for security reasons, which is why this tool asks you to paste the file rather than typing in a URL. The upside is real privacy: your robots.txt and llms.txt are analyzed entirely on your own device, and nothing you paste is transmitted or stored. To grab your live file, open yoursite.com/robots.txt in a browser tab, copy everything, and paste it above.

Related Tools & Resources

Define any unfamiliar crawler, robots.txt, or DNS term in the Networking Glossary, then explore more of our free networking tools: run a DNS lookup to inspect a domain’s records, check the PTR record for an IP to verify a crawler’s identity, look up who owns a domain or IP, or see your own public IP address.

New to this topic? Our guide to AI crawlers, robots.txt, and llms.txt covers what each file actually controls, and which bots are worth allowing.