AI Crawlers, robots.txt, and llms.txt: What to Know

A plain-language guide to how AI bots read your site, and what robots.txt and llms.txt actually control.

AI crawlers visit your website every day, often without you noticing. Some collect text to train large language models. Others fetch pages so chatbots can cite you in answers.

Two text files get mentioned constantly: robots.txt and llms.txt. One of them works, and one of them mostly does not. This guide explains both in plain terms, with no hype.

What are AI crawlers?

AI crawlers are automated bots that download web pages for artificial intelligence systems. Some gather text to train models. Others fetch live pages so AI search tools can answer questions and cite sources. Each bot identifies itself with a user-agent name you can recognize.

Think of AI crawlers in three broad groups.

• Training crawlers collect content to build and improve AI models.
• Search and retrieval crawlers index pages so AI tools can cite them.
• User-triggered fetchers grab a single page when a person asks a chatbot about it.

Most reputable bots announce who they are in their user-agent string. That string is the name you target in robots.txt.

How robots.txt controls AI crawlers

robots.txt is a plain text file at the root of your site. It lists rules for each named bot. You can allow or block a crawler by its user-agent. Reputable AI bots read this file and obey it. The system is voluntary, not enforced.

A robots.txt file uses simple, repeatable blocks. Each block starts with a User-agent line. After that, you add Allow or Disallow rules.

Here is the basic shape of one rule.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

This blocks OpenAI’s training bot but allows its search bot. Each bot is controlled on its own line.

One important limit applies to every robots.txt rule. It is a request, not a lock. Honest bots comply, but bad actors may ignore it.

The main AI crawlers and what each does

Each major AI company runs one or more named crawlers. Some bots train models, while others power AI search and citations. Blocking a training bot is different from blocking a search bot. The table below lists the common bots and their jobs in 2026.

Verified June 2026. Bot names and roles change over time. Recheck the vendor docs before you rely on them.

A few details matter when you read this table.

• GPTBot is for training; OAI-SearchBot is for ChatGPT search results.
• Google-Extended is an opt-out signal, not a bot in your logs.
• Blocking Google-Extended does not remove you from normal Google Search.
• CCBot feeds Common Crawl, an open dataset used to train many models.
• Bingbot indexes pages for Bing, which also feeds Microsoft Copilot answers.

How to allow or block specific bots

You control each bot by editing your robots.txt file. A common strategy allows AI search bots but blocks training bots. That keeps your pages eligible for AI citations while limiting model training. Test your file after changes, because one wrong line can block the wrong bot.

Here is a balanced example you can adapt.

# Allow AI search and citation bots
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Block AI training bots
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

# Opt out of Google AI training only
User-agent: Google-Extended
Disallow: /

Place this file at your domain root, such as yoursite.com/robots.txt. Add a Sitemap line so crawlers find your pages faster.

After editing, check the file in a robots.txt tester. Our AI crawler checker shows which bots your rules allow.

What is llms.txt and why it is contested

llms.txt is a proposed file that points AI models to your best content. A developer named Jeremy Howard suggested it in 2024. It is not an official standard, and no major AI vendor has committed to reading it. Google has said its systems do not use the file.

Status checked June 2026. Adoption stays low, and vendor support could change. Recheck before you rely on it.

robots.txt and llms.txt sound similar, but they do different jobs. robots.txt controls access; llms.txt only suggests what to read.

The idea behind llms.txt is reasonable on paper. It offers a clean, curated map of your key pages. Coding and documentation sites see the most value from it.

The problem is simple: the bots that matter mostly ignore it. Server logs show AI crawlers rarely request the file. For now, treat llms.txt as optional, low-effort future-proofing.

Why this matters for AI search visibility

Search now happens inside AI answers, not only classic links. If a bot cannot read your page, it cannot cite you. A citation in an AI answer can carry more weight than a click. For sites like NetworkCheckr, being the cited source is the real goal.

Much of NetworkCheckr’s traffic comes through Bing and AI search. That makes AI citations a real channel, not a side note.

The takeaway is practical, not technical.

• Allow the AI search bots that send you citations and visits.
• Block training bots only if you have a clear reason.
• Keep robots.txt current, since new bots appear often.

Visibility in AI answers starts with one open door: your robots.txt file.

Frequently asked questions

These answers cover the common questions about AI crawlers and control files. They explain what robots.txt and llms.txt can and cannot do. Use them to decide which bots to allow or block on your own site.

Can robots.txt stop every AI crawler?

No. robots.txt is a voluntary request, not a hard block. Well-known bots from major companies obey it. Some rogue scrapers ignore it entirely. For real enforcement, use firewall or CDN rules instead.

Will blocking GPTBot remove me from ChatGPT search?

No. GPTBot is the training crawler, not the search crawler. ChatGPT search uses OAI-SearchBot instead. Block GPTBot to limit training, but allow OAI-SearchBot to stay in search answers.

Does llms.txt improve my AI search rankings?

No clear evidence supports that today. Major AI vendors have not committed to reading llms.txt. Studies show crawlers rarely fetch the file. It may still help coding and documentation sites. Treat it as optional for now.

What is the difference between robots.txt and llms.txt?

robots.txt controls which bots may access your pages. It is an established web standard that crawlers respect. llms.txt only suggests which content models should prioritize. It is a proposal, not a standard, and adoption is low.

Which AI bots should most websites allow?

Most sites benefit from allowing AI search and citation bots. These include OAI-SearchBot, PerplexityBot, and Bingbot. They can send real traffic and citations your way. Blocking them removes your pages from those AI answers.

Where do I put robots.txt and llms.txt files?

Both files belong in your website’s root directory. A visitor should reach them at yoursite.com/robots.txt and yoursite.com/llms.txt. Crawlers look there first by default. Place them anywhere else and most bots will miss them.

Related tools and resources

These internal pages go deeper on the topics above. Use them to find your IP, read DNS records, or understand a WHOIS lookup. Each one connects back to a free NetworkCheckr tool you can run right now.

• AI Crawler Checker — test which bots your robots.txt allows.
• DNS Records Explained — see what each DNS record type does.
• How to Read a WHOIS Lookup — decode domain ownership records.

References

These sources are primary vendor and standards pages, not summaries. They document the crawler names and the llms.txt proposal directly. Check them when you need the current, official details for your own robots.txt setup.

• OpenAI — Overview of OpenAI Crawlers. platform.openai.com/docs/bots
• Google — Google crawlers and user-agents (Google-Extended). developers.google.com
• Anthropic — ClaudeBot and site-owner controls. support.anthropic.com
• Common Crawl — CCBot. commoncrawl.org/ccbot
• llms.txt proposal. llmstxt.org