# How do AI crawlers read my website, and are we letting them in?

Canonical: https://fortitudemedia.ai/insights/ai-crawlers-vs-google-spiders
Author: David Adams
Published: 2026-03-31
Last reviewed: 2026-09-24
Reviewed by: David Adams
Pillar: Getting named by AI assistants
Summary: How GPTBot, ClaudeBot and PerplexityBot read your site, how to check robots.txt and your firewall for accidental blocks, and which AI crawlers to let in.
Keywords: ai crawlers robots.txt, does chatgpt follow robots.txt, block gptbot allow oai-searchbot, cloudflare blocking ai crawlers, can chatgpt read my website, claudebot perplexitybot
Short answer: AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot fetch your pages but, unlike Googlebot, do not run JavaScript, so your key text must be in the HTML your server sends. Each obeys robots.txt by name, though fetchers acting on a user's question, such as ChatGPT-User, may not. Check robots.txt, your Cloudflare or firewall settings and a text-only view of your key pages. Most B2B firms should let the search crawlers in.
What changed: Fixed the robots.txt example, updated the crawler list for 2026, added Cloudflare's AI bot settings, user-triggered fetchers and sourced crawl data.
Publisher: Fortitude Media Limited

When quoting, attribute to Fortitude Media and link to the canonical URL above.

---
Most founders have never looked at their robots.txt file and could not say what a crawler is. That was fine when the only crawler that mattered was Google's and your web agency handled it. It matters now, because a different set of crawlers decides whether ChatGPT, Claude, Copilot and Perplexity can read your site at all, and firms regularly shut them out without knowing.

A crawler is a program that visits web pages, reads them and follows links, so that a search engine or an AI assistant knows what is there. Google's crawler, Googlebot, has been visiting your site for twenty years. The AI crawlers, with names like GPTBot, ClaudeBot and PerplexityBot, are newer and behave differently. The most important difference is JavaScript: when [Vercel and MERJ studied AI crawler traffic in December 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler), they found that "none of the major AI crawlers currently render JavaScript". Googlebot does.

They are not rare visitors either. In the same study, GPTBot, Claude's crawler, AppleBot and PerplexityBot together made nearly 1.3 billion fetches a month across Vercel's network, a little over 28% of Googlebot's volume. This guide explains how the two kinds of crawler differ, which names to know in 2026, how to check whether you are letting them in, and how to decide whether you should.

## Key takeaways

- Googlebot runs JavaScript; the main AI crawlers fetch pages but do not, so text that only appears after scripts load is invisible to them.
- Each AI company now runs separate crawlers for training, for search and for fetching a page when a user asks, and you can allow or block each by name.
- Blocking a training crawler such as GPTBot does not remove you from ChatGPT search; blocking OAI-SearchBot does.
- robots.txt is not a complete opt-out: OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
- A crawler follows only the most specific group in robots.txt, so private folders must be listed under every crawler you name, not just under the asterisk.
- Cloudflare and other security services can block AI crawlers whatever robots.txt says, so check the firewall as well as the file.
- For most B2B firms, letting the search crawlers in costs little and is the precondition for being named.

## How does Googlebot read your site differently from AI crawlers?

Googlebot renders your pages like a browser, including the JavaScript, and reads them to rank them. The main AI crawlers read only the HTML your server sends, and they read it so an assistant can describe you, cite you or learn from you.

| | Googlebot | AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) |
|---|---|---|
| Runs JavaScript | Yes, in a headless browser | No: they fetch script files but do not run them (Vercel and MERJ, December 2024) |
| Purpose | Rank pages for Search, including AI Overviews and AI Mode | Train models, index for AI search, or fetch a page for one user's question |
| How often it visits | Often, weighted to important pages | Varies by crawler; some now crawl at very high volume |
| Controlled by | robots.txt, Search Console | robots.txt by name, except some user-triggered fetchers |

Because Googlebot renders JavaScript, a script-built site can look fine in Google, and in Gemini and Google's AI features that draw on Google's index, while ChatGPT, Claude and Perplexity see a near-empty page. That gap is the most common technical reason a firm that ranks well is missing from AI answers.

Crawl frequency varies widely by crawler. Some AI crawlers visit rarely; others are heavy. [Cloudflare reported in August 2025](https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/) that Perplexity's declared crawler made 20 to 25 million requests a day across its network. What matters more for your firm is that assistants which search the web live, such as ChatGPT search and Perplexity, can fetch a changed page within days or weeks, while answers drawn from a model's training change only when the model is retrained, which takes months.

## Which AI crawlers should you know by name?

Know the search crawlers and user-triggered fetchers first, because they decide whether you appear in live answers. The training crawlers decide only whether your content helps train future models.

| Company | Name | What it does | Follows robots.txt? | If you block it |
|---|---|---|---|---|
| OpenAI | GPTBot | Collects content that may be used to train models | Yes | Content not used for training; ChatGPT search unaffected |
| OpenAI | OAI-SearchBot | Indexes pages for ChatGPT search | Yes | You are not shown in ChatGPT search answers |
| OpenAI | ChatGPT-User | Visits a page when a user's request needs it | OpenAI says rules "may not apply" | A block may not stop it |
| Anthropic | ClaudeBot | Collects content for model training | Yes | Future content excluded from training |
| Anthropic | Claude-SearchBot | Indexes pages to improve Claude's search results | Yes | May reduce your visibility in Claude's search |
| Anthropic | Claude-User | Fetches pages when a user asks Claude a question | Yes | Claude cannot retrieve your page for users |
| Perplexity | PerplexityBot | Indexes pages for Perplexity's search; not used for training | Yes | You are less likely to appear in Perplexity |
| Perplexity | Perplexity-User | Visits a page when a user asks a question | Perplexity says it "generally ignores" them | A block may not stop it |
| Google | Googlebot | Crawls for Search, including AI Overviews and AI Mode | Yes | You leave Google Search, AI features included |
| Google | Google-Extended | Not a crawler: a token controlling use in Gemini training | Yes | No effect on Search, rankings or AI Overviews |
| Microsoft | Bingbot | Crawls for Bing, which feeds Copilot | Yes | You drop out of Bing and Copilot |
| Apple | Applebot-Extended | Not a crawler: controls use in training Apple's models | Yes | Pages can still appear in Apple's search features |
| Meta | Meta-ExternalAgent | Crawls for AI model training and indexing | Yes | Your content is not collected by it |

The sources are each company's own documentation: [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (April 2026) and [Perplexity](https://docs.perplexity.ai/guides/bots), plus Google's, Apple's and Meta's crawler pages. If your robots.txt still names older Anthropic user agents copied from a 2023 template, replace them: Anthropic now documents only ClaudeBot, Claude-SearchBot and Claude-User.

## Does ChatGPT follow robots.txt?

Its crawlers do. GPTBot and OAI-SearchBot obey robots.txt, but OpenAI's documentation says that because ChatGPT-User visits are "initiated by a user, robots.txt rules may not apply". So a robots.txt block controls training and ChatGPT search, but is not a complete opt-out from ChatGPT reading your page.

The same split exists elsewhere. Perplexity says Perplexity-User "generally ignores robots.txt rules", and Cloudflare's August 2025 report found that Perplexity had used undeclared crawlers to reach sites that had blocked it; Cloudflare removed Perplexity from its verified bots list. In the same tests, Cloudflare found ChatGPT-User fetched the robots.txt file and stopped when it was disallowed. Anthropic says all three of its bots respect robots.txt. If you need content kept away from every AI tool, put it behind a login.

## How do you check whether you are letting AI crawlers in?

Read your robots.txt file, check your Cloudflare or firewall settings, and look at a key page the way a crawler sees it. The five steps below take under an hour.

### Step 1: Read your robots.txt file

Type your domain followed by /robots.txt into a browser. Look for "User-agent:" lines naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended or Meta-ExternalAgent, and read the lines beneath. "Disallow: /" means that crawler is shut out of the whole site.

Then check the structure. A crawler obeys only the group with the most specific user agent that matches it, and [Google's robots.txt documentation](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) is explicit that specific groups and the "*" group "are not combined". If you name GPTBot in its own group with "Allow: /", it ignores any "Disallow: /admin/" you put under "User-agent: *".

### Step 2: Check Cloudflare, your host and your firewall

Security services can block AI crawlers before robots.txt is ever read. Cloudflare [changed its default to block AI crawlers in July 2025](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/), and its [documentation](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) says that from 15 September 2026 new domains block training and agent crawlers on pages that show ads, while search crawlers stay allowed.

In the Cloudflare dashboard, open your domain and go to Security Settings, then Configure AI bot policies (older accounts show a Block AI bots switch). You can set search, agent and training crawlers separately to block on all pages, block on pages with ads, or allow. If you use another host, CDN or security plugin, ask whoever manages it whether any AI or bot protection is on and which names it blocks.

### Step 3: See what a crawler actually receives

Open a key page, right-click and choose "View page source", or ask your developer to fetch it with JavaScript off. Search the source for your main heading and first paragraph. If they are missing, AI crawlers are not seeing them.

Example: an illustrative services page built entirely by script.

| What a visitor sees | What a crawler that does not run JavaScript receives |
|---|---|
| Heading, three paragraphs on the service, two case studies, team photos, contact form | Page title, menu links, footer, and an empty space where the content should be |

The fix is to have the server send the page text in the HTML, with JavaScript adding interaction on top. Our guide to [how your website platform affects whether ChatGPT can read it](/insights/cms-choice-affects-ai-visibility) covers the common builders.

### Step 4: Look at your server logs

Your logs record every visit and its user agent. If GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot never appear, they are blocked or have not found you. If they appear with error codes such as 403, something is refusing them.

### Step 5: Decide and write it down

Record which crawlers you allow and why, so the next developer does not quietly reverse it. Review it every quarter and after any rebuild, change of host or CDN.

## Should you block GPTBot but allow OAI-SearchBot?

You can, and it is a sensible middle ground if you object to training use. Blocking GPTBot keeps your content out of OpenAI's model training, while allowing OAI-SearchBot keeps you eligible for ChatGPT search answers, which is where recommendations increasingly come from.

For most B2B firms we recommend allowing both. Your content is written to win clients, and a training block means future models know less about you. Whatever you choose, repeat the private folders under every named group. This example allows the main AI crawlers everywhere except two private folders:

```
User-agent: GPTBot
Disallow: /admin/
Disallow: /private/

User-agent: OAI-SearchBot
Disallow: /admin/
Disallow: /private/

User-agent: ClaudeBot
Disallow: /admin/
Disallow: /private/

User-agent: Claude-SearchBot
Disallow: /admin/
Disallow: /private/

User-agent: PerplexityBot
Disallow: /admin/
Disallow: /private/

User-agent: *
Disallow: /admin/
Disallow: /private/
```

To block training but stay in ChatGPT search, change only the GPTBot group to "Disallow: /". The same pattern works for ClaudeBot (training) and Claude-SearchBot (search).

## Do AI crawlers slow down your website or add to hosting costs?

For a typical B2B site of a few hundred pages, rarely enough to notice, in our experience. The heavy volumes in the studies are totals across millions of sites.

If your logs show one crawler making thousands of requests a day, slow it rather than block it. Anthropic's bots support the Crawl-delay line in robots.txt, and Cloudflare and most CDNs can rate-limit a named bot. Blocking a search crawler to save a little bandwidth removes you from the answers your buyers read.

## What should you build so both kinds of crawler read you well?

Put the page text in the HTML, use one clear heading hierarchy, and write sections that make sense on their own. Almost everything that helps AI crawlers also helps Google.

Structured data helps search engines read your facts correctly, but Google says its AI features need no special markup, and OpenAI and Anthropic have not said they read it; keep it accurate rather than treating it as a ranking trick. Our guide to [schema markup and AI assistants](/insights/structured-data-schema-markup-ai) explains what is worth adding, and the full list of checks is in our [technical checklist for an AI-ready website](/insights/technical-checklist-ai-ready-website).

## What to do this month

1. Open your robots.txt file and list every AI crawler it names, with what each is allowed to do.
2. Replace any older Anthropic names with ClaudeBot, Claude-SearchBot and Claude-User, and repeat your private folders under every named group.
3. Check the AI bot settings in Cloudflare, or ask your host or IT provider what their bot protection blocks.
4. View the source of your three most important pages and confirm the main text is there without JavaScript.
5. Search your server logs for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and note any error codes.
6. Write down your decision on each crawler and put a quarterly review in the diary.

## Frequently asked questions

### If we block GPTBot, will we disappear from ChatGPT?

Not from ChatGPT search, if OAI-SearchBot is still allowed. Blocking GPTBot stops your content being used for training, so future models may know less about you. Blocking OAI-SearchBot is what removes you from ChatGPT search answers.

### Will blocking AI crawlers protect our Google rankings?

It makes no difference to them. Googlebot is separate, and Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Google-Extended controls use in Gemini training, not AI Overviews, which draw on Google's normal index.

### Is our Webflow or Wix site readable by ChatGPT?

Usually, but check rather than assume. Hosted builders generally send the main page text in the HTML, but text inside embedded widgets, tabs, sliders or third-party apps may only appear after scripts run. The view-source test in Step 3 settles it in two minutes, page by page.

### How quickly will ChatGPT notice a change to our site?

Assistants that search the web live, including ChatGPT search, Perplexity, Copilot and Google's AI features, can reflect a changed page within weeks. Anything the model learned in training changes only when the model is retrained, which takes months. Fix wrong facts early, as set out in [why ChatGPT describes your firm wrongly](/insights/ai-conflicting-information-online).

### Our site is an app-style build. Do we have to rebuild it?

Not necessarily. Most modern frameworks can render public pages on the server or generate them in advance, so the HTML arrives complete. Ask your developer about server-side rendering or static generation for the marketing pages. If the platform cannot do it, that is a strong case for moving the marketing site.

### How often should we review our crawler settings?

Every quarter, and after any rebuild or change of host. New crawlers appear, names change, CDNs change their defaults, and developers copy robots.txt files from old projects.

You cannot be recommended by an assistant that cannot read you, and ten minutes with robots.txt and a page's source will tell you whether that is your problem. For a check of how your site reads to AI crawlers and buyers, [book a free Growth Review](/growth-review); if the site needs rebuilding so the content is in the HTML, that is [Website That Sells](/website-that-sells).

## Sources

- [Vercel and MERJ, The rise of the AI crawler (December 2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler)
- [OpenAI, Overview of OpenAI crawlers (September 2026)](https://developers.openai.com/api/docs/bots)
- [Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (April 2026)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity, Perplexity crawlers (September 2026)](https://docs.perplexity.ai/guides/bots)
- [Google Search Central, How Google interprets the robots.txt specification (August 2026)](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
- [Cloudflare, Content Independence Day: no AI crawl without compensation (July 2025)](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/)
- [Cloudflare Docs, Block AI bots and AI bot policies (July 2026)](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)
- [Cloudflare, Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (August 2025)](https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/)