How do AI crawlers read my website, and are we letting them in?

Most founders have never looked at their robots.txt file and could not say what a crawler is. That was fine when the only crawler that mattered was Google's and your web agency handled it. It matters now, because a different set of crawlers decides whether ChatGPT, Claude, Copilot and Perplexity can read your site at all, and firms regularly shut them out without knowing.

A crawler is a program that visits web pages, reads them and follows links, so that a search engine or an AI assistant knows what is there. Google's crawler, Googlebot, has been visiting your site for twenty years. The AI crawlers, with names like GPTBot, ClaudeBot and PerplexityBot, are newer and behave differently. The most important difference is JavaScript: when Vercel and MERJ studied AI crawler traffic in December 2024, they found that "none of the major AI crawlers currently render JavaScript". Googlebot does.

They are not rare visitors either. In the same study, GPTBot, Claude's crawler, AppleBot and PerplexityBot together made nearly 1.3 billion fetches a month across Vercel's network, a little over 28% of Googlebot's volume. This guide explains how the two kinds of crawler differ, which names to know in 2026, how to check whether you are letting them in, and how to decide whether you should.

#How does Googlebot read your site differently from AI crawlers?

Googlebot renders your pages like a browser, including the JavaScript, and reads them to rank them. The main AI crawlers read only the HTML your server sends, and they read it so an assistant can describe you, cite you or learn from you.

Because Googlebot renders JavaScript, a script-built site can look fine in Google, and in Gemini and Google's AI features that draw on Google's index, while ChatGPT, Claude and Perplexity see a near-empty page. That gap is the most common technical reason a firm that ranks well is missing from AI answers.

Crawl frequency varies widely by crawler. Some AI crawlers visit rarely; others are heavy. Cloudflare reported in August 2025 that Perplexity's declared crawler made 20 to 25 million requests a day across its network. What matters more for your firm is that assistants which search the web live, such as ChatGPT search and Perplexity, can fetch a changed page within days or weeks, while answers drawn from a model's training change only when the model is retrained, which takes months.

#Which AI crawlers should you know by name?

Know the search crawlers and user-triggered fetchers first, because they decide whether you appear in live answers. The training crawlers decide only whether your content helps train future models.

The sources are each company's own documentation: OpenAI, Anthropic (April 2026) and Perplexity, plus Google's, Apple's and Meta's crawler pages. If your robots.txt still names older Anthropic user agents copied from a 2023 template, replace them: Anthropic now documents only ClaudeBot, Claude-SearchBot and Claude-User.

#Does ChatGPT follow robots.txt?

Its crawlers do. GPTBot and OAI-SearchBot obey robots.txt, but OpenAI's documentation says that because ChatGPT-User visits are "initiated by a user, robots.txt rules may not apply". So a robots.txt block controls training and ChatGPT search, but is not a complete opt-out from ChatGPT reading your page.

The same split exists elsewhere. Perplexity says Perplexity-User "generally ignores robots.txt rules", and Cloudflare's August 2025 report found that Perplexity had used undeclared crawlers to reach sites that had blocked it; Cloudflare removed Perplexity from its verified bots list. In the same tests, Cloudflare found ChatGPT-User fetched the robots.txt file and stopped when it was disallowed. Anthropic says all three of its bots respect robots.txt. If you need content kept away from every AI tool, put it behind a login.

#How do you check whether you are letting AI crawlers in?

Read your robots.txt file, check your Cloudflare or firewall settings, and look at a key page the way a crawler sees it. The five steps below take under an hour.

#Step 1: Read your robots.txt file

Type your domain followed by /robots.txt into a browser. Look for "User-agent:" lines naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended or Meta-ExternalAgent, and read the lines beneath. "Disallow: /" means that crawler is shut out of the whole site.

Then check the structure. A crawler obeys only the group with the most specific user agent that matches it, and Google's robots.txt documentation is explicit that specific groups and the "*" group "are not combined". If you name GPTBot in its own group with "Allow: /", it ignores any "Disallow: /admin/" you put under "User-agent: *".

#Step 2: Check Cloudflare, your host and your firewall

Security services can block AI crawlers before robots.txt is ever read. Cloudflare changed its default to block AI crawlers in July 2025, and its documentation says that from 15 September 2026 new domains block training and agent crawlers on pages that show ads, while search crawlers stay allowed.

In the Cloudflare dashboard, open your domain and go to Security Settings, then Configure AI bot policies (older accounts show a Block AI bots switch). You can set search, agent and training crawlers separately to block on all pages, block on pages with ads, or allow. If you use another host, CDN or security plugin, ask whoever manages it whether any AI or bot protection is on and which names it blocks.

#Step 3: See what a crawler actually receives

Open a key page, right-click and choose "View page source", or ask your developer to fetch it with JavaScript off. Search the source for your main heading and first paragraph. If they are missing, AI crawlers are not seeing them.

Example: an illustrative services page built entirely by script.

The fix is to have the server send the page text in the HTML, with JavaScript adding interaction on top. Our guide to how your website platform affects whether ChatGPT can read it covers the common builders.

#Step 4: Look at your server logs

Your logs record every visit and its user agent. If GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot never appear, they are blocked or have not found you. If they appear with error codes such as 403, something is refusing them.

#Step 5: Decide and write it down

Record which crawlers you allow and why, so the next developer does not quietly reverse it. Review it every quarter and after any rebuild, change of host or CDN.

#Should you block GPTBot but allow OAI-SearchBot?

You can, and it is a sensible middle ground if you object to training use. Blocking GPTBot keeps your content out of OpenAI's model training, while allowing OAI-SearchBot keeps you eligible for ChatGPT search answers, which is where recommendations increasingly come from.

For most B2B firms we recommend allowing both. Your content is written to win clients, and a training block means future models know less about you. Whatever you choose, repeat the private folders under every named group. This example allows the main AI crawlers everywhere except two private folders:

User-agent: GPTBot
Disallow: /admin/
Disallow: /private/

User-agent: OAI-SearchBot
Disallow: /admin/
Disallow: /private/

User-agent: ClaudeBot
Disallow: /admin/
Disallow: /private/

User-agent: Claude-SearchBot
Disallow: /admin/
Disallow: /private/

User-agent: PerplexityBot
Disallow: /admin/
Disallow: /private/

User-agent: *
Disallow: /admin/
Disallow: /private/

To block training but stay in ChatGPT search, change only the GPTBot group to "Disallow: /". The same pattern works for ClaudeBot (training) and Claude-SearchBot (search).

#Do AI crawlers slow down your website or add to hosting costs?

For a typical B2B site of a few hundred pages, rarely enough to notice, in our experience. The heavy volumes in the studies are totals across millions of sites.

If your logs show one crawler making thousands of requests a day, slow it rather than block it. Anthropic's bots support the Crawl-delay line in robots.txt, and Cloudflare and most CDNs can rate-limit a named bot. Blocking a search crawler to save a little bandwidth removes you from the answers your buyers read.

#What should you build so both kinds of crawler read you well?

Put the page text in the HTML, use one clear heading hierarchy, and write sections that make sense on their own. Almost everything that helps AI crawlers also helps Google.

Structured data helps search engines read your facts correctly, but Google says its AI features need no special markup, and OpenAI and Anthropic have not said they read it; keep it accurate rather than treating it as a ranking trick. Our guide to schema markup and AI assistants explains what is worth adding, and the full list of checks is in our technical checklist for an AI-ready website.

#What to do this month

1. Open your robots.txt file and list every AI crawler it names, with what each is allowed to do.
2. Replace any older Anthropic names with ClaudeBot, Claude-SearchBot and Claude-User, and repeat your private folders under every named group.
3. Check the AI bot settings in Cloudflare, or ask your host or IT provider what their bot protection blocks.
4. View the source of your three most important pages and confirm the main text is there without JavaScript.
5. Search your server logs for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and note any error codes.
6. Write down your decision on each crawler and put a quarterly review in the diary.

#Frequently asked questions

#If we block GPTBot, will we disappear from ChatGPT?

Not from ChatGPT search, if OAI-SearchBot is still allowed. Blocking GPTBot stops your content being used for training, so future models may know less about you. Blocking OAI-SearchBot is what removes you from ChatGPT search answers.

#Will blocking AI crawlers protect our Google rankings?

It makes no difference to them. Googlebot is separate, and Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Google-Extended controls use in Gemini training, not AI Overviews, which draw on Google's normal index.

#Is our Webflow or Wix site readable by ChatGPT?

Usually, but check rather than assume. Hosted builders generally send the main page text in the HTML, but text inside embedded widgets, tabs, sliders or third-party apps may only appear after scripts run. The view-source test in Step 3 settles it in two minutes, page by page.

#How quickly will ChatGPT notice a change to our site?

Assistants that search the web live, including ChatGPT search, Perplexity, Copilot and Google's AI features, can reflect a changed page within weeks. Anything the model learned in training changes only when the model is retrained, which takes months. Fix wrong facts early, as set out in why ChatGPT describes your firm wrongly.

#Our site is an app-style build. Do we have to rebuild it?

Not necessarily. Most modern frameworks can render public pages on the server or generate them in advance, so the HTML arrives complete. Ask your developer about server-side rendering or static generation for the marketing pages. If the platform cannot do it, that is a strong case for moving the marketing site.

#How often should we review our crawler settings?

Every quarter, and after any rebuild or change of host. New crawlers appear, names change, CDNs change their defaults, and developers copy robots.txt files from old projects.

You cannot be recommended by an assistant that cannot read you, and ten minutes with robots.txt and a page's source will tell you whether that is your problem. For a check of how your site reads to AI crawlers and buyers, book a free Growth Review; if the site needs rebuilding so the content is in the HTML, that is Website That Sells.

#Sources