# Is our website technically ready for ChatGPT and AI search?

Canonical: https://fortitudemedia.ai/insights/technical-checklist-ai-ready-website
Author: David Adams
Published: 2026-03-31
Last reviewed: 2026-09-24
Reviewed by: David Adams
Pillar: Websites that bring work in
Summary: A technical checklist to make your site readable by ChatGPT, Claude and Perplexity: crawler access, JavaScript, headings, schema, titles and dates.
Keywords: ai ready website checklist, gptbot vs oai-searchbot, is cloudflare blocking chatgpt, block ai training allow chatgpt search, does chatgpt use bing, see website as gptbot
Short answer: Your site is ready for AI search when the search crawlers behind ChatGPT, Claude, Perplexity, Copilot and Google can get in, and your key pages show their full text without JavaScript. Check robots.txt, your Cloudflare or firewall settings and Bing Webmaster Tools first. Then give each page one clear topic, proper headings, visible dates and accurate structured data. Most fixes are small and need no rebuild, and none guarantees a recommendation.
What changed: Split crawlers into training, search and user bots, added Bing, Cloudflare's default block, a fetch-as-GPTBot test, a printable checklist, WCAG 2.2 and INP.
Publisher: Fortitude Media Limited

When quoting, attribute to Fortitude Media and link to the canonical URL above.

---
Sooner or later someone on your board or partner group asks whether the website is "ready for AI". The web agency says it is, because the site is fast and ranks well on Google. That answer is often wrong, and not because anyone is being dishonest. A site can do well in Google and still be hard for ChatGPT, Claude or Perplexity to read, because their crawlers fetch pages differently.

The difference is not academic. [Vercel and MERJ found in December 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler) that GPTBot made 569 million requests and Claude's crawler 370 million across Vercel's network in a month, together about 20% of Googlebot's 4.5 billion, and that "none of the major AI crawlers currently render JavaScript." A page that builds its words with scripts can look perfect in Google and empty to ChatGPT.

The problems we find most often are the same: a security setting blocks AI crawlers without anyone knowing, service pages are built with scripts some crawlers never run, and the homepage says nothing a machine can pin down about who the firm is. Each is fixable, most in days. This is the checklist we use, written for the founder who needs to know what to ask. Hand it to whoever looks after your site.

## Key takeaways

- A website is ready for AI search when crawlers can get in, read the full text and tell who you are and what each page covers.
- Each AI company now runs separate crawlers for training, for search and for fetching a page when a user asks, and blocking one is not the same as blocking another.
- Check robots.txt, Cloudflare or firewall settings and Bing Webmaster Tools first, because Copilot searches Bing and ChatGPT search has drawn on Bing's index.
- The main AI crawlers do not run JavaScript, so key text must be in the HTML your server sends.
- Clear headings, one topic per page, visible dates and accurate structured data help machines read and quote you correctly.
- None of this guarantees a recommendation; it removes the technical reasons an assistant might pass you over.

## What does "AI-ready" actually mean for a website?

Three things. AI crawlers are allowed in. They can read the full text of your important pages. And they can tell, without guessing, what each page is about and who your firm is. If any of the three fails, the quality of your content hardly matters.

The technical side is the foundation, not the building. Whether an assistant recommends you also depends on what your pages say, how specific your proof is and what other sites say about you. We explain how AI crawlers differ from Google's in [how AI crawlers read your website](/insights/ai-crawlers-vs-google-spiders).

## The checklist to hand to your developer

Print this and go through it together. Each line is a yes or a no.

| Check | How to test | Who fixes it | Pass or fail |
|---|---|---|---|
| Search crawlers allowed: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot | Open yourdomain/robots.txt and read each named group | Developer | Pass / Fail |
| Training crawlers (GPTBot, ClaudeBot, Google-Extended) set by a deliberate decision | Same file; decision written down | Founder decides, developer sets | Pass / Fail |
| No firewall or CDN block on AI crawlers | Cloudflare: Security Settings, AI bot policies; ask any other host | Developer or host | Pass / Fail |
| Site verified in Bing Webmaster Tools, sitemap submitted | Log in, check indexed pages and crawl errors | Developer or marketing | Pass / Fail |
| Key pages indexed in Google Search Console | URL Inspection on each key page | Developer or marketing | Pass / Fail |
| Key text present without JavaScript | View page source, or fetch as GPTBot (below) | Developer | Pass / Fail |
| One H1 and logical H2 and H3 headings | View source or a headings checker | Content owner | Pass / Fail |
| Each section opens with a direct answer | Read the first two sentences of each section | Content owner | Pass / Fail |
| Organization and Person markup accurate | Rich Results Test or Schema Markup Validator | Developer | Pass / Fail |
| Unique titles and descriptions; no stray noindex or nosnippet | A crawler tool, or view source | Developer | Pass / Fail |
| Published and updated dates on articles | Look on the page and in the markup | Content owner | Pass / Fail |
| Core Web Vitals met | PageSpeed Insights on key pages | Developer | Pass / Fail |
| WCAG 2.2 AA basics: contrast, alt text, keyboard use | An accessibility checker and a keyboard test | Developer and content owner | Pass / Fail |

## Can AI crawlers get into your site?

Check this first, because it overrides everything else. A crawler is the program a company sends to read web pages. Each AI company now runs several, with different jobs, and your robots.txt file, at the root of your domain, decides which get in.

| Company | Training | Search | Fetch when a user asks |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | (none named) | PerplexityBot | Perplexity-User |
| Google | Google-Extended (a control, not a crawler) | Googlebot | (none named) |
| Microsoft | (none named) | Bingbot | (none named) |

The split matters. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) says a site can "allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training", and that for ChatGPT-User, "robots.txt rules may not apply" because a user started the visit. [Anthropic says](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) blocking ClaudeBot excludes future content from training, while blocking Claude-SearchBot or Claude-User reduces your visibility in Claude's answers. [Perplexity says](https://docs.perplexity.ai/guides/bots) its user fetcher "generally ignores robots.txt rules". Google says Google-Extended controls use in Gemini training and does not affect inclusion in Google Search, which includes AI Overviews.

A robots.txt that lets the search and user crawlers in, keeps an admin folder private and sets training crawlers separately looks like this:

```text
# Search and user-request crawlers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Disallow: /admin/

# Training crawlers: change to "Disallow: /" to opt out of training
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /admin/

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.co.uk/sitemap.xml
```

A crawler follows only the most specific group that names it, so repeat private folders under every group. OpenAI says changes can take about 24 hours to take effect. If your file still names older Anthropic user agents from a 2023 template, replace them with ClaudeBot, Claude-SearchBot and Claude-User.

### Check Cloudflare and your firewall

Security services can block AI crawlers before robots.txt is read. Cloudflare [changed its default on 1 July 2025](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/), "changing the default to block AI crawlers unless they pay creators for their content." [Cloudflare's documentation](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) puts the setting under Security Settings, then Configure AI bot policies, with "Block AI bots" on older accounts. It treats search, training and agent crawlers separately, each set to block on all pages, block on pages with ads, or allow. From 15 September 2026, it says, training and agent bots are blocked by default on pages that show ads while search stays allowed. If you use another host or security plugin, ask what its bot protection blocks.

### Check Bing as well as Google

[Microsoft says](https://learn.microsoft.com/en-us/copilot/microsoft-365/manage-public-web-access) Copilot "may fetch information from the Bing search service" when it answers, and ChatGPT search has drawn on Bing's index too. A site that Bing cannot crawl, or has never been submitted to it, loses both. Verify the site in Bing Webmaster Tools, submit your sitemap, and check for crawl errors, just as you would in Google Search Console.

## Can they read your pages without running JavaScript?

They should. Many modern sites send a nearly empty page and let code in the browser fill in the words. Googlebot runs that code, which helps Gemini and Google's AI features. GPTBot, ClaudeBot and PerplexityBot fetch the page but do not run the code, so they may see little or nothing.

### How to see your site the way GPTBot sees it

Open a key page, right-click and choose "View page source", then search for your first paragraph. Or switch off JavaScript in the browser and reload. For an exact test, a developer can run one line that fetches the page with GPTBot's published user agent and counts how often a phrase from your first paragraph appears:

```text
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://www.example.co.uk/services/ | grep -c "a phrase from your first paragraph"
```

A result of 1 or more means the text is in the HTML. A 0 means crawlers that do not run JavaScript cannot see it. An error page may mean your firewall blocks the name, or that it treats a test from your own computer as an impostor, so check the firewall log before concluding either way.

Also watch for text that loads only when a tab or accordion is opened, key facts locked in PDFs or images, and case studies pulled in from another system after the page loads. The fix is usually to have the server send the finished page. We cover platforms in [how your website platform affects whether ChatGPT can read your site](/insights/cms-choice-affects-ai-visibility).

## Is each page clearly about one thing?

It should be. Assistants answer specific questions, so they favour pages that deal with one subject clearly. Five pages that each cover one service properly are easier to quote than one page covering five.

- **Use headings properly.** One H1 that says what the page is about, then H2 and H3 headings in order. Headings phrased as buyers' questions, such as "How long does an ISO 27001 audit take?", make matching easy.
- **Use the right building blocks.** Semantic HTML tags mark the main article, the navigation, lists and tables, which helps crawlers separate content from menus. Comparison data should be a real table, not a picture of one.
- **Put the answer first.** Start each section with a plain sentence that answers its heading, then explain.

How pages link together matters too, as we explain in [how to organise your website so buyers and AI understand it](/insights/website-architecture-ux-and-ai).

## Does your site say plainly who you are?

It should, in words and in structured data. Structured data, often called schema markup, is a set of labels in the page code that states facts in a standard form: this is an organisation called Example Ltd, at this address, with these people. It helps search engines read your facts correctly. Google says its AI features need no special markup, and it stopped showing FAQ rich results on 7 May 2026. OpenAI and Anthropic have not said they read it. Keep it accurate; do not treat it as a ranking trick.

The pieces that matter most for a B2B firm are Organization markup on the homepage, Person markup on partner and author pages, Article markup with published and updated dates on guides, and Breadcrumb markup. A cut-down example for a fictional firm:

```html
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Ltd",
  "url": "https://www.example.co.uk",
  "logo": "https://www.example.co.uk/logo.png",
  "sameAs": ["https://www.linkedin.com/company/example-ltd"]
}
</script>
```

The markup must match what the page says, and the facts must match LinkedIn, Google Business Profile and directories, because conflicting details are a common reason assistants describe firms wrongly. We cover which types are worth the effort in [whether you need schema markup for AI assistants](/insights/structured-data-schema-markup-ai).

## Are your titles, dates and page settings correct?

They are easy to overlook and easy to fix. Every important page needs a unique title and meta description. Articles should show when they were published and last updated, on the page and in the markup. A canonical tag should point to the main version of each page. Check that no important page carries a noindex instruction left over from a test site, and that nobody has added nosnippet or a very small max-snippet setting, which limits how much Google can quote, including in AI Overviews.

## Is the site fast, stable and accessible?

It should be, because slow and awkward pages lose people. [Google's Core Web Vitals guidance](https://developers.google.com/search/docs/appearance/core-web-vitals) sets three targets: the main content appears within 2.5 seconds, the page responds to a tap or click in under 200 milliseconds (Interaction to Next Paint), and the layout shift score stays under 0.1.

For accessibility, the current standard is WCAG 2.2, which [the W3C notes](https://www.w3.org/WAI/standards-guidelines/wcag/) is also an approved ISO standard, ISO/IEC 40500:2025. Level AA asks, among other things, for a contrast ratio of at least 4.5 to 1 for normal text, descriptive alt text on meaningful images, and pages that work with a keyboard. Alt text also gives crawlers words where there would otherwise be none. More in [whether your website's speed and accessibility affect enquiries](/insights/website-accessibility-speed-matter).

## What to do this month

1. Ask your developer to read robots.txt and the Cloudflare or firewall settings, and confirm the search crawlers can reach the site.
2. Decide, in writing, whether to allow GPTBot and ClaudeBot for training, and set robots.txt to match.
3. Verify the site in Bing Webmaster Tools and submit your sitemap.
4. Run the view-source or GPTBot test on your homepage, main service page and best case study.
5. Fix the H1, headings, answer-first openings and any stray noindex on those three pages.
6. Add or correct Organization and Person markup, and validate it before it goes live.

## Frequently asked questions

### What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot indexes pages so they can appear in ChatGPT's search answers. Blocking GPTBot keeps your content out of training but leaves you eligible for ChatGPT search; blocking OAI-SearchBot removes you from those answers.

### Can we block AI training but still appear in ChatGPT search?

Yes. Set GPTBot to "Disallow: /" and leave OAI-SearchBot allowed, as OpenAI's documentation describes. The same split works for Anthropic: block ClaudeBot for training and allow Claude-SearchBot. For most B2B firms we recommend allowing both, because a training block means future models know less about you.

### How do I check whether Cloudflare is blocking ChatGPT from my site?

In the Cloudflare dashboard, open your domain and go to Security Settings, then Configure AI bot policies (or Block AI bots on older accounts), and check how search, training and agent crawlers are set. Then look at the security events log for blocked requests from GPTBot, OAI-SearchBot or ChatGPT-User. If you do not manage Cloudflare yourself, ask whoever does to send a screenshot.

### Does ChatGPT use Bing to find websites?

OpenAI says ChatGPT search uses third-party search providers without naming them, and ChatGPT search has drawn on Bing's index. Microsoft Copilot searches Bing directly. So being crawled and indexed by Bing matters for both, which is why Bing Webmaster Tools is on the checklist.

### Should we block AI crawlers to protect our content?

For most B2B firms, no. If buyers ask ChatGPT and Perplexity who to hire, blocking their search crawlers removes you from the answers. The exception is content you sell, such as paid reports, which belongs behind a login, because user-triggered fetchers may not obey robots.txt.

### Can we do this on WordPress, Squarespace or Wix?

Mostly. WordPress gives the most control. Wix has a robots.txt editor. Squarespace offers [a tick box that blocks a list of AI crawlers](https://support.squarespace.com/hc/en-us/articles/360022347072), including GPTBot and ClaudeBot, rather than direct editing, so check what it covers before you tick it. We compare the options in [template or custom website: which does a B2B firm need?](/insights/template-vs-custom-website-b2b)

### How often should we repeat this check?

Run the full checklist once a year and after any redesign, platform change or new security set-up. Check crawler access and key pages every quarter, because plugins, hosting changes and CDN defaults can quietly undo earlier work. Live-search assistants can reflect a fix within weeks; answers from a model's training change only when it is retrained.

The technical side is the part of AI search most firms can fix quickly, and it is often what stands between good content and being named. For a view of how your site reads to AI crawlers and buyers, [book a free Growth Review](/growth-review); if the site needs rebuilding so the content is in the HTML, that is [Website That Sells](/website-that-sells).

## Sources

- [Vercel and MERJ, The rise of the AI crawler (December 2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler)
- [OpenAI, Overview of OpenAI crawlers (September 2026)](https://developers.openai.com/api/docs/bots)
- [Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (April 2026)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Cloudflare, Content Independence Day: no AI crawl without compensation (July 2025)](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/)
- [Cloudflare Docs, Block AI bots (July 2026)](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)
- [Microsoft Learn, Manage public web access in Microsoft Copilot (August 2026)](https://learn.microsoft.com/en-us/copilot/microsoft-365/manage-public-web-access)
- [Google Search Central, Understanding Core Web Vitals and Google search results (December 2025)](https://developers.google.com/search/docs/appearance/core-web-vitals)
- [W3C Web Accessibility Initiative, WCAG 2 overview (September 2026)](https://www.w3.org/WAI/standards-guidelines/wcag/)