Can ChatGPT see your website? How to check in one minute

ChatGPT can see your website when three things are true: OpenAI's crawlers are allowed in, your content is in the page's own HTML, and the page is public. The quickest test is the free AI visibility checker, which reads your site's rules and shows which AI crawlers are let in. If any of the three fails, ChatGPT is working from other people's descriptions of your business, or from nothing.

The same three conditions apply to Claude and Perplexity, which run crawlers of their own. This guide covers what those crawlers are, the five things that most often block them, and how to check each one without a developer.

At a glance

  • ChatGPT uses three crawlers, and they do different jobs. Blocking one does not block the others.
  • A blocking rule can sit in two places: your robots.txt file, or a firewall in front of the site.
  • AI crawlers read the page's HTML. Content that only appears after scripts run may be invisible to them.
  • Being seen is not being recommended. Access is the first step, not the last.

Which crawlers does ChatGPT use?

OpenAI's crawler documentation describes three, each with its own name in a site's rules.

Crawler What it does If you block it
OAI-SearchBot Finds pages so they can appear in ChatGPT's search answers Your pages are unlikely to be shown or linked in those answers
ChatGPT-User Opens a page when a person asks ChatGPT about it ChatGPT may be unable to open your page for that person
GPTBot Collects pages that may be used to train future models Your content is kept out of training. Search answers are a separate matter

The practical point is in the last row. A business can keep its content out of model training and still appear in ChatGPT's search answers, because the two are controlled by different names. Many sites block all three without meaning to.

What stops ChatGPT from seeing a website?

1. A rule in robots.txt

Every site can publish a small text file, robots.txt, that tells crawlers what they may read. Type your web address followed by /robots.txt into a browser to see yours. Look for lines that name GPTBot, OAI-SearchBot or ChatGPT-User followed by "Disallow: /", or a blanket rule that applies to every crawler. Some website builders and security plugins add these lines without asking.

2. A firewall or security service

The block is not always in a file you can read. Security services that sit in front of a website can turn AI crawlers away before they reach it. Cloudflare, one of the largest, began asking every new domain whether to allow AI crawlers in July 2025, and in 2026 it split its controls into three groups: search, agent and training. So a site can have a clean robots.txt and still be closed to some AI crawlers because of a setting chosen, or left at its default, when the site was set up. A robots.txt check, including ours, cannot see a firewall rule, so ask whoever manages your site to look at the bot settings.

3. Content that only loads with JavaScript

Many modern sites send a nearly empty page and then build the content in the browser with scripts. People never notice. Crawlers can. An analysis of AI crawler traffic by Vercel and MERJ, published in December 2024, found that none of the major AI crawlers it tracked ran JavaScript, OpenAI's three included. To test your own page, open it, choose View Page Source in your browser, and search for a sentence from the middle of the page. If the sentence is not in the source, a crawler that does not run scripts will not find it either.

4. Pages that are not public

Anything behind a login, a password or a "coming soon" screen is closed to crawlers. So is a page that nothing links to and that is missing from your sitemap, because crawlers find pages by following links.

5. Not being in a search index

OpenAI's announcement of ChatGPT search says it draws on third-party search providers as well as content from its partners. A site that the main search engines have not indexed has less chance of surfacing there. If your pages are missing from Google too, start with why a website does not show up on Google.

How do you test it in one minute?

  1. Run the checker. The free AI visibility checker reads your robots.txt and lists ten AI crawlers, from OpenAI, Anthropic, Perplexity, Google and Apple, as allowed or blocked. It also checks whether your content is in the page itself and whether the business is described in structured data.
  2. Ask ChatGPT directly. With search turned on, ask what your business does and which page it used. If it cites your site, it can read it. If it answers only from directories and review sites, or says it cannot open the page, that is a clue worth following up. One failed attempt is not proof, so try a second page.
  3. View the source. Use the test from point 3 above on your homepage and on one service page.

What about Claude, Perplexity and Gemini?

The pattern repeats. Anthropic's help center lists a search crawler, a crawler that fetches pages for a user, and a training crawler for Claude. Perplexity's crawler documentation lists a search crawler and a user-request one. Google works differently. Its AI answers are built on the same index as Google Search, and the name Google-Extended in a site's rules controls whether content may be used for Gemini, not whether the site appears in Search. Our guide to AI crawler readiness lists each name and what it controls.

Should you block AI training crawlers?

That is a business decision, and there are reasonable arguments both ways. Publishers who sell their writing often block training. A law firm or a local business usually wants to be known and described accurately, and gains little from staying out. What matters is that the choice is deliberate. Blocking the training crawlers does not, by itself, remove you from AI search answers. Blocking the search and user-request crawlers can.

Is being seen the same as being recommended?

No. Access only means the assistant is able to read what you say about yourself. Whether it names you depends on how clearly your pages answer the question, what other sources say about you, and how you compare with the alternatives. That is covered in how to get your business recommended by AI. Access comes first, though, because nothing else counts until the crawlers can get in.

Frequently asked questions

Not by that alone. GPTBot gathers content that may be used for training. ChatGPT's search answers rely on a different crawler, OAI-SearchBot, and pages opened for a user come through ChatGPT-User. A site can block the first and allow the other two.

OpenAI does not publish a timeline, so nobody can promise one. The crawlers have to revisit the site first. Keep your sitemap current and check again over the following weeks.

No. An llms.txt file is a guide that points AI tools to your most useful pages. It does not grant or deny access. Access is decided by robots.txt and by any firewall in front of the site.

It can read the public pages its crawlers are allowed to reach and that have their content in the page's HTML. It cannot read pages behind a login, and it may miss content that only appears after scripts run.

Yes. The checker works on any public website, so you can enter a competitor's address and compare the result with your own.

DC

Div Churiwal

Founder of Nexus Multimedia, which builds search and AI visibility programs for law firms and businesses across the United States. Reach the team at info@nexusmultimedia.ai or (619) 736-0704.

Find out which AI crawlers your site lets in.

The free AI visibility checker reads your site's rules and shows ten AI crawlers as allowed or blocked.

Check my site Schedule Strategy Call