How do I check if my website is accidentally blocking ChatGPT's crawler?
TL;DR: Go to yoursite.com/robots.txt and look for "Disallow: /" under User-agent: GPTBot, ChatGPT-User, or OAI-SearchBot. If you see it — or if you use Cloudflare, a security plugin, or a "block AI bots" toggle — you may be invisible to ChatGPT even with a perfect website, because the crawler is never allowed to read it.
The claim
A growing number of small-business sites block AI crawlers without anyone deciding to. It happens three ways: a security plugin ships with AI-bot blocking turned on by default, a CDN or web application firewall (like Cloudflare) added a one-click "Block AI Bots" toggle that some hosts pre-enabled, or a developer copied a robots.txt template from another site that blocked GPTBot for content-scraping reasons never intended for the business's own visibility goals.
Evidence
OpenAI, Anthropic, and Perplexity each publish the exact crawler user-agent strings their systems use, and each respects the robots.txt standard. That means a single Disallow line under the wrong user-agent silently removes a business from that engine's ability to read the site at all — no warning, no notification, just absence from any answer that would have cited the page.
The crawlers to check for
| User-agent | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Crawls pages to train and ground models |
| ChatGPT-User | OpenAI | Fetches a specific page live when a user asks ChatGPT to look at it |
| OAI-SearchBot | OpenAI | Powers ChatGPT's search/citation feature |
| ClaudeBot | Anthropic | Crawls and grounds Claude's answers |
| PerplexityBot | Perplexity | Crawls for Perplexity's indexed answers |
| Google-Extended | Controls use in Gemini and AI Overviews (separate from standard Googlebot) |
Step-by-step: check and fix it
- Open a browser and go to yoursite.com/robots.txt. If the page doesn't exist, you're not blocking anyone at the robots.txt level — move to step 4.
- Scan for any User-agent line matching the table above followed by Disallow: /. That line blocks the entire site from that crawler.
- If your site is on WordPress, check SEO and security plugins (Yoast, Rank Math, Wordfence, iThemes) for an "AI crawler" or "bot blocking" setting — many ship this off, some don't.
- If you're behind Cloudflare, log in and check Security > Bots for a "Block AI Bots" or "Block AI Scrapers" toggle — some plans enabled this by default in 2024–2025.
- Fix any blocking line by changing Disallow: / to Allow: / for the crawlers you want visible, or removing the block rule entirely, then re-check the live robots.txt file to confirm the change published.
FAQ
What's the difference between GPTBot and ChatGPT-User? GPTBot crawls broadly to help train and ground ChatGPT's knowledge. ChatGPT-User only fetches a specific page in real time when a live chat asks ChatGPT to open that exact link.
Will blocking hurt me even if I never edited robots.txt myself? Yes — Cloudflare's bot-management settings and several WordPress security plugins can block AI crawlers by default without any manual robots.txt edit.
How do I unblock the crawlers? Change any Disallow: / line under the relevant user-agent to Allow: /, and separately check for CDN- or plugin-level AI bot blocking, since robots.txt isn't the only place a crawler can be stopped.
Does allowing GPTBot mean OpenAI can do whatever it wants with my content? No — robots.txt only controls crawler access, not usage rights. But blocking the crawler guarantees your business can never be cited by that engine's answers, which is the opposite of the goal here.
How often should I re-check this? Check any time you change hosts, install a new plugin, add a CDN, or switch security tools — those are the moments blocking rules get added silently.
By Pinal Dave Last updated: July 25, 2026