FREE TOOL / AI SEARCH ACCESS

Is your site blocking
AI search bots?

Your robots.txt says one thing. Your CDN or firewall can do another. Enter a domain to see how ten AI bots are treated, on paper and in a live request, and where the two disagree.

Check a site

Takes about 10 seconds. Ten checks per hour per network.

How to read the result

Allowed

robots.txt does not disallow the bot and a request identifying as that bot got a normal page.

Blocked by robots.txt

A rule in robots.txt disallows the bot for the path you tested. That is a policy choice, so check it was deliberate.

Blocked by server

robots.txt disallows the bot and the server also refuses it. Both layers agree.

Mismatch

The one to look at first. robots.txt allows the bot, but a request identifying as that bot is blocked or challenged. In practice that is usually a bot-management, firewall or CDN rule that nobody tied to the robots.txt decision.

Inconclusive

The site refused or failed our ordinary browser request, or the test timed out, so bot results cannot be compared with anything.

What this test cannot show. The live check sends each bot's user-agent string from our server. It does not come from the operator's published IP ranges, so rules that verify bots by IP address can treat this request differently from the real crawler. Read a result as "requests identifying as this bot were blocked", not as proof of what the real crawler experiences.

Bots checked

OpenAI and Perplexity user-agent strings come from their published documentation. Anthropic publishes robots.txt tokens but not full user-agent strings, so the three Anthropic strings are marked approximate in the results. List version 2026-11-20.

Questions

What does the AI Bot Access Check test?

It reads the robots.txt file for the domain, works out what it says for each bot, then makes a separate request identifying as that bot and records whether the server answered, blocked or challenged it. A mismatch is a bot where those two results disagree.

Why can robots.txt allow a bot while the site still blocks it?

robots.txt is a request to well-behaved crawlers. Blocking happens elsewhere: a CDN bot-management setting, a firewall rule, a hosting default or a security plugin. Those are often changed by a different team from the one that maintains robots.txt, so the two drift apart.

If the result says allowed, can ChatGPT, Claude and Perplexity crawl my site?

Not necessarily. This is a user-agent test from one server. A rule that checks the crawler's IP address would treat the real bot differently. Treat a clean result as "nothing obvious is blocking these user-agents", and use your server logs to confirm real crawler visits.

Does the tool store the domain I check?

The domain and result are kept in the server's memory for one hour so repeat checks are fast. The server also logs the domain, the counts of blocked and mismatched bots and a hashed version of your IP address, which cannot be turned back into the address. Page analytics, if you allow them, never include the domain. If you request the emailed report, the domain and a one-line summary are sent with your email address.

Can I check a site I do not own?

Yes, the tool only requests the pages any visitor can request, and identifies itself in a request header. It will not check private network addresses, and it is rate limited.

NEXT STEP

Access is one question. Content is the other.

A page that bots can reach still has to answer what people ask AI tools. The free content audit planner walks through that review one page at a time, in Excel, without an email.

Get the content audit planner

Related reading: choosing robots.txt controls for AI search.