The quiet reason some sites never show up
Sometimes a business is missing from AI answers for a very boring reason: the crawlers were told to stay out. Usually nobody decided that on purpose. A theme shipped with a strict robots.txt file, a security plugin added a rule, or someone flipped a switch in a CDN control panel two years ago and forgot about it.
This is the plumbing check in the Superintelligent Optimization program. It is cheap to fix, and nothing else in the program works well until it is right. We run it as part of every free AI visibility audit.
Robots.txt rules that close the door
Blanket disallows and leftover staging rules
The robots.txt file at the root of your domain tells well-behaved crawlers what they may fetch. A single line left over from a staging site can tell every crawler to skip everything. We read the live file, not the one your developer remembers writing.
Rules that name AI crawlers directly
Some templates and plugins now block AI user agents by name. That can be a reasonable choice. There is a real difference between a crawler that collects content for model training and one that fetches pages to answer a live question, and some owners allow one while refusing the other. The point is that it should be your decision, made on purpose, with the tradeoff explained.
CDN and firewall settings owners don't know are on
One-click "block AI bots" switches
Several CDN and hosting providers, Cloudflare among them, offer a setting that blocks known AI crawlers across the whole site. It overrides whatever your robots.txt says. We find owners who have never heard of the setting and still have it switched on.
Bot challenges, rate limits and firewall rules
Even without that switch, a strict firewall rule, a bot challenge page or an aggressive rate limit can turn crawlers away. Your robots.txt says welcome and the firewall says no. The crawler only hears the no.
How we test from the outside
We request your key pages the way each crawler identifies itself and record what comes back: the page, a challenge screen, an error, or a redirect loop. A user-agent test is not identical to a visit from a crawler's real network, so where you can share server or CDN logs, we check those too.
The crawlers worth knowing by name
These are the user agents we check most often. Descriptions follow how each company presents its crawler publicly, kept deliberately conservative, and we recheck the published documentation during each audit because names and purposes change.
| Crawler | Company | What it is generally for |
|---|---|---|
| GPTBot | OpenAI | Collecting public web content that may be used to improve OpenAI's models |
| OAI-SearchBot | OpenAI | Finding pages to surface in ChatGPT's search features |
| ClaudeBot | Anthropic | Collecting public web content that may be used in developing Anthropic's models |
| PerplexityBot | Perplexity | Indexing pages so they can appear in Perplexity's answers |
| Bingbot | Microsoft | Crawling for the Bing index, which Microsoft Copilot is built on |
Most of these companies also run separate fetchers that visit a page only when a user asks about it. We check those where they are documented.
llms.txt: worth having, not worth overselling
An llms.txt file is a plain-text summary placed at the root of a site, pointing AI tools to the pages that matter most. It is an emerging convention proposed by practitioners, not an official standard, and support across AI tools is uneven and mostly undocumented.
We add one because it is cheap, harmless and easy to keep accurate. We will not tell you it moves the needle by itself, because nobody can show that it does. The pages it points to still have to state their facts clearly, which is what writing content an AI can quote handles.
Getting into the indexes AI answers draw on
Being crawlable is step one. Being indexed by the search engines that AI tools rely on is step two.
Bing Webmaster Tools
ChatGPT search has drawn on Bing's index among other sources, and Copilot is built on Bing. We verify your site in Bing Webmaster Tools, submit a clean sitemap, and review crawl errors and indexed pages.
IndexNow
IndexNow is a protocol that lets a site notify participating search engines, Bing included, when a page is added or changed. It shortens the wait between publishing an update and a search engine noticing it.
Brave Search submission
Claude's web search is reported to use Brave Search, and Brave's own AI answers sit on its index. We check whether your important pages appear there and submit the ones that are missing. Submission is a request, not an instruction; Brave decides what it indexes.
Access is a prerequisite, not a promotion
Opening the door does not earn a mention. It makes sure the rest of the work can be seen: the entity and schema fixes that identify your business, the answer pages, and the outside sources covered under citations and review signals. Whether the answers change afterward shows up in your monthly visibility reports, and the full sequence is laid out on how the engagement works.
Not sure whether your site is blocking AI crawlers right now? Book a free AI visibility audit and we will check.
Common questions
Should I let AI companies train on my site?
That is your call. Some owners allow search-focused crawlers but block training crawlers. We explain what each crawler is described as doing and set the rules you choose, rather than a default someone else picked.
Can a CDN setting really override my robots.txt?
Yes. A robots.txt file is a request that crawlers read. A CDN or firewall block stops the request before it reaches your site, so the crawler never sees your robots.txt at all.
Does adding llms.txt make AI tools recommend me?
No one can show that it does. It is an emerging convention that some tools may read. We add it because it is low effort and low risk, and we are honest that it is a small piece.
How do I know if my site is in Bing's and Brave's indexes?
We check your key pages in both during the audit and report which ones are missing. Once the site is verified in Bing Webmaster Tools, you can also see Bing's indexed pages yourself.