Skip to content
Relevant.aiRelevant.ai
Playbooks

Blocking AI crawlers is not a security decision. It is a sampling decision.

September 10, 2026 · 8 min read

TL;DR
  • Blocking AI crawlers does not necessarily make your brand invisible to AI. It changes the information AI systems can use to understand you.
  • If first-party content is unavailable, AI systems may still form an impression from competitors, resellers, reviews, Reddit, media and comparison pages.
  • That creates three outcomes: well sampled, not sampled, or badly sampled. The third can be the most damaging because the system has enough information to be confident, but not enough to be accurate.
  • Training bots, AI search bots and user-triggered agents have different business purposes. Do not govern all three with one blanket rule.

Most discussions about AI crawlers begin with a technical question: which layer actually blocks the bot? robots.txt relies on the bot behaving. A WAF, a CDN rule or a server block enforces it. So the advice lands in the same place every time: robots.txt is a request, the edge is a wall, use the wall if you mean it.

That answer is correct and it is also answering the wrong question.

This is not an access question. It is a sampling question. And once you see it that way, the whole decision changes shape.

A model does not store your site

A language model does not keep a copy of your website. It keeps a compressed estimate of your website, built from whichever pages it happened to fetch.

That single fact means there are not two possible outcomes here. There are three.

  • Fully sampled:Enough of your content is available for the system to form a reasonably representative picture of your company.
  • Not sampled:Your first-party content is largely unavailable.
  • Badly sampled: Only fragments of your site - or old, incomplete or unrepresentative information - are available, creating a biased picture.
A model that knows nothing about you is one problem. A model that is confidently wrong about you can be a much harder one.

Figure 01. Blocking decides which of three sampling outcomes you land in.

Every article on this topic argues about how to get from outcome one to outcome two. The interesting problem is that most blocking attempts land in outcome three, and outcome three is worse than either of the others. A model that is confidently wrong about you is a harder problem to solve than a model that has never heard of you.

And here is the part that makes it genuinely uncomfortable. Outcome two does not mean invisible.

If you remove your own data, AI may fill in the gaps

When a household does not respond to the US Census, the Bureau does not leave a blank. It uses a technique called hot deck imputation. It looks at the size and type of neighbouring households in the same census tract and fills you in from them.

Figure 02. Non-response does not create a gap in the record. It creates an estimate.

In the 2010 census, about 1.16 million people were added to the count this way. Roughly 0.39 percent of the household population existed in the official record as an estimate made from the people next door.

Refuse the census and you do not disappear. You become an estimate assembled from the people next door.

Language models do the same thing, only far more aggressively, because generalising from neighbours is the entire mechanism rather than a footnote in the methodology. If your own site is unavailable, questions about your company or category do not disappear. Users may still ask which vendors to consider, how one company compares with another, whether a product is worth buying, or what alternatives exist.

The system can still construct an answer from other available evidence, such as:

  • competitors
  • resellers
  • review websites
  • old Reddit discussions
  • media coverage
  • directories
  • comparison pages
  • other third-party sources

So the choice was never present or absent. It is this: described by your own content, or imputed from your neighbours.We call this imputed visibility, and we have not seen anyone price it.

Now the layer question answers itself - robots.txt and a WAF solve different problems

Once you see this as a sampling decision, robots.txt versus WAF stops being a debate about enforcement strength. The layer simply decides which of the three outcomes you land in.

Figure 03. Two layers, two different questions. Most people only ask one of them.
  • robots.txt expresses intent
    • robots.txt is a declaration. It has no enforcement. What it does have is standing. It is the artifact that gets pointed at in a licensing conversation, a compliance review, or a negotiation. It is how you say what you intend. It is not how you make anything happen.
  • A WAF or CDN rule enforces access
    • WAF, CDN and server rules are enforcement. They actually stop the fetch. They say nothing about intent, and a bot operator reading only your robots.txt will never know you objected.

Put together: robots.txt alone gets you a leaky, biased sample plus a paper trail. Enforcement alone gets you a clean block with no declared intent and therefore no negotiating position. These are not competing answers to one question. They answer two different questions.

The mistake underneath all of this

AI crawler is not one thing, and this is where the real damage happens.

There are three distinct kinds of fetch, and OpenAI documents them as three separate agents that operate independently of each other. GPTBot fetches for model training. OAI-SearchBot fetches to power search inside ChatGPT, and opting out means you are not shown in ChatGPT search answers. ChatGPT-User fetches because a person asked it to go and look at your page right now.

Figure 04. Three fetches with three different business meanings, usually governed by one rule.
Training is about future model knowledge
  • If you block a training crawler, you are making a decision about whether your content can contribute to future model training.
Search is about discoverability
  • Search crawlers help AI search systems retrieve and surface current information. Blocking search can therefore affect whether your pages can participate in AI-generated search experiences.
User-triggered agents are different again
  • A user may explicitly ask an AI system to visit or work with a page. That traffic represents immediate user intent.
Do not turn "we do not want our content used for training" into "block every AI user agent." You may solve one problem while creating two more.

A blanket block can create an invisible outage

Imagine your legal or content team decides that your content should not be used to train AI. Engineering then creates one broad rule covering anything that looks like an AI crawler.

  • Intended outcome:training blocked.
  • Possible unintended outcomes:AI search visibility reduced and user-requested access blocked.

Nothing necessarily breaks on your website. There may be no conventional SEO outage, and analytics may not make the impact immediately obvious. But your AI distribution layer has changed.

That is why broad AI blocking should not be treated as a routine infrastructure rule. It is a product, marketing, legal and distribution decision.

The part that is not reversible

There is one more asymmetry worth sitting with, and it is the reason this decision deserves more care than its SEO equivalent.

Figure 05. Same action, wildly different recovery time.

There is no documented way to backfill. A training corpus is a snapshot. If you were blocked while that snapshot was taken, unblocking later does not retroactively insert you. You wait for the next one.

Unblock Googlebot and you are recrawled and back in the index within days. Unblock GPTBot and you are waiting on a model generation. That is a very different risk profile, and it means a block made casually in a sprint can outlive the person who made it.

What should companies actually do?

There is no universal answer. But there are clearer ways to make the decision.

If protecting the content is the priority
  • Declare the policy through robots.txt.
  • Enforce the restriction at the edge.
  • If you genuinely want something unavailable, use enforcement - not just a request.
If AI visibility is the priority
  • Think separately about training, AI search and user-triggered agents. Do not automatically block all three because you object to one.
If infrastructure cost is the problem
  • Rate-limit or manage traffic where appropriate rather than treating every cost problem as an access problem.
Avoid the worst middle ground
  • One of the riskiest outcomes is an inconsistent block where some pages are accessible and others are not. You may give AI systems a biased sample of your brand: enough information to form an opinion, but not enough to form the right one.

And whichever corner you pick, split the rule by agent. Training, search and agent traffic are three decisions, not one.

The question to answer before touching robots.txt

What share of my site has actually been sampled, by which agent, and what does the model currently believe about me as a result?

Almost nobody can answer that, which means almost every block is being made blind. You cannot reason about a sampling decision without knowing your denominator. So, before deciding whether to increase or decrease AI access, understand:

  • which crawlers reach your site
  • which sections they can access
  • which pages matter most
  • what information is being excluded
  • which third-party sources describe your brand
  • how AI systems currently answer important questions about you

That is the whole argument. Once you look at the problem this way, crawler management stops being a binary allow/block conversation. It becomes a question of information quality and representation.

About Us

Relevant.aiis the first scientific measurement platform for AI search. We treat AI visibility as a statistical measurement problem, with denominators, variance and confidence intervals rather than vanity scores.

Sources