Before You Block AI Crawlers, Decide What You Want to Control

Start with the outcome, not a list of bots

Ask the person requesting the change to finish this sentence:

We want to restrict [specific use] of [specific content], while preserving [specific discovery or visitor experience].

For example, “We want our public service guides to remain discoverable in Google Search while reviewing their use in Gemini products” gives a developer something to investigate. “Stop AI taking our content” leaves the affected products and acceptable trade-offs undefined.

Record the page types involved. A public guide, a customer account and an unpublished proposal should not inherit a policy just because they live on the same domain.

What does robots.txt actually control?

Google describes robots.txt as a way to manage which URLs crawlers can access. It is not a reliable way to remove an HTML page from search results: a blocked URL can still appear if Google learns about it elsewhere. Nor is the file access protection for confidential material. Google’s robots.txt introduction

If the objective is privacy, ask the website owner to verify authentication and access permissions. If the objective is search exclusion, review the appropriate indexing control and how the crawler will encounter it. Do not assume a crawl block proves either outcome.

That distinction is useful before any discussion of AI. It prevents a publicly accessible document from being labelled “protected” because a robots rule was added.

Googlebot and Google-Extended serve different purposes

Google’s crawler documentation says Googlebot preferences affect Google Search, including its Search features. Google-Extended is a separate product token governing specified uses of crawled content for Gemini model training and grounding. It has no separate HTTP user-agent string, and Google says it does not affect inclusion or rankings in Google Search. Google’s crawler and product-control documentation

The practical consequence is that an instruction about Gemini use needs a product-specific review. Blocking Googlebot is not an interchangeable implementation.

Keep the official documentation link in the change request. A reviewer should be able to see the stated scope rather than relying on a bot name copied from an old spreadsheet.

What about AI Overviews and AI Mode?

Google’s Search guidance says a page must be indexed and eligible for a search snippet to qualify as a supporting link in AI Overviews or AI Mode. Meeting those conditions does not guarantee selection. Google’s AI features guidance

For a business seeking search visibility, this makes eligibility part of the review. Before approving a restriction, ask whether it changes the conditions under which the intended page can appear.

Keep that question separate from whether the page offers a useful answer. Access controls cannot substitute for clear content, and a useful page still needs to be accessible through the intended discovery route. My guide to content audits and citation evidence explains the editorial side of that work.

If a page is missing from Search, use the new-page indexing checklist to investigate discovery and access before changing your crawler policy.

Write a change brief a developer can verify

Use a short decision record before editing configuration:

This is a suggested working record, not a provider requirement. Its purpose is to make disagreements visible before a rule reaches production.

An illustrative example: public guides and private proposals

Imagine a consultancy with public buying guides and private customer proposals. The owner asks for one site-wide AI block.

I would split the request into two decisions. The proposals need an access review: who can open them, and what happens without an authorised account? The public guides need a product-use decision: which uses should be restricted and which discovery routes matter commercially?

Only after those decisions should the developer propose configuration changes. The same owner may reasonably choose different policies for the two content groups.

This is a fictional scenario. It illustrates a review process, not a claim about a deployed configuration or a traffic result.

Test the configuration without overstating what it proves

Save the previous settings and inspect the deployed result after the change. Review representative URLs from every affected page group, including pages that should remain accessible. Record the exact tests, dates and outcomes.

A rule appearing in the live file proves that the file changed. It does not by itself establish what every service has done with content. Keep the conclusion as narrow as the evidence.

If monitoring later shows a visibility change, first check whether the intended configuration was deployed. Avoid treating a movement in impressions as proof that a particular AI-use policy caused it.

Leave the team with a clear decision

A useful handoff says what was restricted, which content and products are covered, what remains allowed, and what the tests established. It also names any unanswered questions.

The best first step is often a better brief. Before copying a blocklist, agree on the outcome and the discovery routes worth preserving.

If you need help reviewing those trade-offs, explore my technical SEO consulting work. Bring the current settings and the business objective so the review starts with a concrete question.