For most small businesses, AI search crawlers should stay open. Blocking AI crawlers doesn't change your Google rankings, but blocking the wrong ones can keep you out of the answers people now get from tools like ChatGPT.
Blocking crawlers that only collect content to train AI models is a fair choice. The risk is in the details: the settings sit in a few different places, and one wrong line can block far more than you meant to.
"Should I block AI bots?" comes up more and more, often after someone reads that AI companies copy websites without asking. It's a reasonable worry. The answer depends on which bots you mean, because they do different jobs and blocking each one has a different cost.
What is an AI crawler?
A crawler is a program that visits web pages and reads them. Google has used one for decades to build its search results. AI companies now run their own, and they do three different jobs:
- Training crawlers collect pages to train future AI models.
- AI search crawlers fetch pages so an AI tool can answer a question and link to sources. OpenAI's documentation describes a separate crawler for ChatGPT's search answers.
- User-triggered fetches happen when someone asks an AI tool to look at a particular page.
Each company names its crawlers differently, and the list keeps changing. Most of what you read about "blocking AI" is really about the first group.
Does blocking AI crawlers hurt your Google ranking?
Not on its own. Google's crawler documentation says its AI training control "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search".
Google's AI Overviews and AI Mode don't use a separate crawler either. Google's guide to AI features says there are no additional requirements to appear in them. They draw on the same crawling and index as normal search.
The real danger is an accident. A rule meant for one AI bot that ends up applying to every bot can take a whole site out of Google.
What you lose by blocking AI search crawlers
Some customers now ask ChatGPT or similar tools things like "who does websites for accountants in Brisbane" or "what should I check before hiring a mechanic". Those tools answer from pages their search crawlers can read.
OpenAI says sites that opt out of its search crawler won't be shown in ChatGPT's search answers, though they can still appear as plain links. For a business that wants to be found, that's a cost with no upside.
When blocking makes sense
Blocking training crawlers is worth considering if your content is the product:
- original guides, research or courses you charge for, or plan to
- photography or design work you don't want copied
- data, tools or pricing models you built yourself
For a typical trade or professional services site, with service pages, suburb pages and helpful articles, there's little to protect and little to gain either way.
There's also a practical reason some sites block bots: badly behaved crawlers that hammer a site and slow it down. Polite instructions won't stop those, because, as Cloudflare's documentation puts it, "robots.txt directives are voluntary. Crawlers can ignore them." Those need to be dealt with at the hosting level.
Your host may have already decided for you
In July 2025 Cloudflare started blocking AI crawlers by default for new sites. Other hosts and security plugins have their own settings too. If nobody has looked, your site may be blocking AI search, or letting through bots you'd rather keep out, without anyone choosing it.
How we set it up
On the websites we build and look after, we make this decision with you and then set it up properly:
- We check what's reaching your site now. We look at every place a block can live: your site's crawler rules, your hosting and your security settings, and whether they agree with each other.
- We keep the crawlers that bring customers. Google, Bing and AI search stay open, so you can appear in search results and in AI answers.
- We block training if you want it blocked, without touching search.
- We deal with the bots that ignore the rules at the hosting level, using Cloudflare's bot protection and AI Labyrinth, which sends rule-breaking crawlers into a maze of decoy pages that real visitors never see.
If you're not sure what your site lets in, that's what our free health check covers. Our SEO service looks after the technical side from there, or get in touch and we'll take a look.
Not sure what your website lets in?
Book a free website health check. We'll tell you in plain English which search and AI crawlers can reach your site today, and whether that's costing you enquiries.
This article is general information only and isn't legal, financial or technical advice for your situation. Questions? Talk to us.