Should your site opt out of AI training?

Google says it will honor one shared line in robots.txt that tells AI companies not to train on your site, and that for Google, that line will also count as a block on Google-Extended. Nothing changes on your site today, because Google aims to support it next year. This week, open your robots.txt, write down what it already says about AI, and settle the question with your web person before anyone adds a blanket opt-out.

John Mueller, Google Search Advocate, announced it on October 30 on the Google Search Central Blog: “When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended.” The idea comes from the AIPREF working group at the Internet Engineering Task Force, and Google calls it a “draft proposal.”

What opting out of AI training changes

Under Google’s plan, the shared opt-out switches off Google-Extended for your site, and Google-Extended covers more than training.

Google’s crawler documentation describes Google-Extended as the token that decides whether content from your site may be used for training future generations of Gemini models, and for grounding in Gemini Apps and Vertex AI. Grounding is the part an owner feels. Google defines it as “providing content from the Google Search index to the model at prompt time to improve factuality and relevancy.”

So a site that blocks Google-Extended, directly or through the new shared line, steps out of two things:

• Training of future Gemini models.
• Grounding in Gemini Apps. Google’s announcement says sites that allow Google-Extended “are eligible to appear in responses and receive clicks via grounding in Gemini Apps.”

What it does not change

Your Google Search rankings sit outside this switch. Google’s documentation says it plainly: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”

It also does not keep Google’s crawlers away. Google-Extended has no user agent of its own; Google says crawling “is done with existing Google user agent strings” and the token is used “in a control capacity.” If the real worry is bot traffic slowing the site, this line will not fix it.

Company-specific rules keep working too. In Google’s words: “Existing company-specific controls will continue to function alongside this new joint signal.” A site can opt out broadly and still name Google-Extended to let Google in, or the other way round.

Why most local businesses want to be named

A plumber, a dental office or a coffee shop does not sell its words. It sells the visit, the call and the booked job, and its pages exist so people, and the assistants they ask, can find it.

The pressure for this control came from elsewhere. Google says the proposal “reflects feedback from publishers, civil society groups, regulators and other stakeholders.” A newspaper whose articles are the product has a real reason to opt out. A roofer whose site lists services, towns and a contact form usually has the opposite interest: when someone asks Gemini who repairs hail damage nearby, a page Gemini can ground on is a page that can be shown and clicked.

The honest exception is a business whose writing is what it sells, such as paid courses or original recipes; the shared line can suit it.

Other assistants matter as well. Google says other AI providers “can handle it similarly,” but each company decides what its own reading covers, so check its documentation before assuming. Our test of where answer engines get local recommendations shows how much those answers lean on records a business controls, and Google’s May guidance, covered in Google says optimizing for AI is still SEO, still holds: being found by AI starts with being crawlable.

	Leave robots.txt as it is	Add the shared opt-out next year	Opt out broadly, but allow Google-Extended
Training of future Gemini models	Allowed	Opted out	Allowed
Grounding and clicks in Gemini Apps	Eligible	Out	Eligible
Google Search rankings	Unchanged	Unchanged	Unchanged
Other providers that support the line	Whatever your file says today	Opted out, as each one reads it	Opted out, as each one reads it
Usually fits	Most local service businesses	Sites whose writing is the product	Owners who want Google’s answers but not other trainers

The five-minute robots.txt check

Do this before anyone edits the file:

1. Open yourdomain.com/robots.txt in a browser. A “not found” page is fine; Search Console Help says having no file “means that Google can crawl all URLs on your site.”
2. *Look for “User-agent: ” followed by “Disallow: /”. That blocks every crawler the file does not name separately, Googlebot included, and it matters far more than any AI line.
3. Look for “User-agent: Google-Extended”. A “Disallow: /” under it already keeps the site out of Gemini training and grounding today.
4. List every other user-agent name. For any you do not recognize, read that company’s own crawler page before keeping or deleting the rule.
5. Open the robots.txt report in Search Console. Google’s help page says it shows when Google last fetched the file, and its Versions view lists changes from the last 30 days, which tells you whether a plugin or theme update rewrote it.

Then give whoever edits the site one written sentence: “We want AI assistants to be able to name us,” or the opposite.

Reading the file yourself or with us

	On your own	With Beyaoshy
Cost to begin	Nothing; robots.txt and Search Console are free to read	Nothing; the consult is free
Who reads the file	You or your web person	The same team that runs your search work, nothing subcontracted
Whose name the site and Search Console sit in	Yours	Yours, and we work with the access you grant
What you walk away with	Your notes on each line	A written plan for each AI rule, kept whether or not you hire us

See how we handle SEO for Google and Bing, or send your website through the form below and we will tell you what each AI rule in your robots.txt switches on or off.