AIVX Labs
AI news for agencies · Getting found in AI answers

The new AI-training robots.txt line: opt-out advice for clients

By Alan Magliocca · Published November 2026

Quick answer: On October 30, 2026, Google said it plans to support a shared robots.txt AI-training control from the IETF AIPREF work, aiming for 2027, and Microsoft and OpenAI announced plans to honor it too; none of them supports it yet. For most local clients that want to be found, don't add a blanket opt-out, because Google treats it like blocking Google-Extended, which also removes Gemini app grounding; use path-level rules only for content the client truly wants to protect.

Key takeaways

  • The shared signal is a robots.txt group under the user-agent token __usage_ai-training, defined in an individual IETF draft from October 30, 2026.
  • Google aims to support it in 2027, Microsoft says it starts honoring its new options on February 1, 2027, and OpenAI aims to honor it in the coming months.
  • If a site opts out with the shared token and doesn't name Google-Extended, Google plans to treat it as if Google-Extended were disallowed.
  • Google says sites that allow Google-Extended are eligible to appear in Gemini app responses through grounding, so a blanket opt-out has a visibility cost.
  • A wildcard user-agent group does not match the token, so an opt-out has to name it explicitly.

What happened

On October 30, 2026, John Mueller of Google published a shared AI training control on the Search Central blog. Google says it plans to support a shared "policy token" in robots.txt, based on work from the IETF's AIPREF working group, so a site can state its AI-training preference to several AI providers in one step.

The key points from Google's post:

  • Google-Extended mapping. If a site uses the shared token to opt out of training and doesn't name Google-Extended, Google plans to treat that "as if the publisher had disallowed Google-Extended."
  • Existing controls stay. Google says existing company-specific controls keep working alongside the new signal.
  • Gemini grounding. Sites that explicitly allow Google-Extended are "eligible to appear in responses" and receive clicks through grounding in Gemini Apps.
  • Timing. Google says it aims "to support this new control along with other AI providers in 2027." It calls the work a draft proposal.

The syntax lives in an individual Internet-Draft, draft-kelley-aipref-training-preference-signal-00, written by Kevin Kelley of Anthropic and dated October 30, 2026. It has no formal IETF standing yet and expires May 3, 2027. A site writes a normal robots.txt group addressed to the token. The draft's own example is three lines:

  • User-Agent: __usage_ai-training
  • Disallow: /
  • Allow: /press-kit/

That says "don't train on anything except the press kit." Disallow means a preference against training on that path, Allow means a preference in favor, and a robots.txt with no group naming the token expresses no preference. The token is matched without regard to case.

Search Engine Journal reported that Microsoft and OpenAI announced plans the same day. Microsoft will start honoring its new options on February 1, 2027 and is adding page-level controls and a separate grounding preference. OpenAI aims to honor the signal in the next few months, and its current robots.txt rules stay in effect. SEJ also notes that a wildcard group (User-Agent: *) never matches the token, and that the token doesn't control crawling.

What it means for agency owners

Clients will hear "you can now block AI with one line" and ask whether they should. For most small and local businesses, the answer is no.

A local business wants AI assistants to know it exists, what it does and where it is. Google's post ties the shared opt-out to Google-Extended, and Google says allowing Google-Extended is what makes a site eligible for grounding in Gemini Apps. SEJ points out that Google's approach gives no described way to keep content out of Gemini training while still allowing Gemini app grounding. So a blanket opt-out could trade away a visibility channel the client is paying you to build.

According to SEJ's summary of Google's crawler documentation, Google-Extended does not affect inclusion in Google Search. The opt-out is about Gemini training and grounding, not normal rankings.

There are clients for whom an opt-out makes sense: a photographer whose portfolio is the product, a course creator with paid lessons on public URLs, or a publisher whose articles are the business. For them, path-level rules let you protect the valuable content while leaving service pages, contact details and FAQs open.

Nothing changes on the day you publish the line. No provider supports it yet, so there is no rush to edit a client's robots.txt before the end of 2026.

What to do this week

  1. Inventory robots.txt. For each client, record what the file says now about Google-Extended, GPTBot and other AI crawlers. Many sites have rules a previous developer added and nobody remembers.
  2. Ask one question. Does the client want AI assistants to describe and recommend the business? Write the answer in the client's file.
  3. Sort clients into three groups: visibility-first, protect-some-content and protect-everything. Use the decision guide below.
  4. Don't publish the shared line yet unless the client is in the protect group and understands the Gemini trade-off. Support starts in 2027.
  5. Set a review date. Put January 15, 2027 on your calendar to check provider documentation before Microsoft's February 1, 2027 start.

Three client types, three robots.txt choices

Visibility-first (most local businesses). Add no group for the token. Silence expresses no preference, which keeps things as they are. Review any old rules that block AI crawlers by accident.

Protect some content (creators, course sellers, photographers). Address the token and disallow only the paths that hold the paid or original work, for example:

  • User-Agent: __usage_ai-training
  • Disallow: /members/
  • Disallow: /portfolio/

Service pages, the about page and contact details stay open. Remember that under Google's plan this opt-out is read as disallowing Google-Extended for those paths.

Protect everything (publishers, some professional firms). Disallow the whole site for the token, with Allow lines for a press kit or about page if the client wants those used. Explain in writing that this may reduce how the business appears in Gemini app answers.

What the line does not do

Set expectations so no client thinks one line is a force field:

  • It's a preference, not a lock. It relies on providers honoring it.
  • It doesn't stop crawling. SEJ notes the token doesn't control crawling.
  • It isn't live. Google aims for 2027, and OpenAI and Microsoft have their own timelines.
  • It doesn't remove content already used. Nothing in the announcements says so.
  • Rules are still a draft. The IETF document can change before it expires in May 2027.

This is general guidance, not legal advice. If a client is asking because of a copyright or contract concern, suggest they check with a lawyer.

How AIVX Labs helps

AIVX Labs doesn't edit robots.txt files. Its AI visibility check shows whether AI assistants recommend a client, which is the number at stake when a client asks about opting out; see running an AI-answer visibility check. Record the client's decision in their workspace, and if the robots.txt review turns up other site problems, the Website Analyzer lists fixes in order; RICO can run it with the credit cost shown first. For how the same choice plays out for Google's AI answers, see Google's AI answers now show more links.

BrightStage's guide to llms.txt optimization covers a related file some sites use to guide AI systems.

Sources

Written by Alan Magliocca, founder of AIVX Labs and BrightStage AI (Podavinci LLC). Written with AI assistance and checked against the linked sources.

Read our featured article on LinkedIn

Get AIVX Labs verified for $1: RICO, your AI operator, with 7 days of full access.

Get AIVX Labs Verified for $1

Follow along:LinkedInYouTubeTikTokXFacebookInstagramSkool community

Frequently asked questions

Will opting out of AI training hurt Google Search rankings?

According to Search Engine Journal's summary of Google's documentation, Google-Extended doesn't affect inclusion in Google Search. The trade-off Google describes is in Gemini app grounding.

Does User-Agent: * cover the new token?

No. Search Engine Journal reports that a wildcard group never matches the token, so an opt-out must name it directly.

Is Anthropic part of this?

The draft was written by Kevin Kelley of Anthropic. Search Engine Journal reports that Anthropic had not published its own announcement as of October 30, 2026.