One Line to Opt Out: Google, Microsoft, OpenAI and Anthropic Back a Shared AI Training Signal

For years, a website owner who wanted to keep content out of AI training had to maintain a growing list of crawler names, one for each company and each bot. That chore may finally get simpler. On October 30, 2026, OpenAI, Google, Microsoft and Anthropic said they will back one shared robots.txt instruction that tells every participating company the same thing: do not use this content to train generative AI models.

The idea is a single purpose-based token called __usage_ai-training. Instead of naming bots, a publisher names a purpose. The change matters most for small publishers and independent blogs, which rarely have staff to track every new crawler name. It does not solve every dispute between publishers and AI labs, and it does not reach back in time. Still, as a piece of plumbing for the web, it is the most coordinated step yet toward a common language for AI permissions.

What the shared token actually is

The token lives in robots.txt, the plain text file that has told crawlers which parts of a site they may visit since 1994. A publisher adds a group that names the shared token as the user agent, then lists what is off limits. Reports describe the basic form like this:

User-Agent: __usage_ai-training
Disallow: /

A site can also carve out exceptions. A press kit folder, for example, could stay open while the rest of the site is closed to training, using an Allow line under the same group. The token itself does not name a company or a bot. It names a use: training generative AI models. That distinction is the whole point. Crawler names change, split into several agents, or get replaced. A purpose-based signal survives those changes.

The proposal behind the token is an Internet-Draft titled draft-kelley-aipref-training-preference-signal-00, written by Kevin Kelley, a technical lead for standards at Anthropic, for the Internet Engineering Task Force (IETF) AI Preferences working group. Reports stress that it is an individual submission, not yet an official IETF standard, with an expiry date of May 3, 2027.

Why publishers wanted one line instead of many

Until now, opting out meant listing each company's controls separately: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot, Claude-User and Claude-SearchBot for Anthropic, plus Microsoft's own signals. Each new agent added another line to maintain, and a missed line could leave a door open. Large publishers with legal teams could manage that list. Independent blogs and local newsrooms mostly could not, and many simply never set any AI preference at all. The shared token collapses that list into one entry that every participating company reads the same way.

Who is honoring it, and when

The four companies are not moving at the same speed, and publishers should plan around the slowest date, not the announcement date. Anthropic has reportedly already started implementing the signal. Microsoft will reportedly begin honoring the new options as it crawls from February 1, 2027, and will stop recognizing its NoArchive and NoCache tags as training or grounding signals after March 1, 2027. Google aims to support the token in early 2027, according to its crawler documentation, and plans to read it as if the publisher had disallowed Google-Extended, the control that covers future Gemini model training and grounding in Gemini Apps and on Vertex AI. OpenAI aims to honor the signal within the next few months, while its current robots.txt policy stays in effect.

Meta Platforms reportedly expressed support but made no commitment on when or whether it will implement the signal, so publishers cannot rely on it yet. Smaller AI developers, open-weight model builders and overseas labs have made no stated commitment at all, so the token only works against companies that choose to read it.

The Google and Microsoft split on grounding

The two largest signatories reportedly read the same token differently. Google treats the shared token as a block on both training and Gemini grounding, the step where a model pulls live web content to build an answer. Microsoft limits the token to training and handles grounding separately, through a page-level ai-grounding preference that can also be set by meta tag or HTTP header, with Allow, Disallow and DisallowExtended settings.

That split changes the trade-off for publishers. On Google, one line removes a site from both training and Gemini's live answers. On Microsoft, a site can opt out of training while staying eligible for grounded answers that may send citations and clicks back. Publishers who care about both should check each company's documentation rather than assuming one line means the same thing everywhere.

What the token does not do

The scope is narrow by design, and several common assumptions about it are wrong. First, it covers only future training. It cannot undo content that has already been collected, and it says nothing about deleting existing training sets or paying publishers for material already used. The models deployed through 2025 and 2026 were trained before this signal existed, and a new Disallow line does not reach back to them.

Second, it does not block crawling itself, and it does not remove a site from search results. A crawler may still visit pages for indexing while respecting the training preference. Third, it does not cover non-generative uses such as classification or ranking models, and it does not generally cover retrieval at answer time, which Microsoft and Google handle through their separate grounding controls. One more trap: a wildcard group does not trigger the preference, so a generic block for all bots does not cover AI. Name the token explicitly.

Frequently Asked Questions

What is the __usage_ai-training token?

It is a shared robots.txt user agent token that tells participating AI companies not to use a site's content to train generative AI models. A publisher adds it as a group in robots.txt with Disallow lines marking what is off limits, and can add Allow lines for exceptions such as a press kit folder.

When will Google, Microsoft, OpenAI and Anthropic honor it?

Anthropic has reportedly already started implementing it. Microsoft will reportedly begin honoring it from February 1, 2027, Google aims to support it in early 2027, and OpenAI aims to honor it within the next few months.

Does the token delete content that was already used for training?

No. It covers only future training. It does not remove content from existing training sets, does not pay publishers for past use, does not block crawling itself, and does not remove a site from search results.