Google, OpenAI, Microsoft back one AI training opt-out

Google, Microsoft, OpenAI and Anthropic say they will honour one robots.txt signal that lets sites refuse AI training, though rollout runs into 2027.

Google, Microsoft, OpenAI and Anthropic said on 30 October 2026 that they will support a single signal in a website's robots.txt file that blocks the use of its pages for training generative AI models, reported by SiliconANGLE, Engadget and The Next Web. The proposal is a draft submitted to the Internet Engineering Task Force (IETF), the body that maintains web standards, and it is not yet a formal standard. For publishers, including Indian news and fintech sites, it could replace a growing list of crawler-by-crawler rules with one line.

What the proposal does

Today a website owner who does not want AI training on their pages must find each AI company's crawler by name and write a separate rule for it. Companies that run several crawlers need several rules. The draft, written by Anthropic's standards lead Kevin Kelley according to Engadget and The Next Web, adds a single "policy token" called usageai-training. A site lists the paths it does not want used for training, and every company that honours the token should respect it.

Engadget's example blocks a whole site for training while leaving the press kit open. The token covers only generative AI training. SiliconANGLE notes it would not apply to models that perform "exclusively non-generative tasks", and it does not remove a site from Google Search. It also applies only to future collection: material already gathered is not affected.

Who starts when

The companies are not moving at the same speed, and this is where readers should be careful about the headline.

- Anthropic says it has started work, will keep its existing crawler controls alongside the new token, and adds that support will take time. - OpenAI says it will implement the signal "in the coming months". Its GPTBot controls continue to apply. - Microsoft will honour the preferences from 1 February 2027. They replace the NoArchive and NoCache tags it has used since 2023. - Google says it will follow in 2027. If a site uses the new token without mentioning Google-Extended, Google's existing control for Gemini training, Google will treat it as a block on Google-Extended. - Meta has said it supports the idea of a common signal, but has not said it will honour it, according to The Next Web.

Background: what robots.txt is

Robots.txt is a plain text file at the root of a website, like a notice on a shop door saying which rooms visitors may enter. It has no legal force; it works because well-behaved crawlers choose to obey it. The new token states a preference about use, not a technical lock.

Limits and open questions

The draft is an individual submission that the IETF has not endorsed, per The Next Web. Private deals between a site and an AI company can override the token. Microsoft says the preferences "do not establish licensing or payment arrangements", so AI firms are not made to pay.

Microsoft is also adding a separate setting, AI Grounding, for when an AI answer fetches a live page to respond to a user. It warns that blocking it will "significantly reduce" a page's visibility in AI answers, though Bing ranking is unaffected. None of the three articles quotes a publisher reacting, so how publishers will judge the draft is not yet known.

What it means for readers

For anyone who runs a website, such as a fintech blog or a news site, the change is small today and larger in 2027. Nothing is needed this week, since only Anthropic has begun work. A site owner who wants to opt out can later add one group to robots.txt, while keeping existing GPTBot and Google-Extended rules until each company confirms support. It does not settle copyright disputes or help content already collected.

What to watch next

- Whether the IETF takes up the draft and whether other AI developers, including Meta, commit to it. - OpenAI's promised rollout "in the coming months". - Microsoft's 1 February 2027 start date and Google's 2027 timeline. - Whether publishers say the token is enough, or press harder for payment terms.

Key takeaways

- Four major AI companies said on 30 October 2026 they will honour one robots.txt signal for refusing generative AI training. - The signal is an IETF draft from an individual author, not an adopted standard, and rollout runs from "coming months" to 2027. - It covers future collection only, expresses a preference rather than a lock, and sets no licensing or payment terms. - Site owners keep their existing crawler rules for now, since none of the companies other than Anthropic has begun honouring the token.

Sources

- [SiliconANGLE: Google, Anthropic and others to support new AI crawling opt-out standard](https://siliconangle.com/2026/10/30/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard/) - [The Next Web: Google, Microsoft, OpenAI and Anthropic back one AI training opt-out](https://thenextweb.com/news/ai-training-opt-out-robots-txt-google-openai-microsoft-anthropic) - [Engadget: Google joins OpenAI and Anthropic in promising new scraping controls](https://engadget.com/2303225/google-openai-anthropic-ai-crawling-controls)

Tags: AI governance Anthropic OpenAI AI industry Regulation