OpenAI, Google, Microsoft, and Anthropic agree on unified robots.txt AI training standard
In a landmark joint agreement, OpenAI, Google, Microsoft, and Anthropic have backed an IETF standard introducing a universal '__usage_ai-training' token in robots.txt to simplify publisher opt-outs.
In a rare display of industry-wide coordination, OpenAI, Google, Microsoft, and Anthropic have announced formal support for a unified web-crawling protocol aimed at standardizing how publishers and website owners opt out of generative AI model training.
The initiative, submitted as a formal Internet Engineering Task Force (IETF) draft led by Anthropic technical lead Kevin Kelley, introduces a single universal policy directive: '__usage_ai-training'. Rather than forcing webmasters to maintain complex, evolving lists of company-specific user-agent tokens (such as GPTBot, ClaudeBot, Bytespider, and Google-Extended), site operators can now specify a single rule in their robots.txt file to forbid or permit content harvesting across all participating AI laboratories.
Anthropic has confirmed that its ingestion pipelines have already implemented full enforcement for the new directive. OpenAI confirmed it will complete integration within sixty days, while Google and Microsoft have committed to full platform-wide compliance across their commercial indexing and training infrastructure. Digital publishing associations and copyright coalitions have widely commended the standard as a vital step toward transparency and content sovereignty.
Why it matters
Publishers previously had to configure dozens of proprietary crawler rules (GPTBot, ClaudeBot, Google-Extended). A single standard token provides legal clarity and automated compliance.