OpenAI says it will honor a common website AI-training preference signal

The short version

OpenAI says it will honor a common robots.txt signal through which website operators can express preferences about use of their content for foundational generative AI training. OpenAI says it plans to implement the signal in coming months and that its existing crawler controls will remain in place.

What changed

OpenAI committed to honoring a shared robots.txt training-preference signal intended to let operators express the same choice to multiple participating AI providers.

What it means for you

Publishers and teams that use public-web content should watch the standard's implementation, but retain current per-crawler controls until providers document their actual support.

OpenAI says it will honor a common robots.txt signal that lets website operators state whether their content may be used for foundational generative AI training. Rather than maintaining a separate rule for every crawler, an operator can communicate the same preference to multiple providers that participate in the signal.

The work builds on the IETF AI Preferences working group. OpenAI says it intends to implement the signal in coming months, but has not provided an implementation date or detailed behavior. It also says the shared signal will not replace its existing crawler controls.

Publishers should keep current crawler-specific rules in place and follow explicit support statements from their hosting, CDN, and AI providers. Teams building crawling, retrieval, or training workflows should treat the shared preference signal and existing site-specific controls separately until each provider documents the scope it enforces.

Fact check

VerifiedOpenAI announced it will honor a common robots.txt signal for preferences about foundational generative AI training and says the signal can communicate the same choice to multiple participating providers.Evidence

VerifiedOpenAI says it aims to implement the signal in coming months and that it will not replace existing robots.txt crawler controls.Evidence