OpenAI will honor one training signal. Make providers leave a compliance receipt

THE SIGNAL IN ONE SENTENCE

A small website can have a surprisingly large robots.txt file. One rule for the search crawler. One for the training crawler. Another for the shopping bot. Three more for companies that renamed their crawlers after the last time anyone checked. Then a new provider arrives, the person who maintained the file leaves, and the independent publisher is back in a control panel asking a very modern question: Which robot is this rule talking to? OpenAI announced on October 30 that it plans to honor a common robots.txt signal for foundational generative AI training preferences. Instead of writing separate rules for every participating provider, a website operator could use one cross-provider policy token. The proposed token is called __usage_ai-training. In the active Internet-Draft, a site can put that token in the User-Agent line and then use ordinary Allow and Disallow rules for specific paths. A site could express a preference against training across the whole domain while allowing a press kit, for example. The draft does not invent a new robots.txt grammar. It uses the path matching and group structure operators already know. It also keeps content-use preferences separate from traffic controls. A preference against AI training is not automatically an instruction that a crawler may not retrieve a page. That distinction matters. OpenAI says the shared signal will not replace its existing crawler controls. The company aims to implement the new signal in the coming months and says it will publish more detail when it does. So this is a commitment, not a completed deployment. The underlying document is also an individual Internet-Draft. The IETF page states plainly that it is not endorsed by the IETF and has no formal standing in the standards process. It may change, be replaced or expire. Still, the idea is useful. A shared signal lowers the maintenance burden for publishers. It can express a path-specific preference without forcing a local newspaper, illustrator, forum operator or tiny research archive to maintain a collection of vendor names like a zookeeper who has lost the clipboard. But the signal creates a second problem. How does the website operator know anyone honored it? A line in robots.txt is an instruction placed at the edge of the property. It does not produce a receipt. It does not identify every provider that read the file. It does not show which version was active when a page was retrieved. It does not prove that a training pipeline excluded the content. It does not say what happened to copies collected earlier. The draft itself says the preferences that apply to an asset are the ones expressed in the robots.txt file in effect when the asset was retrieved. That makes timing and version history essential. The plain-language signal is simple: One shared preference is better than twenty vendor-specific guesses. One shared preference without evidence is still a trust exercise. Every participating provider should leave a compliance receipt. Start with the declaration. Record the exact robots.txt file, retrieval time, response status and cryptographic digest. Preserve the policy-token group that was evaluated and the path rule that matched. If a cached copy was used, say when it was fetched and when the provider planned to refresh it. The website operator should be able to point to the version that was active at a particular moment. Without that record, a later dispute becomes a contest between two screenshots. Next, record the interpretation. Show the policy token, path, result and precedence rule. Did the provider read the path as allowed, disallowed or unspecified for AI training? Did a more specific proprietary token override the shared signal? Did a private agreement apply? The draft permits private agreements to override the public preference and says consumers may document that a proprietary token is more specific than the shared token. Those are reasonable possibilities, but they should not be invisible. If a contract changed the result, the receipt should name the agreement, effective date and covered paths without publishing confidential commercial terms. Then record the use. The shared token covers using an asset to modify the learned parameters of a generative AI model. The draft excludes models used exclusively for non-generative classification, ranking or scoring, even when they can generate a rationale. That definition gives providers a boundary to implement. It also creates edge cases. What about evaluation sets? What about retrieval indexes? What about synthetic-data generation? What about safety classifiers trained from the same page? What about a multimodal model that serves both generative and non-generative tasks? The receipt should state the purpose category used by the provider. If the content was not used for foundation-model training but entered another system, the operator should be able to see the distinction. Do not make the publisher reverse-engineer the provider's internal dictionary from a policy blog. Record the asset boundary too. A robots.txt rule attaches to a path, but a modern page can include images, transcripts, feeds, embedded documents, API responses and mirrors. A receipt should list the canonical URL and the related assets considered part of the decision. It should say whether redirects, alternate language versions, subdomains, query parameters and syndicated copies were evaluated separately. The goal is not to print a billion-line report for every crawl. The provider can publish aggregated records and offer event-level evidence when the operator needs to investigate a particular path. The aggregate layer should answer practical questions: How many URLs were evaluated? How many expressed a preference against training? How many were included, excluded, deferred or unresolved? How many relied on a private agreement or vendor-specific rule? How long did it take for a robots.txt change to affect downstream systems? How many parsing conflicts occurred? Those numbers should be broken out by time period and provider system, not hidden inside one global percentage. The event layer should support an audit. For a specific asset, show the declaration version, matched rule, interpretation, collection time, intended use, processing status and any later correction. If the provider excluded the content, say at which stage. If it was already in a dataset, say whether it was removed, quarantined, down-weighted, blocked from a future run or left unchanged under a stated policy. This is where the current announcement is quiet. OpenAI says it will honor the common signal and implement it in coming months. It does not yet publish the operational details, historical treatment, audit interface, refresh interval or evidence that a website operator will receive. That is not proof of failure. It is the work still to be done. The receipt also needs failure states. Robots.txt can be unavailable, malformed, redirected, blocked by a network error or changed during a crawl. A parser can disagree about precedence. A content pipeline can lose the URL that supplied an asset. A partner can provide a dataset without complete path-level provenance. Do not turn those cases into silent permission. Label them. Unreachable. Ambiguous. No preference expressed. Preference against training. Preference in favor. Overridden by a documented agreement. Pending verification. Excluded. Correction required. A shared vocabulary for failure is almost as valuable as the shared token. Publishers need their own controls as well. Keep versioned copies of robots.txt. Test rules against representative paths. Separate traffic rules from usage preferences. Monitor server logs for crawler requests where possible. Keep canonical URLs stable. Attach licenses and provenance to downloadable files. Record syndication and hosting relationships that may put the same work on another domain. Most importantly, do not describe the signal as a legal switch. The Internet-Draft defines a machine-readable preference. It does not settle copyright, contract, privacy, competition or database law. It does not create payment. It does not guarantee enforcement. It does not compel a provider that has not adopted it. It does not prove that older copies disappeared. The mechanism can support a legal or commercial position. It is not the whole position. Providers should also avoid turning support for the token into a marketing badge. "We honor publisher preferences" is too broad unless the provider explains the exact token, version, effective date, covered systems, refresh interval, exception rules, historical handling and verification path. That explanation should be stable enough for a publisher to cite and specific enough for an auditor to test. The IETF work is still moving. That is fine. The web often improves through drafts, experiments and implementation feedback before a standard settles. OpenAI's commitment may encourage other providers to converge on one mechanism. If several major consumers interpret the same token consistently, the maintenance benefit could be real. But interoperability needs more than shared syntax. It needs shared evidence. The useful sequence is: The site publishes a versioned preference. The provider retrieves it. The provider records the matched rule. The content pipeline carries that decision with the asset. The training system enforces the decision. The provider publishes aggregate results and supports path-level investigation. Corrections propagate through retained copies and future runs. The operator can verify the outcome without taking the provider's word for it. That is the compliance receipt. The new signal may simplify the first step. Now make the rest of the chain visible.