Website owners should not rush to replace their existing crawler rules. OpenAI announced on 30 October 2026 that it plans to honour a common robots.txt signal for foundational generative-AI training, but said implementation is expected in the coming months. The proposal is also an Internet-Draft, not an approved IETF standard.
The useful action now is to decide how your content may be used, audit the controls already in place and keep discovery separate from model training. A blanket block can remove a site from useful AI search surfaces without necessarily expressing the intended training preference.
What changed — and what did not
OpenAI says the proposed shared signal would let a website operator communicate one training preference to multiple participating providers. The draft uses a policy token named __usage_ai-training inside ordinary robots.txt groups, so preferences can be expressed for the whole site or selected paths.
This is an announcement of intended support, not confirmation that the control is already active. OpenAI says its existing crawler controls will continue to apply. The IETF page also labels the proposal an individual Internet-Draft with no formal standing in the standards process, which means the detail may change.
Treat the draft as a direction of travel. Keep your present controls documented and monitor each provider's implementation notice before treating the shared token as an enforced policy.
Separate discovery, training and user-triggered access
OpenAI currently documents three different jobs. OAI-SearchBot helps content appear in ChatGPT search results. GPTBot crawls content that may be used to improve generative-AI foundation models. ChatGPT-User may visit a page when a person asks ChatGPT to retrieve it, and OpenAI notes that robots.txt rules may not apply to those user-initiated actions.
Those are different business decisions. A restaurant may want its public menu discoverable in search while keeping a private resource library out of training. A creative studio may allow its service pages to be found but restrict a licensed archive. A publisher may want path-by-path rules rather than one site-wide answer.
Do not copy a crawler block from another website without understanding the consequence. OpenAI's current documentation says a site can allow OAI-SearchBot while disallowing GPTBot. That separation is more useful than treating every AI access route as the same thing.
Use a four-step content preference review
Start with a short review that a founder, editor and technical owner can all understand. The aim is not to produce a perfect policy document. It is to make the intended use, current configuration and next review visible.
- List the public content groups that matter: service pages, articles, product documentation, customer areas, licensed work and downloadable assets.
- For each group, decide whether discovery is useful, whether training use is acceptable and whether any contract, licence or confidentiality term limits that choice.
- Compare the decision with the current robots.txt file, page-level indexing controls and access permissions. Do not use robots.txt as a substitute for authentication.
- Name an owner and a review trigger. Recheck when a provider publishes implementation details, when the draft changes materially or when the content rights change.
An illustrative website decision
Consider a small Manchester design studio. Its public service pages and case-study summaries help buyers understand the offer, so the studio wants them available for search discovery. Its paid templates and client workspaces have different rights and should already sit behind proper access controls rather than relying on crawler instructions.
The team records that public discovery is useful, training use needs a separate decision and private material must stay access-controlled. The technical owner confirms the present OAI-SearchBot and GPTBot rules, then adds a diary note to review the common signal after OpenAI publishes implementation details. This is an illustrative workflow, not a legal recommendation or a reported client result.
What to measure after any change
Record the exact robots.txt change, deployment time and affected paths. Check the live file and server logs where available. For discovery, use connected Search Console or analytics data and compare equivalent periods; do not infer impact from one successful page request.
OpenAI says its search systems may take about 24 hours to adjust after a robots.txt update. That is an operational note, not a promise that a page will appear in an answer. A successful deployment is also not proof of Google or ChatGPT indexation.
The immediate founder action is simple: write down the intended content policy before editing the file. If the team cannot distinguish public discovery, model training and private access, fix that decision layer first.
Sources & further reading
AI-assisted Klarnow editorial analysis. Sources checked on 5 November 2026. The shared signal is an Internet-Draft and OpenAI implementation was announced for the coming months; it is not presented as a live standard. The studio example is illustrative.