Direct answer: The robots.txt AI training opt-out is a new shared line, User-Agent: __usage_ai-training followed by Disallow: /, that Google, Microsoft, OpenAI and Anthropic said on 30 October 2026 they will honour as your “no generative AI training” preference. Nobody reads it yet: Microsoft starts on 1 February 2027, Google in early 2027, OpenAI “in the coming months”, and Anthropic says it is implementing it. It does not change Google or Bing rankings. But at Google it is planned to act like a Google-Extended block, which also takes your pages out of Gemini’s cited answers, so Gulf and Egypt sites should decide path by path, not with one blanket line.
Key facts (checked 1 November 2026, 09:15 Asia/Gaza)
- What: a vendor-neutral robots.txt policy token,
__usage_ai-training, defined in the individual IETF draft draft-kelley-aipref-training-preference-signal-00 (Kevin Kelley, Anthropic, 30 Oct 2026, expires 3 May 2027). - Status: not an IETF standard. The datatracker says the draft “is not endorsed by the IETF and has no formal standing” (IETF).
- Who backs it: Google, Microsoft Bing and OpenAI posts on 30 Oct 2026; Anthropic via Kelley’s ai-control list message (15:39 UTC, 17:39 Asia/Gaza).
- Start dates: Microsoft 1 Feb 2027; Google “early 2027” (crawler docs) / “2027” (blog); OpenAI “coming months”; Anthropic “already started implementing”, no date.
- Wildcards do not count: a
User-agent: *group never matches the token. No token group means no preference. - Rankings: Google and Microsoft both say search inclusion and ranking are not affected.
- Google catch: a token opt-out with no Google-Extended line is read as a Google-Extended disallow, which also covers grounding in Gemini Apps and Vertex AI.
- Microsoft extras: page-level
ai-trainingandai-groundingsettings; NoArchive/NoCache stop counting as AI preferences after 1 Mar 2027.
The robots.txt AI training opt-out became a cross-industry signal on Friday 30 October 2026. Google, Microsoft and OpenAI each published a post saying they will honour one shared robots.txt line as a website’s preference about generative AI training. The same afternoon, Anthropic engineer Kevin Kelley submitted the IETF draft that defines it and wrote that Anthropic has “already started implementing support”. For news publishers, online stores, SaaS companies and government portals in the UAE, Saudi Arabia and Egypt, this is the first time one line can speak to all the big model makers at once. It also raises a question English-language coverage has mostly skipped: if you are publishing in Arabic, do you really want to disappear from AI answers?
This guide explains what the token does and doesn’t do, and when each company starts reading it. It gives copy-ready snippets for WordPress, Nginx and Cloudflare sites, plus a decision table built for Gulf and Egypt site types. Every date and quote below comes from the four primary announcements and the IETF draft. Where a detail is our reading rather than a company statement, we say so.
What is the robots.txt AI training opt-out (__usage_ai-training)?
robots.txt is the plain-text file at the root of every website that tells crawlers what they may fetch. Its rules are standardised in RFC 9309. Until now, a site that wanted to say “don’t train AI on my content” had to name each company’s crawler or control token one by one: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on. Google put it plainly: website owners were “managing separate instructions for every AI developer”.
The new draft keeps the robots.txt grammar exactly as it is. Instead of a crawler name, the User-Agent line holds a policy token, __usage_ai-training. The rules under it say where you are against, or in favour of, AI training. This is the draft’s own example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That group “expresses a preference against AI training for all paths except those under /press-kit/, for which it expresses a preference in favor” (IETF draft). Path matching works like any robots.txt rule, with the longest match winning. The definition of training comes from the IETF AIPREF working group’s vocabulary draft: “Using an asset to modify the learned parameters of a generative AI model.” Training non-generative models used only for classification, ranking or scoring is excluded.
It is a preference about use, not a gate
The draft says the token “is not a directive for crawling behavior”. Crawlers still follow the rules for their own names. A crawler *may* choose not to fetch pages it can’t use, but nothing in robots.txt technically stops a bot. If you need enforcement, that is a firewall or bot-management job.
Which companies will honour it, and when?
| Provider | What it said on 30 Oct 2026 | When | Existing controls |
|---|---|---|---|
| Will treat a token opt-out “as if the publisher had disallowed Google-Extended” unless Google-Extended is explicitly specified (blog) | “In 2027” (blog); “early 2027” (crawler docs) | Google-Extended keeps working | |
| Microsoft Bing | Adopts the shared AI Training preference, adds page-level training and grounding settings (Bing blog) | Honoured as it crawls from 1 Feb 2027 | NoArchive / NoCache retired as AI signals after 1 Mar 2027 |
| OpenAI | Will honour the shared robots.txt signal for “foundational generative AI training” (OpenAI) | “In the coming months”; details later | GPTBot and OAI-SearchBot rules stay in place |
| Anthropic | “Has already started implementing support for this draft” (Kelley, IETF list) | No date; “it’ll take us some time” | Complements existing crawler controls |

Two things are missing from the table. First, Apple, Meta and Amazon did not announce support on 30 October, so their crawlers still need their own rules if you want to address them. Second, the signal is voluntary. OpenAI calls its participation a “voluntary effort”, and the draft has no enforcement mechanism.
Does the robots.txt AI training opt-out affect Google or Bing rankings?
No. Google’s crawler documentation says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (Google). Microsoft says the new preferences apply “without changing eligibility for Bing search results” (Bing). The token also doesn’t change how Googlebot or Bingbot crawl, because those crawlers keep reading their own groups. If you have spent years on SEO for an Arabic site, adding the token won’t move your blue-link rankings.
Will blocking AI training remove my site from Gemini, Copilot or ChatGPT answers?
This is the question that matters most for visibility, and the answer is different at each company. Training changes what future models *know*. Grounding decides which pages an assistant reads and cites, with links, when it answers a question *today*. The two are easy to confuse, and the companies treat them differently.

- Google: Google-Extended covers training of future Gemini models *and* grounding, meaning content from the Google Search index supplied at prompt time, in Gemini Apps and Grounding with Google Search on Vertex AI (crawler docs). So a token opt-out without a Google-Extended line would, on our reading of Google’s two documents, also remove your pages from Gemini’s grounded answers. Google’s blog states the flip side: “Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” AI Overviews and AI Mode in Google Search are governed by Googlebot rules, not by this token.
- Microsoft: the token covers training only. Grounding in Bing, Copilot and Microsoft 365 has its own page-level setting with three values:
Allow(default),DisallowExtended(Microsoft’s own products only, not third-party grounding APIs such as Web IQ) andDisallow(Bing). - OpenAI: ChatGPT’s search features use OAI-SearchBot, which OpenAI documents as independent of GPTBot, its training crawler (OpenAI crawler docs). OpenAI hasn’t yet said how the shared token maps onto these.
- Anthropic: Kelley says the token complements Anthropic’s existing crawler controls; implementation details have not been published.
“Neither choice guarantees inclusion, traffic, or payment.”
Microsoft Bing, on allowing training and grounding, 30 October 2026

Google-Extended, GPTBot, ClaudeBot and the new token: what’s the difference?
| Name in robots.txt | Type | Who reads it | What it controls |
|---|---|---|---|
__usage_ai-training | Shared policy token (new) | Google, Microsoft, OpenAI, Anthropic, once live | Generative AI training preference, by path |
Google-Extended | Google control token (no separate crawler) | Gemini training and Gemini Apps / Vertex AI grounding | |
GPTBot | OpenAI crawler | OpenAI | Content that may be used to train OpenAI foundation models |
OAI-SearchBot | OpenAI crawler | OpenAI | Appearing in ChatGPT search answers |
ClaudeBot | Anthropic crawler | Anthropic | Content that may contribute to training |
User-agent: * | Wildcard | Every crawler | Crawl access. Never an AI training preference under the draft |
Bingbot meta ai-training / ai-grounding | Page-level setting | Microsoft, from 1 Feb 2027 | Training override per page; grounding per page |
The draft’s precedence rule is short: “Specific expressions SHOULD take precedence over general ones”, and a company may treat its own proprietary token as more specific. Google has done exactly that with Google-Extended. The draft also says “private agreements MAY override preferences”, so a licensing deal with an AI company can take priority over the token.
Why doesn’t my existing User-agent: * block count?
⚠️ Common mistake
If your robots.txt says User-agent: * and Disallow: /, you’ve blocked crawling, not stated an AI training preference. The draft is explicit: wildcard groups “MUST NOT be interpreted as matching the policy token”, and if there is no __usage_ai-training group, “no preference has been expressed”. You have to write the token by name.
Kelley put the design choice this way: “a robots.txt file that doesn’t use this mechanism, as is true of every one published today, expresses no preference at all” (IETF list). It protects millions of old robots.txt files from being reread in ways their owners never intended. The flip side: until you add the line, the new signal says nothing on your behalf. ifynx.com is in the same position. As of this morning its robots.txt has no __usage_ai-training group, like almost every site on the web.
The Arabic trade-off: why Gulf and Egypt sites have leverage
Most English coverage treats opting out as the obvious choice for publishers. In Arabic the maths is different. There is far less high-quality Arabic text on the open web than English text, so a well-edited Arabic news archive, a bilingual product catalogue or a Saudi or Emirati government services guide is a scarcer resource for model makers. That scarcity gives Arabic content owners leverage, and it is also why AI assistants often answer Arabic questions with thin or translated sources.
The same scarcity cuts the other way for visibility. If your Arabic pages are the best answer to a question about Saudi e-invoicing, UAE visa services or Egyptian shipping rules, being read and cited by Gemini, Copilot or ChatGPT is free distribution. Opting out of training at Google, without an explicit Google-Extended allow, also opts you out of Gemini grounding. For a store or a SaaS company that wants to be recommended, that can cost more than it protects.
Our rule of thumb
Protect what you sell, expose what sells you. Paid journalism, research, courses and proprietary data: opt out of training. Product pages, documentation, pricing, help centres and press material: keep them readable, and decide on grounding separately for each provider.
Should my site opt out? A decision table for Gulf and Egypt sites
| Site type | Training (__usage_ai-training) | Google-Extended | Bing grounding (page level) | Why |
|---|---|---|---|---|
| Arabic/English news publisher (paid or ad-funded) | Disallow /; allow /press/, about and contact pages | Leave unspecified (blocked via token) unless Gemini referrals matter | DisallowExtended on premium articles | Archive is the product; licensing talks need a clear opt-out on record |
| E-commerce store (UAE, KSA, Egypt) | Allow, or disallow only /blog/ and reviews | Allow | Allow | AI shopping answers drive product discovery; product data is meant to spread |
| SaaS / fintech documentation | Allow /docs/; disallow customer-specific or gated paths | Allow | Allow | Developers ask assistants how to integrate; wrong answers cost support time |
| Government and semi-government portals | Follow the entity’s data policy; usually allow public service guides | Allow for public guides | Allow | Citizens ask AI in Arabic; accurate cited answers reduce call-centre load |
| Universities, research and course sites | Disallow course material and papers; allow admissions pages | Case by case | DisallowExtended on paid courses | Course content is the asset; admissions info should be found |
| Banks and regulated entities | Disallow customer-facing tools and calculators; allow product and fee disclosures | Allow disclosures | Allow disclosures | Accurate fee and product answers matter more than keeping them out |

This table isn’t legal advice. Copyright and data-use rules differ between the UAE, Saudi Arabia and Egypt, and we didn’t find a regulator statement in any of the three countries on this token as of 1 November 2026. If your organisation has a legal or data-governance team, run the decision past them.
Ready robots.txt snippets for WordPress, Nginx and Cloudflare
1. Opt out of AI training for the whole site
# Opt out of generative AI training for the whole site
User-Agent: __usage_ai-training
Disallow: /
# Keep normal crawling and search rules as they are
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml
2. Opt out of training but stay eligible for Gemini answers
Because Google-Extended covers both training and grounding, Google gives you no setting for “cite me, but don’t train on me”. Adding an explicit Google-Extended allow keeps Gemini grounding, and it also allows Gemini training. Choose this knowingly.
# Opt out of training at providers that read the shared token...
User-Agent: __usage_ai-training
Disallow: /
# ...but stay eligible for Gemini grounding (note: this also
# allows Gemini training, because Google-Extended covers both)
User-agent: Google-Extended
Allow: /
3. Path-level: protect archives, keep products and docs open
# News archive and paid research: no training.
# Product pages, press kit and docs: training allowed.
User-Agent: __usage_ai-training
Disallow: /
Allow: /products/
Allow: /press/
Allow: /docs/
Google’s blog post describes the site-wide case only. It hasn’t said yet how path-level token rules will be translated into Google-Extended rules, so re-check Google’s documentation before relying on path rules at Google.
WordPress
WordPress serves a virtual robots.txt unless a real robots.txt file sits in the web root. You have three options. Upload a physical file. Use the robots.txt editor in your SEO plugin. Or add a small must-use plugin that prepends the group to the virtual file:
<?php
// wp-content/mu-plugins/ai-training-pref.php
// Adds the shared AI training opt-out to WordPress's virtual robots.txt.
// Ignored if a physical robots.txt file exists in the web root.
add_filter( 'robots_txt', function ( $output, $public ) {
if ( ! $public ) { return $output; }
return "User-Agent: __usage_ai-training\nDisallow: /\n\n" . $output;
}, 10, 2 );
Nginx (multi-site servers)
# Serve a static robots.txt for every vhost you manage
location = /robots.txt {
alias /etc/nginx/robots/robots.txt;
default_type text/plain;
add_header Cache-Control "public, max-age=3600";
}
Cloudflare
If you turn on Cloudflare’s managed robots.txt (Bot Preference Sync), Cloudflare prepends its own block before your file. That block includes Content Signals and vendor rules such as Google-Extended and GPTBot (Cloudflare docs). Cloudflare’s documentation, last updated 22 October 2026, does not mention __usage_ai-training, so add the token in your origin file and check the merged result. A managed Google-Extended: Disallow line counts as “explicitly specified” at Google, so it overrides whatever your token group says.
Microsoft’s page-level settings and the NoArchive/NoCache deadline
Microsoft is the only company with a full transition plan. If you use NoArchive or NoCache today to keep pages out of Copilot, those tags stop counting as AI preferences after 1 March 2027. Microsoft also says that once it detects the new preferences on a site, it will “disregard any NoArchive and NoCache preferences that may remain”. Add the token alone and you could lose your grounding restriction. Publish the page-level grounding setting at the same time:
<!-- Page level, Microsoft only (from 1 Feb 2027) -->
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">
# Or as HTTP response headers
X-Robots-Tag: Bingbot: ai-training:disallow
X-Robots-Tag: Bingbot: ai-grounding:disallowextended
- 15 Dec 2026: temporary grounding preferences open in Bing Webmaster Tools, only for sites using NoArchive or NoCache on that date.
- 1 Feb 2027: Microsoft starts honouring AI Training and AI Grounding preferences as it crawls.
- 1 Mar 2027: NoArchive and NoCache are no longer read as training or grounding preferences.
- 30 Sep 2027: any temporary preference still in place expires.
How to test your robots.txt after the change
- Fetch the live file with curl, not your CMS preview. Confirm
200,text/plainand that the__usage_ai-traininggroup appears exactly once. - Check the merged file behind your CDN. Cloudflare or another CDN may prepend managed rules. Read the whole response top to bottom.
- Spell the token exactly: two leading underscores,
usage, underscore,ai-training. Matching is case-insensitive, but a typo is just an unknown user agent that every crawler ignores. - Keep crawl rules separate. Don’t delete your Googlebot or
User-agent: *rules. The token adds a use preference; it doesn’t replace access rules. - Check for conflicts. If an explicit
Google-ExtendedorGPTBotgroup exists, it decides for that company, whatever the token group says. - Watch the file size. RFC 9309 requires parsers to handle at least 500 kibibytes. The token exists partly so you can stop duplicating long vendor lists.
- Validate in Search Console and Bing Webmaster Tools robots.txt reports. Search Console may flag newer lines it doesn’t understand; that alone isn’t an error.
- Log the date you changed it. The draft ties preferences to the robots.txt in force when a page was fetched, and nothing is retroactive.
curl -s https://example.com/robots.txt | grep -n -A3 -i "__usage_ai-training"
curl -sI https://example.com/robots.txt | grep -i -E "^(HTTP|content-type)"
What the token doesn’t do
- It isn’t retroactive. Content already collected isn’t covered; the preference applies to assets retrieved while it is in force.
- It isn’t a standard yet. It’s an individual draft that expires on 3 May 2027 unless revised.
- It isn’t enforcement. Compliance is voluntary. Bots that ignore robots.txt today will ignore this too.
- It isn’t payment. Microsoft says the preferences “do not establish licensing or payment arrangements”.
- It doesn’t cover AI Overviews or AI Mode in Google Search, which follow Googlebot rules and Google’s separate Search controls.
How iFynx helps
iFynx (آيفينيكس) builds bilingual Arabic and English websites, apps and AI products for companies in the UAE, Saudi Arabia and Egypt. We also work on how those products show up in AI search. For this change we offer a short review. We map which of your paths are product, which are assets and which are neither, then write the robots.txt and page-level settings for your CMS and CDN. Then we set up a check that the live file stays correct after every deploy. See our AI and data services or AI development in the UAE.
Frequently asked questions
How do I block AI training in robots.txt?
Add a group whose User-Agent line is the shared policy token __usage_ai-training, followed by Disallow: / (or the paths you want to protect). Google, Microsoft, OpenAI and Anthropic said on 30 October 2026 that they will honour it, but none reads it yet: Microsoft starts on 1 February 2027, Google in early 2027, OpenAI in the coming months, and Anthropic says it has started implementing it. Keep your existing GPTBot, Google-Extended or ClaudeBot rules, because the providers say those keep working.
Does __usage_ai-training affect my Google or Bing rankings?
No, according to both companies. Google says Google-Extended, which the shared token maps to, does not affect inclusion in Google Search and is not a ranking signal. Microsoft says its training and grounding preferences do not change eligibility for traditional Bing search results. The token is not a crawl directive either, so Googlebot and Bingbot keep following your normal rules.
Will blocking AI training remove my site from Gemini, Copilot or ChatGPT answers?
It depends on the provider. At Google, a shared-token opt-out without an explicit Google-Extended line is planned to work as a Google-Extended block, and Google-Extended also covers grounding in Gemini Apps, so your pages would stop being eligible to appear with links in Gemini answers. At Microsoft, training and grounding are separate, so Copilot and Bing AI answers are controlled by a page-level grounding setting instead. OpenAI’s ChatGPT search uses OAI-SearchBot, which is separate from training. AI Overviews and AI Mode in Google Search are not governed by the token.
What is the difference between Google-Extended, GPTBot, ClaudeBot and __usage_ai-training?
Google-Extended, GPTBot and ClaudeBot are company-specific names: each tells one company something. The draft calls Google-Extended a proprietary policy token, while GPTBot and ClaudeBot are crawler names. __usage_ai-training is one vendor-neutral token that any participating company reads as your AI training preference. Under the draft, specific rules can take precedence over the general one, so an explicit Google-Extended group still wins at Google.
Should Arabic news sites and online stores in the Gulf and Egypt opt out?
There is no single right answer and this is not legal advice. Sites whose content is the product, such as paid news, research and course material, have the strongest reason to opt out of training. Stores, SaaS documentation and government service pages usually gain more from being understood and cited by AI assistants, especially in Arabic, where good web content is scarcer than in English. A common middle path is a path-level rule: protect archives and paid content, allow product pages, documentation and press material.
Related reading
- WordPress 7.1.4 security release: update checklist for Gulf and Egypt sites
- PHP vulnerability 2026 (php2shell): patch checklist for WordPress and Laravel
- GPT-6.1 Terra: price, Arabic computer use and invoice automation
- Bilingual WordPress without waiting
- Google UCP + Shopify native checkout: opt-out UX for agentic storefronts
- What is iFynx? Company profile and services
Sources
- IETF, draft-kelley-aipref-training-preference-signal-00 (Kevin Kelley, 30 Oct 2026) and datatracker status
- Kevin Kelley, ai-control list announcement (30 Oct 2026, 15:39 UTC)
- IETF AIPREF, vocabulary draft -08 (13 Sep 2026)
- Google Search Central Blog, Supporting a shared AI training control for websites (30 Oct 2026); Google common crawlers documentation
- Microsoft Bing Blog, Your content, your choice: clearer preferences for training and grounding (30 Oct 2026)
- OpenAI, Making it easier for websites to express AI training preferences (30 Oct 2026); OpenAI crawlers
- Cloudflare, robots.txt setting documentation (updated 22 Oct 2026); IETF, RFC 9309
- Coverage: PPC Land (31 Oct 2026), The Next Web
- Hero photo: Victor Grigas / Wikimedia Foundation, Wikimedia Commons, CC BY-SA 3.0, cropped. In-body photo: Tony Webster, Wikimedia Commons.jpg), CC BY 2.0, cropped.
