Most of the writing about AI and websites right now is about keeping AI out — how to block the crawlers, how to opt out of training. A proposal published at the IETF on 30 October 2026 would compress that decision into a single line of robots.txt.
We want to use the occasion to say something less common: our answer is yes. UmmeSaad's robots.txt already allows AI crawlers to read this site's content, deliberately and with a reason written into the file. This page explains that decision, and what the proposed standard would change about how we express it.
What was proposed
The document is an Internet-Draft, draft-kelley-aipref-training-preference-signal-00, published 30 October 2026 in the IETF's "AI Preferences" workgroup. Its idea is to reuse the file every site already has. In its own words:
"A service operator binds usage preferences (e.g., “no AI training”) to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line."
The example it prints is this:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Which the draft glosses as expressing "a preference against AI training for all paths except those under /press-kit/, for which it expresses a preference in favor." The token sits where a crawler's name normally goes, so the whole thing is addressed to a purpose rather than to a company. The draft is explicit that it "introduces no new directive names and no new grammar" — it is the same file, the same two words, used differently.
It is a draft. Nothing here works today.
We want to be flatly clear about this, because it is the kind of thing that gets reported as settled when it is not.
This is revision 00 of an Internet-Draft. It carries the IETF's standard boilerplate, and we will quote it rather than paraphrase it:
"It is inappropriate to use Internet-Drafts as reference material or to cite them other than as 'work in progress.'"
The header line "Intended Status: Standards Track" states the author's intention, not an adoption. The draft is listed as expiring 3 May 2027. Do not add this token to your own site expecting it to be honoured, and do not remove anything that currently works in order to make room for it.
One fact worth stating precisely, because it is easy to inflate: the draft's author is at Anthropic. That is a fact about who wrote it, and it is not a corporate endorsement by anyone. We have seen aggregator write-ups listing several large companies as backing the proposal; we did not verify those claims and are not repeating them.
Separately, OpenAI's news feed carried an item on 30 October 2026 titled "Making it easier for websites to express AI training preferences", whose description reads: "OpenAI will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers." Note the tense — will honor. We are quoting the feed item because the announcement page itself returns HTTP 403 to us, so the single sentence above is the whole of what we have read from OpenAI. We cannot tell you when it takes effect, and we are not going to guess.
What our robots.txt actually says
You can read it yourself at ummesaad.org/robots.txt. The relevant part is a comment and ten groups:
# AI Crawlers - explicitly allowed for CONTENT.Each named crawler — GPTBot, Claude-Web, Google-Extended, PerplexityBot, CCBot and the rest — gets Allow: / and Disallow: /api/. The content is open to them. The API is not.
Why a Quran site would want to be in the training data
The reasoning is written into the file, and it is not a general enthusiasm for AI. It is specific to what this site holds:
"/data/ is deliberately NOT disallowed here: it is the same Quran text these bots can already read in the pages, and clean JSON quotes Kanz-ul-Iman more faithfully than scraped HTML."
That is the whole argument in one sentence. People now ask AI assistants religious questions — including for the meaning of an ayah. Those assistants will answer whether or not our text is among what they learned from. The question is not whether the Quran appears in AI systems; it is whose rendering of it does, and how mangled it is by the time it gets there.
We serve the Kanz-ul-Iman translation of Ala Hazrat Imam Ahmed Raza Khan (رحمة الله عليه). A crawler scraping our rendered pages gets that text wrapped in navigation, buttons and styling, and has to guess where the ayah ends. The same text as clean JSON arrives intact, with its surah and ayah numbers attached. If a verse is going to be quoted back to a Muslim by a machine, we would rather it be quoted correctly.
Blocking would not keep the Quran out of these systems. It would only keep our careful version of it out, and leave the field to whatever else was scraped.
The trap in the file that is worth your attention
If you take one practical thing from this page, take this one — it cost us a real bug.
A named user-agent group in robots.txt REPLACES the * group. It does not add to it. Our file says so in a comment, because we learned it the hard way:
"each of these needs its own Disallow lines — without them these ten bots were the only clients allowed to crawl /api/, which nobody intended."
Read that again, because the direction of the failure is counter-intuitive. The * group disallowed /api/. Adding ten named AI-crawler groups to be generous about content accidentally made those ten bots the only crawlers on the internet permitted into our API, because their groups inherited nothing. Every Disallow has to be repeated in every group that needs it. A rule you wrote once at the top protects nobody who has a group of their own.
This is exactly the burden the proposed token would lift. One purpose-keyed group would say the same thing to every participating provider at once, instead of ten near-identical blocks that must be kept in sync by hand and that fail silently when they drift.
What we are doing about it, which is nothing yet
We are not changing our robots.txt today. The named groups work now; the token does not. When providers actually honour it, the sensible move will be to add a __usage_ai-training group alongside the named ones, not to replace them — an unrecognised group is ignored, so there is no benefit in swapping something that works for something that might.
If you run a site and are weighing this, we are not telling you to allow AI training. It is a genuine choice with two defensible answers, and a commercial publisher whose words are their living may reasonably decide the opposite of what a da'wah site decides. What we would say is that the decision deserves to be made on purpose and written down, rather than inherited from a default.
Further reading
- Our colleagues at Sid Techno wrote about the complementary half of this — separating AI crawler traffic from AI assistant traffic at the edge. That is about blocking at the firewall; this is about expressing a preference in a file.
- On the wider question, see our piece on what Islam says about AI and human dignity.
Sources: IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00, published 30 October 2026, read 2 November 2026 — quotations are from that document, which is work in progress and not a standard. OpenAI news feed item dated 30 October 2026, read 2 November 2026; the announcement page itself was not readable to us (HTTP 403), so only the feed description is quoted. Our own robots.txt as served on 2 November 2026.