On October 30, the four companies that train most of the world’s frontier models did something they had never done together: they agreed on how a website says no. OpenAI, Google, Microsoft, and Anthropic each announced they will honor a single shared signal in a site’s robots.txt file expressing that its content should not be used to train generative AI models. For publishers who have spent three years maintaining a separate opt-out for every lab’s crawler, that is a genuine breakthrough. It is also, on inspection, a voluntary request with staggered deadlines, an unfinished standard, and a Meta-shaped hole in the middle of it. Both of those things are true at once, and the second half is where the story actually lives.

What was agreed

The mechanism is deliberately boring, which is the point. A publisher adds a rule to robots.txt — the plain-text file crawlers conventionally read before fetching pages — addressed to a new user-agent label, __usage_ai-training, with Disallow: /. Any participating company reading that rule is supposed to treat it as a preference against using the site’s content for AI training.

The signal itself is defined in an Internet-Draft titled “Machine-Readable Training Preference Signal,” submitted to the Internet Engineering Task Force on October 30 by Kevin Kelley, Anthropic’s technical lead for standards, who announced it to the IETF’s AI Preferences working group mailing list the same afternoon. OpenAI published “Making it easier for websites to express AI training preferences”; Google’s post on the Search Central Blog was “Supporting a shared AI training control for websites”; Microsoft’s Bing blog offered “Your content, your choice: clearer preferences for training and grounding.”

The timelines, however, are not shared at all. Anthropic says it has already begun implementing support. OpenAI says it aims to honor the signal for foundational generative AI training “in the coming months.” Google targets early 2027. Microsoft will start honoring the training preference as it crawls from February 1, 2027. A publisher flipping the switch today is therefore making a request that one company may act on soon, one in a few months, and two sometime next year. That stagger matters for any newsroom deciding whether the signal changes its legal or commercial posture right now: it doesn’t, not yet.

A publisher's newsroom desk at dusk, monitors glowing, print editions stacked beside a keyboard

What it doesn’t do

The limits are structural, and they are worth stating plainly because the announcement’s framing invites overreading.

First, it is a preference, not a block. Robots.txt has always worked on convention: a crawler reads it and chooses to comply. The shared signal inherits that honor system. It does not technically prevent a page from being fetched, indexed, or shown in search results, and it does not govern every use of web content — answers grounded in retrieved pages, for instance, sit in a different bucket from training, and Microsoft’s own post treats training and grounding as separate preferences.

Second, it is not yet a standard. Kelley’s draft is an individual Internet-Draft submission, not an adopted IETF standard, and the working group discussing AI preferences has not declared consensus on it — one of its chairs said as much at an October 21 meeting. What the four companies announced is a set of company commitments to a proposal, not an industry-wide rule, and commitments can be revised. Microsoft’s note that its older NoArchive and NoCache signals will stop counting as training or grounding preferences after March 1, 2027 is a reminder that the ground keeps moving under publishers’ feet.

Third, it says nothing about the past. The signal governs future training use going forward from each company’s implementation date. It does not reach data already ingested, models already trained, or the lawsuits already in progress. On the same day as the announcement, the Wikimedia Foundation filed an amicus brief in The New York Times’ case against OpenAI in the Southern District of New York — a useful reminder that the courts are still deciding the questions this voluntary mechanism sidesteps.

The Meta-shaped hole

The most conspicuous name in the announcement is the one that isn’t a party to it. Meta voiced support for a common signal but did not commit to honoring it, and Meta’s own trajectory on October 30 ran in the opposite direction. The company delayed enforcement of its revised business terms — which would expand the scope of web content Meta may use for its AI systems — from October 30 to November 20, after pushback from news publishers and advertisers and complaints to antitrust authorities in the European Union and the United Kingdom.

That split is the real map of the industry right now. Four labs chose coordination on a narrow, voluntary control. The company with the largest social distribution chose to keep its own terms, on its own timetable. For a publisher, the practical consequence is blunt: the shared signal does not produce a single web-wide opt-out. It produces a four-company opt-out plus a separate Meta negotiation, conducted under commercial terms rather than a standard.

Why it still matters

None of that makes the agreement empty. Coordination failures were the publishers’ core complaint: every lab ran its own crawler, its own control, its own timeline, and opting out meant doing the work four times and trusting four separate implementations. A single token collapses that to one line, and a single line is something small publishers — the ones without legal teams to track each lab’s documentation — can actually deploy. Standards also have a ratchet effect: once four major labs read the same token, the cost for the fifth to ignore it rises, and the token becomes the obvious thing for regulators and courts to point at when they ask what the industry considers feasible.

The right way to read October 30 is as infrastructure, not settlement. The web just got its first shared vocabulary for refusing AI training. Whether that vocabulary becomes a norm depends on the unglamorous work ahead: the IETF process reaching consensus, the staggered implementations actually shipping on schedule, Meta deciding whether support becomes commitment, and publishers testing — not assuming — that the signal is honored. The one line in robots.txt is real. The enforcement behind it is still a promise, dated at different times, by different companies, in different words. Watch the dates: early 2027 for Google, February 1, 2027 for Microsoft, “the coming months” for OpenAI — and November 20 for Meta’s terms. That calendar, not the announcement, is where this story gets decided.