One Robots.txt Switch for AI Training

The conversation

AI-generated episode: the script is written with AI from the cited sources and read by synthetic voices; the hosts are fictional.

Sveva

Hello and welcome to Hot Podcast! Today: the humble robots.txt file, a tiny text file with a surprisingly big job. Stefano, Matteo, are we ready to talk about website doors and who gets to knock?

Stefano

Ready! The news is from Unite.AI, published October 31, 2026. It reports that Google announced on October 30 that it plans to support a shared AI training control in robots.txt, based on IETF AI Preferences work and building on Google-Extended. Google aims for 2027.

Stefano

The article also says Microsoft and OpenAI made same-day announcements. Per that coverage, OpenAI says it will honor the common signal, and Anthropic has reportedly started implementing support. These are the companies' own statements, not independent proof of adoption.

Sveva

So the big idea is one way to say your training preferences, instead of juggling a different rule for every provider?

Stefano

That's my analysis, yes. A shared signal could reduce guesswork for site owners. But it's still a preference signal, support is planned, and timing varies. Also, training is not the same as search visibility or citations.

Matteo

Which is where people trip. Someone blocks a bot to protect content, then wonders why they vanished from AI answers. Robots.txt is tiny, but one wrong line can feel like locking the front door with yourself outside.

Sveva

Matteo, is there a skill on HotSkills that connects to this?

Matteo

Yes: llm-crawler-access-check, from the buildwithclaude repository. Its description says it reads only robots.txt, returns a per-agent allow or block table, names the rule behind each verdict, and suggests lines to change. It distinguishes training crawlers from search crawlers.

Stefano

Here's a hypothetical example. A small publisher wants to limit training but stay visible in AI search. The skill could show which agents are blocked and why. Then they could compare that table with new shared guidance once providers support it.

Matteo

One note: the repository's heat metric was 7.0 in the November 1, 2026 snapshot. That's a repository measure, not a Hot vote. And it only reviews robots.txt, so it won't tell you everything about page-level controls.

Sveva

So the takeaway: read the news as a plan, not a finished switch. Know what you're blocking, and why.

Stefano

Exactly. Audit first, change deliberately, and watch each provider's real implementation dates.

Sveva

Thanks, both! You can find the episode sources on HotSkills. Until next time, keep your robots.txt tidy and your curiosity open. Goodbye!