Menu

Individual skill

agent-ready-ai-rules

0upvotes

Sub-skill de agent-ready-cloudflare: Implement AI Bot Rules in robots.txt

skills/agent-ready-cloudflare/ai-rules/SKILL.md

Read pinned source

Source commit ec4099f92fdeadcf1771c2a3f64efa4970733c89

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
Yes
Repository scan complete
No
Skill scan complete
Yes
Observed behavior
Instructions only
Last checked
Nov 11, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings)
133
Repository last updated
Nov 1, 2026
License
Apache-2.0

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt
# Implement AI Bot Rules in robots.txt

Add explicit `User-agent` entries for AI crawlers in your robots.txt per
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).

## Requirements

- Add `User-agent` blocks for AI search crawlers: `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`. These fetch the pages AI assistants use in their answers, so blocking them affects whether you appear in AI answers
- Add `User-agent` blocks for AI training crawlers: `GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`, `Bytespider`, `Amazonbot`. Blocking these does not affect whether your pages appear in AI search results
- User-triggered fetchers (`ChatGPT-User`, `Claude-User`, `Perplexity-User`) retrieve pages when a user asks; some vendors do not apply robots.txt to them
- Set `Allow` and/or `Disallow` rules matching your content policy
- A `User-agent: *` group with rules also passes, but applies one policy to every crawler. Explicit entries let you treat search and training crawlers differently

## Cloudflare
Back to top