Individual skill
geo-crawlers
AI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which AI crawlers can access the site. Provides a complete access map and recommendations for maximizing AI visibility while maintaining appropriate control.
skills/geo-crawlers/SKILL.md
Source commit 989cae01e8ebbc42a9ec798eb9e7cb423f5ec89c
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- No
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Not checked yet
- Last checked
- Nov 9, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 11,639
- Repository last updated
- Nov 12, 2026
- License
- MIT
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- engineering Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Read excerpt
# AI Crawler Access Analysis Skill
## Purpose
This skill analyzes a website's accessibility to AI crawlers -- the bots that AI companies use to discover, index, and train on web content. If AI crawlers are blocked, the site's content cannot appear in AI-generated responses regardless of its quality. Crawler access is the foundational technical requirement for GEO.
## Key Insight
As of early 2026, many websites inadvertently block AI crawlers through overly aggressive robots.txt rules, inherited from legacy SEO configurations. An Originality.ai 2025 study found that over 35% of the top 1,000 websites block at least one major AI crawler, and 5-10% block all AI crawlers. Blocking AI crawlers is the single fastest way to become invisible in AI-generated search results.
---
## Complete AI Crawler Reference
