Individual skill

robots-txt

0upvotes

Use when applies to any public website. Use when auditing technical SEO foundations or diagnosing crawl coverage gaps.

skills/robots-txt/SKILL.md

Read pinned source

Source commit e8d14d049010e2ce25afcd5e8609dfcbbe63e47e

What was checked

Matches the observed repository HEAD: Yes
Scanner rules are current: Yes
Repository scan complete: No
Skill scan complete: Yes
Observed behavior: Network or broad shell
Last checked: Nov 9, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings): 74,680
Repository last updated: Oct 31, 2026
License: MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

frontend Repository topic
front-end-development Repository topic
front-end-developer-tool Repository topic
guidelines Repository topic
checklist Repository topic
reference Repository topic
web-development Repository topic
html Repository topic
css Repository topic
javascript Repository topic
marketing Derived category

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files

SKILL.mdcore · 1,931 bytes
references/rule.mddoc · 3,070 bytes

Read excerpt


# Publish a robots.txt file

robots.txt is the first file crawlers fetch; misconfigured directives can silently block search engines from crawling your entire site, killing organic visibility.

## Quick Reference

- Serve a valid `robots.txt` at `/robots.txt` on the production domain, returning HTTP 200
- Include a `Sitemap:` directive pointing to your XML sitemap
- Never disallow crawling of CSS/JS assets that render your pages
- Avoid blocking all crawlers with `Disallow: /` on a live site

## Check

Fetch `/robots.txt` on the live domain and verify it returns HTTP 200, uses correct `User-agent` / `Disallow` / `Allow` syntax, and includes a `Sitemap:` directive pointing to the XML sitemap. Check for accidental `Disallow: /` directives.