Individual skill

wigolo-crawl

Local-first multi-page crawl with sitemap, BFS, DFS, and URL-map strategies, anchor-fragment dedup, rate limiting, robots.txt respect, and automatic local cache population. Use when the user wants to index documentation, crawl a docs site, extract all pages under a path, or says "crawl", "index this site", "get all the docs", "bulk extract". Prefer when crawled pages should land in a reusable local cache for later `cache` / `find_similar` queries.

skills/wigolo-crawl/SKILL.md

Source commit 0b38aebe471952c4412a07d7a08e615824ff36e1

What was checked

Matches the observed repository HEAD

Yes

Scanner rules are current

Yes

Repository scan complete

No

Skill scan complete

Yes

Observed behavior

Network or broad shell

Last checked

Nov 2, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.