Individual skill
crawl4ai
Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction. Triggers on crawl4ai, crwl, scrape JS-heavy site, scrape SPA, headless browser scrape, schema-based extraction, batch crawl, sitemap crawl, web data pipeline. SKIP when a static HTML page can be read with `defuddle` / `fetch-web` — those are faster cold-start and don't need a browser.
SKILL.md
Source commit 98f71583e27fe07f3f662398bc3c1c59af75e70c
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- No
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Not checked yet
- Last checked
- Nov 6, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 50
- Repository last updated
- Oct 5, 2026
- License
- NOASSERTION
Observed agent integration: codex
- ai-agents Repository topic
- ai-skills Repository topic
- anthropic Repository topic
- claude Repository topic
- claude-code Repository topic
- claude-skills Repository topic
- crawl4ai Repository topic
- data-extraction Repository topic
- web-crawling Repository topic
- web-scraping Repository topic
- data Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
.editorconfigasset · 2,630 bytes.gitattributesasset · 1,140 bytes.github/pull_request_template.mddoc · 4,549 bytes.gitignoreasset · 309 bytes.markdownlint-cli2.yamlasset · 2,183 bytesAGENTS.mddoc · 2,764 bytesCHANGELOG.mddoc · 7,171 bytesCONTRIBUTING.mddoc · 2,225 bytesLICENSEasset · 2,034 bytesLICENSE-APACHEasset · 11,284 bytesLICENSE-MITasset · 1,069 bytesREADME.mddoc · 6,930 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesSKILL.mdcore · 16,690 bytesVERSIONasset · 6 bytesevals/README.mddoc · 1,980 bytesevals/eval-01-spa-markdown.mddoc · 2,819 bytesevals/eval-02-extract-products.mddoc · 3,278 bytesevals/eval-03-topic-domain-crawl.mddoc · 3,886 bytesevals/eval-04-render-cached-html.mddoc · 5,126 bytesfixtures/README.mddoc · 1,137 bytesfixtures/sample-products-expected.jsonasset · 225 bytesfixtures/sample-products-schema.jsonasset · 286 bytesfixtures/sample-products.htmlasset · 702 bytesreferences/anti-detection.mddoc · 5,817 bytesreferences/cli-guide.mddoc · 8,398 bytesreferences/complete-sdk-reference.mddoc · 230,893 bytesreferences/content-filters.mddoc · 5,045 bytesreferences/escalation.mddoc · 7,094 bytesreferences/recipes.mddoc · 5,746 bytesreferences/sdk-guide.mddoc · 10,229 bytesreferences/troubleshooting.mddoc · 5,916 bytesreferences/url-discovery.mddoc · 5,348 bytesscripts/basic_crawler.pyruns · 2,284 bytesscripts/batch_crawl.pyruns · 3,736 bytesscripts/batch_extract.pyruns · 3,518 bytesscripts/extract_with_llm.pyruns · 2,746 bytesscripts/extract_with_schema.pyruns · 2,223 bytesscripts/generate_schema.pyruns · 3,103 bytestemplates/browser.ymlasset · 920 bytestemplates/crawler.ymlasset · 1,329 bytestemplates/css_schema.jsonasset · 414 bytestemplates/extract_css.ymlasset · 290 bytestemplates/extract_llm.ymlasset · 636 bytestemplates/filter_bm25.ymlasset · 344 bytestemplates/filter_pruning.ymlasset · 399 bytestemplates/login_crawler.ymlasset · 642 bytestests/README.mddoc · 2,628 bytestests/run_all_tests.pyruns · 1,583 bytestests/test_advanced_patterns.pyruns · 2,400 bytestests/test_basic_crawling.pyruns · 1,761 bytestests/test_data_extraction.pyruns · 2,356 bytestests/test_fixtures.pyruns · 2,420 bytestests/test_markdown_generation.pyruns · 3,290 bytes
No licensed excerpt is available. You can inspect the pinned source.