Not in the top 1,000 by GitHub stars
brettdavies/crawl4ai-skill
Scrape JavaScript-heavy sites and extract structured data via reusable CSS schemas. Portable agent skill wrapping the Crawl4AI CLI and Python SDK.
History
Review weekly changes to SKILL.md before updating. Dates mark the weeks in which we observed each version; checks apply to the recorded commits.
View weekly changes
SKILL.md
Week of Nov 2, 2026 · Line counts unavailable
Before Not checked yet → After Network or broad shell
First observation → 98f71583e27f
No earlier version was observed for comparison.
View on GitHubAdd to my skills
Verified commit 98f71583e27fe07f3f662398bc3c1c59af75e70c
git clone --filter=blob:none --no-checkout https://github.com/brettdavies/crawl4ai-skill.git .hotskills/brettdavies-crawl4ai-skill && git -C .hotskills/brettdavies-crawl4ai-skill checkout --detach 98f71583e27fe07f3f662398bc3c1c59af75e70c
# Copy each selected skill directory from .hotskills/brettdavies-crawl4ai-skill into .agents/skills/<skill-name>/50GitHub stars
1star in 7 days
1skill in this repository
0commits in 30 days
Heat
1.7- Star growth, last 7 daysNot available
- Star growth, last 24 hoursNot available
- Rank movement, last 7 daysNot available
- Commits, last 30 daysNot available
- Accounts asking about it through MCP, last 7 daysNot available
- Hot votes this weekNot available
Heat never changes the order by stars, and sponsorship never touches it. How heat works
Stars over time
30-day change not yet available: not enough history.
Individual skills in this repository
- crawl4ai
/
How it works
crawl4ai
When your agent uses it
Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction. Triggers on crawl4ai, crwl, scrape JS-heavy site, scrape SPA, headless browser scrape, schema-based extraction, batch crawl, sitemap crawl, web data pipeline. SKIP when a static HTML page can be read with `defuddle` / `fetch-web` — those are faster cold-start and don't need a browser.
What it loads into context
Frontmatter
143 tokensAlways available to the agent
SKILL.md
3,736 tokensWhen the skill is activated
Reference documents
76,943 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Overview
- Invoked with a URL argument
- When NOT to use this skill
- When stuck
- Quick Start
- Core Concepts
- Markdown Generation (Primary Use Case)
- Data Extraction
- Advanced Patterns
- Common Use Cases
- Resources
- Best Practices
- Troubleshooting
- License
Scripts executed separately
scripts/basic_crawler.pyscripts/batch_crawl.pyscripts/batch_extract.pyscripts/extract_with_llm.pyscripts/extract_with_schema.pyscripts/generate_schema.pytests/run_all_tests.pytests/test_advanced_patterns.pytests/test_basic_crawling.pytests/test_data_extraction.pytests/test_fixtures.pytests/test_markdown_generation.py
Files
.editorconfigasset2,630 bytes.gitattributesasset1,140 bytes.github/pull_request_template.mddoc4,549 bytes.gitignoreasset309 bytes.markdownlint-cli2.yamlasset2,183 bytesAGENTS.mddoc2,764 bytesCHANGELOG.mddoc7,171 bytesCONTRIBUTING.mddoc2,225 bytesLICENSEasset2,034 bytesLICENSE-APACHEasset11,284 bytesLICENSE-MITasset1,069 bytesREADME.mddoc6,930 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesSKILL.mdcore16,690 bytesVERSIONasset6 bytesevals/README.mddoc1,980 bytesevals/eval-01-spa-markdown.mddoc2,819 bytesevals/eval-02-extract-products.mddoc3,278 bytesevals/eval-03-topic-domain-crawl.mddoc3,886 bytesevals/eval-04-render-cached-html.mddoc5,126 bytesfixtures/README.mddoc1,137 bytesfixtures/sample-products-expected.jsonasset225 bytesfixtures/sample-products-schema.jsonasset286 bytesfixtures/sample-products.htmlasset702 bytesreferences/anti-detection.mddoc5,817 bytesreferences/cli-guide.mddoc8,398 bytesreferences/complete-sdk-reference.mddoc230,893 bytesreferences/content-filters.mddoc5,045 bytesreferences/escalation.mddoc7,094 bytesreferences/recipes.mddoc5,746 bytesreferences/sdk-guide.mddoc10,229 bytesreferences/troubleshooting.mddoc5,916 bytesreferences/url-discovery.mddoc5,348 bytesscripts/basic_crawler.pyruns2,284 bytesscripts/batch_crawl.pyruns3,736 bytesscripts/batch_extract.pyruns3,518 bytesscripts/extract_with_llm.pyruns2,746 bytesscripts/extract_with_schema.pyruns2,223 bytesscripts/generate_schema.pyruns3,103 bytestemplates/browser.ymlasset920 bytestemplates/crawler.ymlasset1,329 bytestemplates/css_schema.jsonasset414 bytestemplates/extract_css.ymlasset290 bytestemplates/extract_llm.ymlasset636 bytestemplates/filter_bm25.ymlasset344 bytestemplates/filter_pruning.ymlasset399 bytestemplates/login_crawler.ymlasset642 bytestests/README.mddoc2,628 bytestests/run_all_tests.pyruns1,583 bytestests/test_advanced_patterns.pyruns2,400 bytestests/test_basic_crawling.pyruns1,761 bytestests/test_data_extraction.pyruns2,356 bytestests/test_fixtures.pyruns2,420 bytestests/test_markdown_generation.pyruns3,290 bytes
No excerpt is shown for this license. Read the source on GitHub.
View the complete source on GitHubSkills included
- crawl4ai
Repository facts
- License
- No license detected
- Created
- Dec 3, 2025
- Last pushed
- Oct 5, 2026
- Forks
- 13
- Open issues
- 0
- Project activity
- Active
Before you install
Network or broad shellAutomated signals, not a guarantee
Checked Nov 6, 2026
Not fully analyzed: some files were too long, unreadable or missing. This never lowers the level.
Show 15 notes
SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedSKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzed
- No broad tool permissions
- 2 dangerous shell commandsin 1 file: references/anti-detection.md:65
- 484 network referencesin 30 files: references/complete-sdk-reference.md:1 references/cli-guide.md:40 SKILL.md:14 and 27 more
- 2 obfuscation signalsin 2 files: references/complete-sdk-reference.md:1427 scripts/basic_crawler.py:55
- No instruction overrides
- 12 executable scriptsin 12 files: scripts/basic_crawler.py:1 scripts/batch_crawl.py:1 scripts/batch_extract.py:1 and 9 more
- 16 references to secretsin 2 files: references/complete-sdk-reference.md:357 templates/extract_llm.yml:10
- No requests to vote in Hot
Show all 516 signals by file
.github/pull_request_template.md
.markdownlint-cli2.yaml
- Network accessline 6
# a stale calver compared to this file is the signal that the per-repo copy has drifted. # # Documentation: https://github.com/DavidAnson/markdownlint-cli2 - Network accessline 7
# # Documentation: https://github.com/DavidAnson/markdownlint-cli2 # Rules: https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md - Network accessline 8
# Documentation: https://github.com/DavidAnson/markdownlint-cli2 - Network accessline 9
# Rules: https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md
AGENTS.md
- Network accessline 2
name: crawl4ai description: Portable agent skill for JS-aware crawling and schema-based extraction (wraps the Crawl4AI CLI and Python SDK). repository: https://github.com/brettdavies/crawl4ai-skill - Network accessline 3
description: Portable agent skill for JS-aware crawling and schema-based extraction (wraps the Crawl4AI CLI and Python SDK). repository: https://github.com/brettdavies/crawl4ai-skill license: MIT OR Apache-2.0 - Network accessline 4
repository: https://github.com/brettdavies/crawl4ai-skill
CHANGELOG.md
CONTRIBUTING.md
- Network accessline 12
Open an issue at <https://github.com/brettdavies/crawl4ai-skill/issues>. Include reproducer steps, the agent host you - Network accessline 35
This bundle wraps Crawl4AI for agent hosts. Upstream library bugs belong at the [Crawl4AI repo](https://github.com/unclecode/crawl4ai), not here. Open issues here for: bundle layout, SKILL.md routing, - Network accessline 36
[Crawl4AI repo](https://github.com/unclecode/crawl4ai), not here. Open issues here for: bundle layout, SKILL.md routing,
LICENSE
LICENSE-APACHE
README.md
- Network accessline 3
Scrape JavaScript-heavy sites and extract structured data via reusable CSS schemas. A portable agent skill that wraps the [Crawl4AI](https://crawl4ai.com/) CLI and Python SDK, written in the Anthropic SKILL.md format and consumable by any - Network accessline 4
the [Crawl4AI](https://crawl4ai.com/) CLI and Python SDK, written in the Anthropic SKILL.md format and consumable by any - Network accessline 24
# Claude Code git clone https://github.com/brettdavies/crawl4ai-skill.git ~/.claude/skills/crawl4ai - Network accessline 25
git clone https://github.com/brettdavies/crawl4ai-skill.git ~/.claude/skills/crawl4ai - Network accessline 49
crwl https://example.com -c "wait_until=networkidle,page_timeout=60000" -o markdown - Network accessline 50
crwl https://example.com -o json -v --bypass-cache - Network accessline 59
async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com") - Network accessline 60
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com") print(result.markdown[:500]) - Network accessline 61
result = await crawler.arun("https://example.com") - Network accessline 99
crwl https://docs.example.com -o markdown > docs.md - Network accessline 105
# Derive the schema once (uses LLM) ./scripts/generate_schema.py https://shop.example.com "products with name, price, image" shop_schema.json - Network accessline 106
./scripts/generate_schema.py https://shop.example.com "products with name, price, image" shop_schema.json - Network accessline 108
# Apply the saved schema (no LLM cost per request) ./scripts/extract_with_schema.py https://shop.example.com shop_schema.json products.json - Network accessline 109
./scripts/extract_with_schema.py https://shop.example.com shop_schema.json products.json - Network accessline 115
for url in news1.com news2.com news3.com; do crwl "https://$url" -f templates/filter_bm25.yml -o markdown-fit - Network accessline 116
crwl "https://$url" -f templates/filter_bm25.yml -o markdown-fit
SKILL.md
- Network accessline 14
Crawl4AI wraps a headless browser (Playwright) plus a markdown-aware content pipeline. Use it when defuddle/curl can't - Network accessline 73
# Basic crawling - returns markdown crwl https://example.com - Network accessline 74
crwl https://example.com - Network accessline 76
# Get markdown output crwl https://example.com -o markdown - Network accessline 77
crwl https://example.com -o markdown - Network accessline 79
# JSON output with cache bypass crwl https://example.com -o json -v --bypass-cache - Network accessline 80
crwl https://example.com -o json -v --bypass-cache - Network accessline 92
async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com") - Network accessline 93
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com") print(result.markdown[:500]) - Network accessline 94
result = await crawler.arun("https://example.com") - Network accessline 156
crwl https://docs.example.com -o markdown # raw markdown - Network accessline 157
crwl https://docs.example.com -o markdown-fit # filtered (noise removed) - Network accessline 158
crwl https://docs.example.com -f templates/filter_bm25.yml -o markdown-fit # BM25-relevance filter - Network accessline 159
crwl https://docs.example.com -f templates/filter_pruning.yml -o markdown-fit # quality-based filter - Network accessline 193
./scripts/generate_schema.py https://shop.example.com "products with name, price, image" shop_schema.json - Network accessline 194
./scripts/extract_with_schema.py https://shop.example.com shop_schema.json products.json - Network accessline 200
crwl https://shop.example.com -e templates/extract_css.yml -s shop_schema.json -o json - Network accessline 211
./scripts/extract_with_llm.py https://news.example.com "Extract headlines, dates, summaries" news.json - Network accessline 217
crwl https://news.example.com -e templates/extract_llm.yml -o json - Network accessline 232
crwl https://example.com -c "wait_for=css:.ajax-content,scan_full_page=true,page_timeout=60000" - Network accessline 233
crwl https://example.com -C templates/crawler.yml # all options in a YAML file - Network accessline 249
urls = ["https://site1.com", "https://site2.com", "https://site3.com"] - Network accessline 277
crwl https://site.com/login -C templates/login_crawler.yml - Network accessline 278
crwl https://site.com/protected -c "session_id=user_session" - Network accessline 289
crwl https://example.com -B templates/browser.yml - Network accessline 306
result = await crawler.arun( url="raw:" + html_string, config=CrawlerRunConfig(base_url="https://example.com", screenshot=True, pdf=True), - Network accessline 307
url="raw:" + html_string, config=CrawlerRunConfig(base_url="https://example.com", screenshot=True, pdf=True), ) - Network accessline 308
config=CrawlerRunConfig(base_url="https://example.com", screenshot=True, pdf=True),
evals/eval-01-spa-markdown.md
- Network accessline 7
A user comes to you with this URL: `https://app.example.com/dashboard`. They say:
evals/eval-02-extract-products.md
- Network accessline 7
A user comes to you with this URL: `https://shop.example.com/category/widgets`. They say:
evals/eval-04-render-cached-html.md
- Network accessline 10
> need a screenshot of how it would render. I don't want to make another network > fetch. The page uses relative image URLs against > `https://example.com/` for resolution. And I need the **full page**, not just - Network accessline 11
> fetch. The page uses relative image URLs against > `https://example.com/` for resolution. And I need the **full page**, not just > what fits in the default viewport — it's a long article." - Network accessline 12
> `https://example.com/` for resolution. And I need the **full page**, not just - Network accessline 44
1. **No-network choice** — Did you choose a code path that does NOT fetch `https://example.com/` again? (0 if you - Network accessline 46
2. **`raw:` vs `file://`** — Did you pick `raw:` (HTML in hand as a string) over `file://` (you'd have to write the HTML to disk first)? (0 for unnecessary disk I/O; 10 for `raw:` direct.) 3. **`base_url` set** — Did you set `base_url="h - Network accessline 47
to disk first)? (0 for unnecessary disk I/O; 10 for `raw:` direct.) 3. **`base_url` set** — Did you set `base_url="https://example.com/"` so the relative image URLs in the HTML resolve correctly? (0 if you forgot; 10 if set and explained - Network accessline 48
3. **`base_url` set** — Did you set `base_url="https://example.com/"` so the relative image URLs in the HTML resolve
references/anti-detection.md
- Network accessline 29
crwl https://example.com -B templates/browser.yml - Dangerous shell operationline 65
The bundled `crawl4ai-setup` warns about Patchright install when it can't run `apt-get` (needs sudo for OS-level - Dangerous shell operationline 69
sudo "$(uv tool dir)/crawl4ai/bin/python3" -m patchright install --with-deps - Network accessline 82
headless=True, proxy_config={ "server": "http://proxy.example.com:8080", - Network accessline 83
proxy_config={ "server": "http://proxy.example.com:8080", "username": "user", - Network accessline 84
"server": "http://proxy.example.com:8080", - Network accessline 94
# templates/browser.yml proxy_config: server: "http://proxy.example.com:8080" - Network accessline 95
proxy_config: server: "http://proxy.example.com:8080" username: "user" - Network accessline 96
server: "http://proxy.example.com:8080" - Network accessline 133
crwl https://example.com -B templates/browser.yml -o all -v
references/cli-guide.md
- Network accessline 40
# Basic crawling - returns markdown crwl https://example.com - Network accessline 41
crwl https://example.com - Network accessline 43
# Specify output format crwl https://example.com -o markdown - Network accessline 44
crwl https://example.com -o markdown - Network accessline 46
# Verbose JSON output with cache bypass crwl https://example.com -o json -v --bypass-cache - Network accessline 47
crwl https://example.com -o json -v --bypass-cache - Network accessline 56
# Extract structured data using CSS schema crwl "https://www.infoq.com/ai-ml-data-eng/" \ - Network accessline 57
crwl "https://www.infoq.com/ai-ml-data-eng/" \ - Network accessline 81
# Using config file crwl https://example.com -B browser.yml - Network accessline 82
crwl https://example.com -B browser.yml - Network accessline 84
# Using direct parameters crwl https://example.com -b "headless=true,viewport_width=1280,user_agent_mode=random" - Network accessline 85
crwl https://example.com -b "headless=true,viewport_width=1280,user_agent_mode=random" - Network accessline 120
# Using config file crwl https://example.com -C crawler.yml - Network accessline 121
crwl https://example.com -C crawler.yml - Network accessline 123
# Using direct parameters crwl https://example.com -c "css_selector=#main,delay_before_return_html=2,scan_full_page=true" - Network accessline 124
crwl https://example.com -c "css_selector=#main,delay_before_return_html=2,scan_full_page=true" - Network accessline 198
# Simple question crwl https://example.com -q "What is the main topic discussed?" - Network accessline 199
crwl https://example.com -q "What is the main topic discussed?" - Network accessline 201
# View content then ask questions crwl https://example.com -o markdown # See content first - Network accessline 202
crwl https://example.com -o markdown # See content first - Network accessline 203
crwl https://example.com -q "Summarize the key points" - Network accessline 204
crwl https://example.com -q "What are the conclusions?" - Network accessline 206
# Combined with advanced crawling crwl https://example.com \ - Network accessline 207
crwl https://example.com \ - Network accessline 216
- Saves configuration in `~/.crawl4ai/global.yml` - Any LiteLLM-supported provider works; pick a current model identifier from [LiteLLM Providers](https://docs.litellm.ai/docs/providers). `ollama/*` providers need no token. - Network accessline 218
[LiteLLM Providers](https://docs.litellm.ai/docs/providers). `ollama/*` providers need no token. - Network accessline 223
# CSS-based extraction crwl https://example.com \ - Network accessline 224
crwl https://example.com \ - Network accessline 229
# LLM-based extraction (schema lives inside extract_llm.yml; no -s needed) crwl https://example.com \ - Network accessline 230
crwl https://example.com \ - Network accessline 252
crwl https://example.com -f filter_bm25.yml -o markdown-fit - Network accessline 275
crwl https://example.com \ - Network accessline 284
crwl https://example.com \ - Network accessline 294
crwl https://example.com \ - Network accessline 304
# First crawl and view crwl https://example.com -o markdown - Network accessline 305
crwl https://example.com -o markdown - Network accessline 307
# Then ask questions crwl https://example.com -q "What are the main points?" - Network accessline 308
crwl https://example.com -q "What are the main points?" - Network accessline 309
crwl https://example.com -q "Summarize the conclusions"
references/complete-sdk-reference.md
- Network accessline 1
<!-- Source: https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/complete-sdk-reference.md - Network accessline 2
Source: https://github.com/unclecode/crawl4ai/blob/main/docs/md_v2/complete-sdk-reference.md Pinned to upstream commit 3a75dd3 (2026-03-07). Pull a fresh copy with: curl -sL https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/m - Network accessline 3
Pinned to upstream commit 3a75dd3 (2026-03-07). Pull a fresh copy with: curl -sL https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/md_v2/complete-sdk-reference.md \ -o crawl4ai/references/complete-sdk-reference.md - Network accessline 4
curl -sL https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/md_v2/complete-sdk-reference.md \ - Network accessline 77
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://www.example.com", - Network accessline 78
result = await crawler.arun( url="https://www.example.com", ) - Network accessline 79
url="https://www.example.com", - Network accessline 127
You can then make POST requests to `http://localhost:11235/crawl` to perform crawls. **Production usage** is discouraged until our new Docker approach is ready (planned in Jan or Feb 2025). - Network accessline 160
async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com") - Network accessline 161
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com") print(result.markdown[:300]) # Print first 300 chars - Network accessline 162
result = await crawler.arun("https://example.com") - Network accessline 169
- **`AsyncWebCrawler`** launches a headless browser (Chromium by default). - It fetches `https://example.com`. - Network accessline 170
- It fetches `https://example.com`. - Network accessline 188
async with AsyncWebCrawler(config=browser_conf) as crawler: result = await crawler.arun( url="https://example.com", - Network accessline 189
result = await crawler.arun( url="https://example.com", config=run_conf - Network accessline 190
url="https://example.com", - Network accessline 223
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://news.ycombinator.com", config=config) - Network accessline 224
result = await crawler.arun("https://news.ycombinator.com", config=config) - Network accessline 272
raw_html = "<div class='item'><h2>Item 1</h2><a href='https://example.com/item1'>Link 1</a></div>" - Network accessline 347
async with AsyncWebCrawler(config=browser_config) as crawler: result = await crawler.arun( url="https://openai.com/api/pricing/", config=crawler_config - Network accessline 348
result = await crawler.arun( url="https://openai.com/api/pricing/", config=crawler_config ) - Network accessline 349
url="https://openai.com/api/pricing/", config=crawler_config - Access to secretsline 357
[Credential-related source omitted] - Network accessline 374
# Start adaptive crawling result = await adaptive.digest( start_url="https://docs.python.org/3/", - Network accessline 375
result = await adaptive.digest( start_url="https://docs.python.org/3/", query="async context managers" - Network accessline 376
start_url="https://docs.python.org/3/", - Network accessline 401
async def quick_parallel_example(): urls = [ "https://example.com/page1", - Network accessline 402
urls = [ "https://example.com/page1", "https://example.com/page2", - Network accessline 403
"https://example.com/page1", - Network accessline 404
"https://example.com/page2", - Network accessline 405
"https://example.com/page3" - Network accessline 500
async with AsyncWebCrawler(config=browser_config) as crawler: result = await crawler.arun( url="https://www.kidocode.com/degrees/technology", config=crawler_config - Network accessline 501
result = await crawler.arun( url="https://www.kidocode.com/degrees/technology", config=crawler_config ) - Network accessline 502
url="https://www.kidocode.com/degrees/technology", config=crawler_config - Network accessline 594
async with AsyncWebCrawler(config=browser_cfg) as crawler: result = await crawler.arun("https://example.com") - Network accessline 595
result = await crawler.arun("https://example.com") - Network accessline 604
crawler = AsyncWebCrawler(config=browser_cfg) await crawler.start() result1 = await crawler.arun("https://example.com") - Network accessline 605
await crawler.start() result1 = await crawler.arun("https://example.com") result2 = await crawler.arun("https://another.com") - Network accessline 606
result1 = await crawler.arun("https://example.com") - Network accessline 607
result2 = await crawler.arun("https://another.com") - Network accessline 635
screenshot=True async with AsyncWebCrawler(config=browser_cfg) as crawler: result = await crawler.arun("https://example.com/news", config=run_cfg) - Network accessline 637
result = await crawler.arun("https://example.com/news", config=run_cfg) - Network accessline 739
async with AsyncWebCrawler(config=browser_cfg) as crawler: result = await crawler.arun( url="https://example.com/blog", - Network accessline 740
result = await crawler.arun( url="https://example.com/blog", config=run_cfg - Network accessline 741
url="https://example.com/blog", - Network accessline 784
await crawler.arun( url="https://example.com", - Network accessline 785
url="https://example.com", - Network accessline 805
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://example.com", - Network accessline 806
result = await crawler.arun( url="https://example.com", config=run_config - Network accessline 807
url="https://example.com", - Network accessline 1022
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com/posts", config=run_config) - Network accessline 1023
result = await crawler.arun("https://example.com/posts", config=run_config) - Network accessline 1101
# Minimal usage: The default dispatcher will be used results = await crawler.arun_many( urls=["https://site1.com", "https://site2.com"], - Network accessline 1102
results = await crawler.arun_many( urls=["https://site1.com", "https://site2.com"], config=CrawlerRunConfig(stream=False) # Default behavior - Network accessline 1103
urls=["https://site1.com", "https://site2.com"], - Network accessline 1122
# Process results as they complete async for result in await crawler.arun_many( urls=["https://site1.com", "https://site2.com", "https://site3.com"], - Network accessline 1123
async for result in await crawler.arun_many( urls=["https://site1.com", "https://site2.com", "https://site3.com"], config=config - Network accessline 1124
urls=["https://site1.com", "https://site2.com", "https://site3.com"], - Network accessline 1139
) results = await crawler.arun_many( urls=["https://site1.com", "https://site2.com", "https://site3.com"], - Network accessline 1140
results = await crawler.arun_many( urls=["https://site1.com", "https://site2.com", "https://site3.com"], config=my_run_config, - Network accessline 1141
urls=["https://site1.com", "https://site2.com", "https://site3.com"], - Network accessline 1188
results = await crawler.arun_many( urls=[ "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf", # → pdf_config - Network accessline 1189
urls=[ "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf", # → pdf_config "https://blog.python.org/", # → blog_config - Network accessline 1190
"https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf", # → pdf_config - Network accessline 1191
"https://blog.python.org/", # → blog_config - Network accessline 1192
"https://github.com/microsoft/playwright", # → github_config - Network accessline 1193
"https://httpbin.org/json", # → api_config - Network accessline 1194
"https://example.com/" # → default_config - Network accessline 1259
print(result.url) # e.g., "https://example.com/" - Possible obfuscationline 1427
f.write(base64.b64decode(result.screenshot)) - Network accessline 1663
{ "server": "http://proxy.example.com:8080", - Network accessline 1664
"server": "http://proxy.example.com:8080", - Access to secretsline 1678
[Credential-related source omitted] - Network accessline 1728
async with AsyncWebCrawler(config=browser_conf) as crawler: result = await crawler.arun("https://example.com") - Network accessline 1729
result = await crawler.arun("https://example.com") - Access to secretsline 1856
[Credential-related source omitted] - Access to secretsline 1858
[Credential-related source omitted] - Access to secretsline 1871
[Credential-related source omitted] - Access to secretsline 1910
[Credential-related source omitted] - Network accessline 1944
async with AsyncWebCrawler(config=browser_conf) as crawler: # 4) Execute the crawl result = await crawler.arun(url="https://example.com/news", config=run_conf) - Network accessline 1946
result = await crawler.arun(url="https://example.com/news", config=run_conf) - Network accessline 1976
viewport_width=1280, viewport_height=720, proxy_config="http://user:pass@proxy:8080", - Network accessline 1977
viewport_height=720, proxy_config="http://user:pass@proxy:8080", user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36", - Network accessline 1978
proxy_config="http://user:pass@proxy:8080", - Access to secretsline 1993
[Credential-related source omitted] - Network accessline 2192
# Combined patterns and functions with AND logic secure_docs = CrawlerRunConfig( url_matcher=["https://*", lambda url: '.doc' in url], - Network accessline 2193
secure_docs = CrawlerRunConfig( url_matcher=["https://*", lambda url: '.doc' in url], match_mode=MatchMode.AND # Must be HTTPS AND contain .doc - Network accessline 2194
url_matcher=["https://*", lambda url: '.doc' in url], - Network accessline 2204
- **None (default)**: When `url_matcher` is None or not set, the config matches ALL URLs - **String patterns**: Glob-style patterns like `"*.pdf"`, `"*/api/*"`, `"https://*.example.com/*"` - Network accessline 2205
- **String patterns**: Glob-style patterns like `"*.pdf"`, `"*/api/*"`, `"https://*.example.com/*"` - Network accessline 2240
viewport_width=1280, viewport_height=720, proxy_config="http://user:pass@myproxy:8080", - Network accessline 2241
viewport_height=720, proxy_config="http://user:pass@myproxy:8080", text_mode=True - Network accessline 2242
proxy_config="http://user:pass@myproxy:8080", - Network accessline 2258
async with AsyncWebCrawler(config=browser_cfg) as crawler: result = await crawler.arun( url="https://example.com/news", - Network accessline 2259
result = await crawler.arun( url="https://example.com/news", config=run_cfg - Network accessline 2260
url="https://example.com/news", - Access to secretsline 2300
[Credential-related source omitted] - Access to secretsline 2306
[Credential-related source omitted] - Network accessline 2341
async with AsyncWebCrawler(config=browser_config) as crawler: result = await crawler.arun( url="https://example.com", - Network accessline 2342
result = await crawler.arun( url="https://example.com", config=run_config - Network accessline 2343
url="https://example.com", - Network accessline 2364
result = await crawler.arun( url="https://example.com", - Network accessline 2365
url="https://example.com", - Network accessline 2396
result = await crawler.arun( url="https://example.com", - Network accessline 2397
url="https://example.com", - Network accessline 2405
run_config = CrawlerRunConfig() result = await crawler.arun(url="https://example.com", config=run_config) - Network accessline 2406
result = await crawler.arun(url="https://example.com", config=run_config) - Network accessline 2420
async with AsyncWebCrawler(config=browser_config) as crawler: run_config = CrawlerRunConfig() result = await crawler.arun(url="https://example.com", config=run_config) - Network accessline 2422
result = await crawler.arun(url="https://example.com", config=run_config) - Network accessline 2448
async with AsyncWebCrawler(config=browser_config) as crawler: result = await crawler.arun( url="https://example.com", - Network accessline 2449
result = await crawler.arun( url="https://example.com", config=run_config - Network accessline 2450
url="https://example.com", - Network accessline 2492
) async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com", config=config) - Network accessline 2494
result = await crawler.arun("https://example.com", config=config) - Network accessline 2545
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com/docs", config=config) - Network accessline 2546
result = await crawler.arun("https://example.com/docs", config=config) - Network accessline 2598
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com", config=config) - Network accessline 2599
result = await crawler.arun("https://example.com", config=config) - Network accessline 2704
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com", config=config) - Network accessline 2705
result = await crawler.arun("https://example.com", config=config) - Network accessline 2761
) async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://news.example.com/tech", config=config) - Network accessline 2762
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://news.example.com/tech", config=config) if result.success: - Network accessline 2763
result = await crawler.arun("https://news.example.com/tech", config=config) - Network accessline 2820
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com/tech-article", config=config) - Network accessline 2821
result = await crawler.arun("https://example.com/tech-article", config=config) - Network accessline 2941
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://news.ycombinator.com", - Network accessline 2942
result = await crawler.arun( url="https://news.ycombinator.com", config=config - Network accessline 2943
url="https://news.ycombinator.com", - Network accessline 2994
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://news.ycombinator.com", - Network accessline 2995
result = await crawler.arun( url="https://news.ycombinator.com", config=config - Network accessline 2996
url="https://news.ycombinator.com", - Network accessline 3118
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://news.ycombinator.com/newest", - Network accessline 3119
result = await crawler.arun( url="https://news.ycombinator.com/newest", config=config - Network accessline 3120
url="https://news.ycombinator.com/newest", - Network accessline 3144
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://example.com/blog-post", - Network accessline 3145
result = await crawler.arun( url="https://example.com/blog-post", config=config - Network accessline 3146
url="https://example.com/blog-post", - Network accessline 3225
async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://news.ycombinator.com", config=config) - Network accessline 3226
result = await crawler.arun(url="https://news.ycombinator.com", config=config) - Network accessline 3254
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://example.org/iframe-demo", - Network accessline 3255
result = await crawler.arun( url="https://example.org/iframe-demo", config=config - Network accessline 3256
url="https://example.org/iframe-demo", - Network accessline 3287
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://store.boschrexroth.com/en/us/p/hydraulic-cylinder-r900999011", - Network accessline 3288
result = await crawler.arun( url="https://store.boschrexroth.com/en/us/p/hydraulic-cylinder-r900999011", config=config, - Network accessline 3289
url="https://store.boschrexroth.com/en/us/p/hydraulic-cylinder-r900999011", - Network accessline 3308
For a full runnable example, see [`shadow_dom_crawling.py`](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/shadow_dom_crawling.py). - Network accessline 3351
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://news.ycombinator.com/newest", - Network accessline 3352
result = await crawler.arun( url="https://news.ycombinator.com/newest", config=config - Network accessline 3353
url="https://news.ycombinator.com/newest", - Access to secretsline 3378
[Credential-related source omitted] - Network accessline 3390
async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://news.ycombinator.com", config=config) - Network accessline 3391
result = await crawler.arun(url="https://news.ycombinator.com", config=config) - Network accessline 3453
async def main(): articles = await extract_main_articles("https://news.ycombinator.com/newest") - Network accessline 3454
articles = await extract_main_articles("https://news.ycombinator.com/newest") - Network accessline 3483
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://example.com", - Network accessline 3484
result = await crawler.arun( url="https://example.com", config=config - Network accessline 3485
url="https://example.com", - Network accessline 3502
images=[ # List of images found MediaItem( src="https://example.com/image.jpg", - Network accessline 3503
MediaItem( src="https://example.com/image.jpg", alt="Image description", - Network accessline 3504
src="https://example.com/image.jpg", - Network accessline 3518
internal=[ # List of internal links Link( href="https://example.com/page", - Network accessline 3519
Link( href="https://example.com/page", text="Link text", - Network accessline 3520
href="https://example.com/page", - Network accessline 3578
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://example.com/article", - Network accessline 3579
result = await crawler.arun( url="https://example.com/article", config=config - Network accessline 3580
url="https://example.com/article", - Network accessline 3630
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://news.ycombinator.com", # Example site - Network accessline 3631
result = await crawler.arun( url="https://news.ycombinator.com", # Example site config=config - Network accessline 3632
url="https://news.ycombinator.com", # Example site - Network accessline 3645
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://news.ycombinator.com", # Another pass - Network accessline 3646
result = await crawler.arun( url="https://news.ycombinator.com", # Another pass config=config - Network accessline 3647
url="https://news.ycombinator.com", # Another pass - Network accessline 3675
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://news.ycombinator.com", - Network accessline 3676
result = await crawler.arun( url="https://news.ycombinator.com", config=config - Network accessline 3677
url="https://news.ycombinator.com", - Network accessline 3718
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://news.ycombinator.com", - Network accessline 3719
result = await crawler.arun( url="https://news.ycombinator.com", config=config - Network accessline 3720
url="https://news.ycombinator.com", - Network accessline 3742
# Re-use the same crawler session result2 = await crawler.arun( url="https://news.ycombinator.com", # same URL but continuing session - Network accessline 3743
result2 = await crawler.arun( url="https://news.ycombinator.com", # same URL but continuing session config=next_page_conf - Network accessline 3744
url="https://news.ycombinator.com", # same URL but continuing session - Network accessline 3770
wait_for="css:.commit" ) result = await crawler.arun(url="https://github.com/search", config=config) - Network accessline 3772
result = await crawler.arun(url="https://github.com/search", config=config) - Network accessline 3816
async with AsyncWebCrawler(config=browser_cfg) as crawler: result = await crawler.arun( url="https://github.com/microsoft/TypeScript/commits/main", - Network accessline 3817
result = await crawler.arun( url="https://github.com/microsoft/TypeScript/commits/main", config=config1 - Network accessline 3818
url="https://github.com/microsoft/TypeScript/commits/main", - Network accessline 3849
) result2 = await crawler.arun( url="https://github.com/microsoft/TypeScript/commits/main", - Network accessline 3850
result2 = await crawler.arun( url="https://github.com/microsoft/TypeScript/commits/main", config=config_next - Network accessline 3851
url="https://github.com/microsoft/TypeScript/commits/main", - Network accessline 3933
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://twitter.com/search?q=AI", - Network accessline 3934
result = await crawler.arun( url="https://twitter.com/search?q=AI", config=config - Network accessline 3935
url="https://twitter.com/search?q=AI", - Network accessline 3983
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://www.example.com") - Network accessline 3984
result = await crawler.arun("https://www.example.com") - Network accessline 4002
"internal": [ { "href": "https://kidocode.com/", - Network accessline 4003
{ "href": "https://kidocode.com/", "text": "", - Network accessline 4004
"href": "https://kidocode.com/", - Network accessline 4008
}, { "href": "https://kidocode.com/degrees/technology", - Network accessline 4009
{ "href": "https://kidocode.com/degrees/technology", "text": "Technology Degree", - Network accessline 4010
"href": "https://kidocode.com/degrees/technology", - Network accessline 4070
async with AsyncWebCrawler() as crawler: # Crawl a documentation site (great for testing) result = await crawler.arun("https://docs.python.org/3/", config=config) - Network accessline 4072
result = await crawler.arun("https://docs.python.org/3/", config=config) - Network accessline 4130
✅ Successfully crawled: https://docs.python.org/3/ - Network accessline 4138
1. https://docs.python.org/3.15/ - Network accessline 4232
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://scikit-learn.org/", config=config) - Network accessline 4233
result = await crawler.arun("https://scikit-learn.org/", config=config) - Network accessline 4260
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://docs.example-api.com/", config=config) - Network accessline 4261
result = await crawler.arun("https://docs.example-api.com/", config=config) - Network accessline 4293
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://your-website.com/", config=config) - Network accessline 4294
result = await crawler.arun("https://your-website.com/", config=config) - Network accessline 4378
async with AsyncWebCrawler() as crawler: result = await crawler.arun( "https://www.example.com", - Network accessline 4379
result = await crawler.arun( "https://www.example.com", config=crawler_cfg - Network accessline 4380
"https://www.example.com", - Network accessline 4426
"images": [ { "src": "https://cdn.prod.website-files.com/.../Group%2089.svg", - Network accessline 4427
{ "src": "https://cdn.prod.website-files.com/.../Group%2089.svg", "alt": "coding school for kids", - Network accessline 4428
"src": "https://cdn.prod.website-files.com/.../Group%2089.svg", - Network accessline 4481
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com", config=crawler_cfg) - Network accessline 4482
result = await crawler.arun("https://example.com", config=crawler_cfg) - Network accessline 4519
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://www.example.com", config=crawler_cfg) - Network accessline 4520
result = await crawler.arun("https://www.example.com", config=crawler_cfg) - Network accessline 4610
# 4. Run the crawl and extraction result = await crawler.arun( url="https://example.com/crypto-prices", - Network accessline 4612
url="https://example.com/crypto-prices", - Network accessline 4715
https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html - Network accessline 4842
async with AsyncWebCrawler(verbose=True) as crawler: result = await crawler.arun( url="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html", - Network accessline 4843
result = await crawler.arun( url="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html", extraction_strategy=strategy, - Network accessline 4844
url="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html", - Network accessline 4891
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://example.com", - Network accessline 4892
result = await crawler.arun( url="https://example.com", config=config - Network accessline 4893
url="https://example.com", - Network accessline 4968
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://www.example.com/products", - Network accessline 4969
result = await crawler.arun( url="https://www.example.com/products", config=config - Network accessline 4970
url="https://www.example.com/products", - Access to secretsline 5010
[Credential-related source omitted] - Network accessline 5013
# Get sample HTML for context async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com/products") - Network accessline 5014
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com/products") html = result.markdown.fit_html - Network accessline 5015
result = await crawler.arun("https://example.com/products") - Network accessline 5033
async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://example.com/products", - Network accessline 5034
result = await crawler.arun( url="https://example.com/products", config=config - Network accessline 5035
url="https://example.com/products", - Network accessline 5057
[ { "url": "https://example.com", - Network accessline 5058
{ "url": "https://example.com", "label": "email", - Network accessline 5059
"url": "https://example.com", - Network accessline 5063
}, { "url": "https://example.com", - Network accessline 5064
{ "url": "https://example.com", "label": "url", - Network accessline 5065
"url": "https://example.com", - Network accessline 5066
"label": "url", "value": "https://support.example.com", "span": [210, 235] - Network accessline 5067
"value": "https://support.example.com", - Network accessline 5178
usage = TokenUsage() schema = JsonCssExtractionStrategy.generate_schema( url="https://example.com/products", - Network accessline 5179
schema = JsonCssExtractionStrategy.generate_schema( url="https://example.com/products", query="extract product name and price", - Network accessline 5180
url="https://example.com/products", - Access to secretsline 5195
[Credential-related source omitted] - Access to secretsline 5229
[Credential-related source omitted] - Access to secretsline 5305
[Credential-related source omitted] - Network accessline 5326
# 4. Let's say we want to crawl a single page result = await crawler.arun( url="https://example.com/products", - Network accessline 5327
result = await crawler.arun( url="https://example.com/products", config=crawl_config - Network accessline 5328
url="https://example.com/products", - Access to secretsline 5417
[Credential-related source omitted] - Network accessline 5432
async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler: # Example page url = "https://www.nbcnews.com/business" - Network accessline 5433
# Example page url = "https://www.nbcnews.com/business" result = await crawler.arun(url=url, config=crawl_config) - Network accessline 5434
url = "https://www.nbcnews.com/business" - Network accessline 5490
# Define configurations config1 = CrawlerRunConfig( url="https://example.com/page1", session_id=session_id - Network accessline 5491
config1 = CrawlerRunConfig( url="https://example.com/page1", session_id=session_id ) - Network accessline 5492
url="https://example.com/page1", session_id=session_id - Network accessline 5493
) config2 = CrawlerRunConfig( url="https://example.com/page2", session_id=session_id - Network accessline 5494
config2 = CrawlerRunConfig( url="https://example.com/page2", session_id=session_id ) - Network accessline 5495
url="https://example.com/page2", session_id=session_id - Network accessline 5513
async def crawl_dynamic_content(): url = "https://github.com/microsoft/TypeScript/commits/main" - Network accessline 5514
url = "https://github.com/microsoft/TypeScript/commits/main" - Network accessline 5592
async with AsyncWebCrawler() as crawler: session_id = "dynamic_content_session" url = "https://example.com/dynamic-content" - Network accessline 5594
url = "https://example.com/dynamic-content" - Network accessline 5637
async with AsyncWebCrawler() as crawler: session_id = "commit_session" url = "https://github.com/example/repo/commits/main" - Network accessline 5638
session_id = "commit_session" url = "https://github.com/example/repo/commits/main" crawler.crawler_strategy.set_hook("on_execution_started", on_execution_started) - Network accessline 5639
url = "https://github.com/example/repo/commits/main" - Network accessline 5666
async with AsyncWebCrawler() as crawler: session_id = "integrated_session" url = "https://github.com/example/repo/commits/main" - Network accessline 5668
url = "https://github.com/example/repo/commits/main" - Network accessline 5770
# Example 2: (Optional) Simulate a login scenario # (We do NOT create or close pages here, just do quick steps if needed) # e.g., await page.goto("https://example.com/login") - Network accessline 5771
# (We do NOT create or close pages here, just do quick steps if needed) # e.g., await page.goto("https://example.com/login") # e.g., await page.fill("input[name='username']", "testuser") - Network accessline 5772
# e.g., await page.goto("https://example.com/login") - Network accessline 5861
# 4) Run the crawler on an example page url = "https://example.com" - Network accessline 5862
url = "https://example.com" - Network accessline 5933
async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://example.com") - Network accessline 5934
result = await crawler.arun(url="https://example.com") - Network accessline 5953
async with AsyncWebCrawler(config=browser_config) as crawler: result = await crawler.arun( url="https://example.com", - Network accessline 5954
result = await crawler.arun( url="https://example.com", config=crawler_config, - Network accessline 5955
url="https://example.com", - Network accessline 5964
urls = ["https://example.com/1", "https://example.com/2"]
references/content-filters.md
references/escalation.md
- Network accessline 41
reference is 5900+ lines; jump to the anchor (e.g. `#extraction-strategies`, `#content-processing`) rather than scanning linearly. 3. **Upstream documentation** — `https://docs.crawl4ai.com/` for current docs. Confirm the installed versi - Network accessline 42
scanning linearly. 3. **Upstream documentation** — `https://docs.crawl4ai.com/` for current docs. Confirm the installed version with `python -c "import importlib.metadata; print(importlib.metadata.version('crawl4ai'))"` (or `crwl --versi - Network accessline 43
3. **Upstream documentation** — `https://docs.crawl4ai.com/` for current docs. Confirm the installed version with - Network accessline 44
`python -c "import importlib.metadata; print(importlib.metadata.version('crawl4ai'))"` (or `crwl --version`) and check against the skill's [`VERSION`](../VERSION) file if surface names look wrong. 4. **Upstream source + issues** — `https - Network accessline 45
check against the skill's [`VERSION`](../VERSION) file if surface names look wrong. 4. **Upstream source + issues** — `https://github.com/unclecode/crawl4ai`. Search `Issues` for the error message verbatim before re-deriving a fix. The c - Network accessline 46
4. **Upstream source + issues** — `https://github.com/unclecode/crawl4ai`. Search `Issues` for the error message - Network accessline 78
Symptom: `crwl https://app.example.com -c "wait_for=css:.results"` times out at 30s, but the page renders fine in a - Network accessline 81
What I did: queried `qmd query --collection solutions "playwright wait_for selector timeout shadow DOM"` (no hit), then re-ran `crwl https://app.example.com -o html | rg -i "results" | head` to confirm the selector exists in raw HTML. It - Network accessline 82
re-ran `crwl https://app.example.com -o html | rg -i "results" | head` to confirm the selector exists in raw HTML. It - Network accessline 100
Symptom: a blog post says `crwl --depth 3 https://example.com` for recursive crawling, but `crwl --help` shows no - Network accessline 103
What I did: ran `crwl --version` to confirm installed version, then re-ran the query against the upstream README (`https://github.com/unclecode/crawl4ai`) for the actual recursive-crawl entry point. Recursive crawling is an SDK-only - Network accessline 104
(`https://github.com/unclecode/crawl4ai`) for the actual recursive-crawl entry point. Recursive crawling is an SDK-only
references/recipes.md
- Network accessline 11
crwl https://docs.example.com/guide -o markdown > guide.md - Network accessline 17
crwl https://docs.example.com/guide -f templates/filter_pruning.yml -o markdown-fit > guide.md - Network accessline 28
crwl https://app.example.com -c "wait_for=css:.results,page_timeout=60000" -o markdown - Network accessline 34
crwl https://app.example.com -c "wait_for=js:document.querySelector('.results') !== null,page_timeout=60000" - Network accessline 45
# 1. Derive a schema from one URL (one LLM call) ./scripts/generate_schema.py https://shop.example.com/category "products with name, price, link" shop_schema.json - Network accessline 46
./scripts/generate_schema.py https://shop.example.com/category "products with name, price, link" shop_schema.json - Network accessline 48
# 2. Sanity check the schema on a single page (no LLM) ./scripts/extract_with_schema.py https://shop.example.com/category shop_schema.json /tmp/check.json - Network accessline 49
./scripts/extract_with_schema.py https://shop.example.com/category shop_schema.json /tmp/check.json - Network accessline 69
for url in $(cat news_urls.txt); do slug=$(echo "$url" | sed 's|https://||; s|/|_|g') - Network accessline 70
slug=$(echo "$url" | sed 's|https://||; s|/|_|g') - Network accessline 121
# 1. Fill in templates/login_crawler.yml with selectors + post-login wait condition # 2. Login (cookies persist under the session_id) crwl https://site.com/login -C templates/login_crawler.yml - Network accessline 123
crwl https://site.com/login -C templates/login_crawler.yml - Network accessline 125
# 3. Access protected pages, reusing the session crwl https://site.com/dashboard -c "session_id=user_session" -o markdown - Network accessline 126
crwl https://site.com/dashboard -c "session_id=user_session" -o markdown - Network accessline 127
crwl https://site.com/profile -c "session_id=user_session" -o markdown - Network accessline 139
url="raw:" + html_string, config=CrawlerRunConfig( base_url="https://example.com", # for relative-link resolution - Network accessline 140
config=CrawlerRunConfig( base_url="https://example.com", # for relative-link resolution screenshot=True, - Network accessline 141
base_url="https://example.com", # for relative-link resolution - Network accessline 165
crwl https://example.com -o markdown # preview - Network accessline 166
crwl https://example.com -q "What are the main conclusions?" # ask - Network accessline 167
crwl https://example.com -q "Summarize in 3 bullets" - Network accessline 170
First-time setup prompts for the LLM provider and API token; stored in `~/.crawl4ai/global.yml`. Any LiteLLM-supported provider works — pick a current model identifier from <https://docs.litellm.ai/docs/providers>. - Network accessline 171
provider works — pick a current model identifier from <https://docs.litellm.ai/docs/providers>.
references/sdk-guide.md
- Network accessline 21
async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com") - Network accessline 22
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com") print(result.markdown[:500]) - Network accessline 23
result = await crawler.arun("https://example.com") - Network accessline 46
async with AsyncWebCrawler(config=browser_config) as crawler: result = await crawler.arun( url="https://example.com", - Network accessline 47
result = await crawler.arun( url="https://example.com", config=crawler_config - Network accessline 48
url="https://example.com", - Network accessline 73
user_agent="custom-agent", # Custom user agent proxy_config={ # Proxy settings "server": "http://proxy:8080", - Network accessline 74
proxy_config={ # Proxy settings "server": "http://proxy:8080", "username": "user", - Network accessline 75
"server": "http://proxy:8080", - Network accessline 284
urls = ["https://site1.com", "https://site2.com", "https://site3.com"] - Network accessline 346
await crawler.arun("https://site.com/login", config=login_config) - Network accessline 348
# Subsequent crawls - reuse session config = CrawlerRunConfig(session_id="user_session") await crawler.arun("https://site.com/protected", config=config) - Network accessline 350
await crawler.arun("https://site.com/protected", config=config)
references/troubleshooting.md
- Network accessline 15
crwl https://example.com -c "wait_for=css:.dynamic-content,page_timeout=60000" - Network accessline 21
crwl https://example.com -c "wait_for=js:document.querySelector('.content') !== null,page_timeout=60000" - Network accessline 27
crwl https://example.com -c "wait_for=css:.list-item,scan_full_page=true,page_timeout=60000" - Network accessline 42
crwl https://example.com -B templates/browser.yml -o all -v - Network accessline 65
Symptom: after `crwl https://site.com/login -C login_crawler.yml`, the second crawl with `session_id=user_session` - Network accessline 78
crwl https://site.com/protected -c "session_id=user_session" -o all -v | rg -i "session\|cookie" - Network accessline 131
If a regression appears between two versions, check the upstream `CHANGELOG.md` (`https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md`) for the specific delta. The 0.7 → 0.8 transition in - Network accessline 132
(`https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md`) for the specific delta. The 0.7 → 0.8 transition in
scripts/basic_crawler.py
scripts/batch_crawl.py
- Executable scriptline 1
#!/usr/bin/env -S PYTHONDONTWRITEBYTECODE=1 uv run --script
scripts/batch_extract.py
- Executable scriptline 1
#!/usr/bin/env -S PYTHONDONTWRITEBYTECODE=1 uv run --script
scripts/extract_with_llm.py
- Executable scriptline 1
#!/usr/bin/env -S PYTHONDONTWRITEBYTECODE=1 uv run --script - Network accessline 17
Example: ./extract_with_llm.py https://news.example.com "Extract headlines, dates, summaries" news.json - Network accessline 18
./extract_with_llm.py https://news.example.com "Extract headlines, dates, summaries" news.json
scripts/extract_with_schema.py
scripts/generate_schema.py
- Executable scriptline 1
#!/usr/bin/env -S PYTHONDONTWRITEBYTECODE=1 uv run --script - Network accessline 15
Example: ./generate_schema.py https://shop.example.com "products with name, price, image" shop_schema.json - Network accessline 16
./generate_schema.py https://shop.example.com "products with name, price, image" shop_schema.json
templates/browser.yml
templates/extract_llm.yml
- Network accessline 2
# LLMExtractionStrategy template. Copy, fill, pass via # `crwl <url> -e extract_llm.yml -o json`. # Provider list: https://docs.litellm.ai/docs/providers - Network accessline 3
# `crwl <url> -e extract_llm.yml -o json`. # Provider list: https://docs.litellm.ai/docs/providers # Field reference: `references/complete-sdk-reference.md#extraction-strategies`. - Network accessline 4
# Provider list: https://docs.litellm.ai/docs/providers - Access to secretsline 10
[Credential-related source omitted]
templates/login_crawler.yml
- Network accessline 1
# SPDX-License-Identifier: MIT OR Apache-2.0 # Login-and-persist-session CrawlerRunConfig template. # Step 1: `crwl https://site.com/login -C login_crawler.yml` - Network accessline 2
# Login-and-persist-session CrawlerRunConfig template. # Step 1: `crwl https://site.com/login -C login_crawler.yml` # Step 2: `crwl https://site.com/protected -c "session_id=user_session"` - Network accessline 3
# Step 1: `crwl https://site.com/login -C login_crawler.yml` - Network accessline 4
# Step 2: `crwl https://site.com/protected -c "session_id=user_session"`
tests/run_all_tests.py
- Executable scriptline 1
#!/usr/bin/env -S PYTHONDONTWRITEBYTECODE=1 uv run --script
tests/test_advanced_patterns.py
- Executable scriptline 1
#!/usr/bin/env -S PYTHONDONTWRITEBYTECODE=1 uv run --script - Network accessline 21
config1 = CrawlerRunConfig(session_id=session_id) result1 = await crawler.arun("https://example.com", config=config1) - Network accessline 22
result1 = await crawler.arun("https://example.com", config=config1) - Network accessline 27
config2 = CrawlerRunConfig(session_id=session_id) result2 = await crawler.arun("https://example.org", config=config2) - Network accessline 28
result2 = await crawler.arun("https://example.org", config=config2) - Network accessline 40
headless=True, proxy_config={ "server": "http://proxy.example.com:8080", - Network accessline 41
proxy_config={ "server": "http://proxy.example.com:8080", "username": "user", - Network accessline 42
"server": "http://proxy.example.com:8080", - Network accessline 54
urls = ["https://example.com", "https://example.org"]
tests/test_basic_crawling.py
- Executable scriptline 1
#!/usr/bin/env -S PYTHONDONTWRITEBYTECODE=1 uv run --script - Network accessline 31
async with AsyncWebCrawler(config=browser_config) as crawler: result = await crawler.arun( url="https://example.com", - Network accessline 32
result = await crawler.arun( url="https://example.com", config=crawler_config - Network accessline 33
url="https://example.com",
tests/test_data_extraction.py
tests/test_fixtures.py
- Executable scriptline 1
#!/usr/bin/env -S PYTHONDONTWRITEBYTECODE=1 uv run --script
tests/test_markdown_generation.py
- Executable scriptline 1
#!/usr/bin/env -S PYTHONDONTWRITEBYTECODE=1 uv run --script - Network accessline 19
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com") - Network accessline 20
result = await crawler.arun("https://example.com") - Network accessline 40
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com", config=config) - Network accessline 41
result = await crawler.arun("https://example.com", config=config) - Network accessline 58
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com", config=config) - Network accessline 59
result = await crawler.arun("https://example.com", config=config) - Network accessline 78
async with AsyncWebCrawler() as crawler: result = await crawler.arun("https://example.com", config=config) - Network accessline 79
result = await crawler.arun("https://example.com", config=config)
How the signals fit together
Facts from a static read of the files at the pinned commit. They describe what the code does, not why.
Some files could not be read; the facts below still hold.
Signal confidence: 496 facts
- Network access in prose of documentation.github/pull_request_template.md:143Low confidence
- Network access in prose of documentation.github/pull_request_template.md:144Low confidence
- Network access in prose of documentationAGENTS.md:2Low confidence
- Network access in prose of documentationAGENTS.md:3Low confidence
- Network access in prose of documentationAGENTS.md:4Low confidence
- Network access in prose of documentationCHANGELOG.md:5Low confidence
- Network access in prose of documentationCHANGELOG.md:6Low confidence
- Network access in prose of documentationCONTRIBUTING.md:12Low confidence
- Network access in prose of documentationCONTRIBUTING.md:35Low confidence
- Network access in prose of documentationCONTRIBUTING.md:36Low confidence
- Network access in prose of documentationREADME.md:3Low confidence
- Network access in prose of documentationREADME.md:4Low confidence
- Network access in code of documentationREADME.md:24Low confidence
- Network access in code of documentationREADME.md:25Low confidence
- Network access in code of documentationREADME.md:49Low confidence
- Network access in code of documentationREADME.md:50Low confidence
- Network access in code of documentationREADME.md:59Low confidence
- Network access in code of documentationREADME.md:60Low confidence
- Network access in code of documentationREADME.md:61Low confidence
- Network access in code of documentationREADME.md:99Low confidence
- Network access in code of documentationREADME.md:105Low confidence
- Network access in code of documentationREADME.md:106Low confidence
- Network access in code of documentationREADME.md:108Low confidence
- Network access in code of documentationREADME.md:109Low confidence
- Network access in code of documentationREADME.md:115Low confidence
- Network access in code of documentationREADME.md:116Low confidence
- Network access in prose of the SKILL.mdSKILL.md:14Low confidence
- Network access in code of the SKILL.mdSKILL.md:73High confidence
- Network access in code of the SKILL.mdSKILL.md:74High confidence
- Network access in code of the SKILL.mdSKILL.md:76High confidence
- Network access in code of the SKILL.mdSKILL.md:77High confidence
- Network access in code of the SKILL.mdSKILL.md:79High confidence
- Network access in code of the SKILL.mdSKILL.md:80High confidence
- Network access in code of the SKILL.mdSKILL.md:92High confidence
- Network access in code of the SKILL.mdSKILL.md:93High confidence
- Network access in code of the SKILL.mdSKILL.md:94High confidence
- Network access in code of the SKILL.mdSKILL.md:156High confidence
- Network access in code of the SKILL.mdSKILL.md:157High confidence
- Network access in code of the SKILL.mdSKILL.md:158High confidence
- Network access in code of the SKILL.mdSKILL.md:159High confidence
- Network access in code of the SKILL.mdSKILL.md:193High confidence
- Network access in code of the SKILL.mdSKILL.md:194High confidence
- Network access in code of the SKILL.mdSKILL.md:200High confidence
- Network access in code of the SKILL.mdSKILL.md:211High confidence
- Network access in code of the SKILL.mdSKILL.md:217High confidence
- Network access in code of the SKILL.mdSKILL.md:232High confidence
- Network access in code of the SKILL.mdSKILL.md:233High confidence
- Network access in code of the SKILL.mdSKILL.md:249High confidence
- Network access in code of the SKILL.mdSKILL.md:277High confidence
- Network access in code of the SKILL.mdSKILL.md:278High confidence
and 446 more
Works with
| Agent | Status | Evidence | Install to |
|---|---|---|---|
| Claude Code | Native | Frontmatter field argument-hint in SKILL.md | .claude/skills/<skill-name>/ |
| Codex | Native | Manifest AGENTS.md | .agents/skills/<skill-name>/ |
| Cursor | Compatible | Standard SKILL.md SKILL.md | .cursor/skills/<skill-name>/ |
| Gemini CLI | Compatible | Standard SKILL.md SKILL.md | .gemini/skills/<skill-name>/ |
| GitHub Copilot | Compatible | Standard SKILL.md SKILL.md | .github/skills/<skill-name>/ |
| OpenCode | Compatible | Standard SKILL.md SKILL.md | .opencode/skills/<skill-name>/ |
- Native The repository ships a manifest, folder or frontmatter field for this agent.
- Compatible A standard SKILL.md that this agent can read.
- Unknown No evidence either way.
Derived automatically from paths, manifests and frontmatter. Not tested on each agent.
