Individual skill

octocode-scraping

Use when extracting or mapping public web content into a local cited corpus: scrape or crawl a URL/docs site, pull tables/pricing/product fields, diagnose blocked or thin pages, or answer from saved pages. Phrases like scrape this URL, crawl the docs, build a corpus, extract pricing. Prefer keyless fetch; ask before hosted spend. Live clicks/HAR/perf → octocode-chrome-devtools.

skills/octocode-scraping/SKILL.md

Read pinned source

Source commit c265e3f9413161c900cbf4aa70d451b8e6b3920a

What was checked

Matches the observed repository HEAD: Yes
Scanner rules are current: No
Repository scan complete: No
Skill scan complete: Yes
Observed behavior: Not checked yet
Last checked: Nov 4, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings): 958
Repository last updated: Nov 6, 2026
License: MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

ai Repository topic
claude-ai Repository topic
cursor-ai Repository topic
development Repository topic
mcp Repository topic
modelcontextprotocol Repository topic
octocode Repository topic
ai-agents Repository topic
ai-tools Repository topic
code-intelligence Repository topic
writing Derived category
docs Derived category

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files

README.mddoc · 965 bytes
SKILL.mdcore · 5,283 bytes
docs/ADDING_A_VENDOR.mddoc · 4,134 bytes
docs/PROVIDERS.mddoc · 2,325 bytes
references/browser-scraping.mddoc · 1,953 bytes
references/data-contract.mddoc · 2,494 bytes
references/extraction-quality.mddoc · 1,105 bytes
references/failure-recovery.mddoc · 1,531 bytes
references/providers.mddoc · 1,040 bytes
references/route-selection.mddoc · 1,068 bytes
references/scraping-policy.mddoc · 1,336 bytes
references/scrapingant.mddoc · 1,716 bytes
references/session-corpus.mddoc · 1,465 bytes
references/user-inputs.mddoc · 1,109 bytes
references/website-analysis.mddoc · 2,975 bytes
scripts/README.mddoc · 1,270 bytes
scripts/corpus-find.mjsruns · 4,079 bytes
scripts/corpus-inspect.mjsruns · 2,028 bytes
scripts/corpus-run.mjsruns · 8,997 bytes
scripts/dom-find.mjsruns · 2,130 bytes
scripts/fetch-and-brief.mjsruns · 1,457 bytes
scripts/fetch.mjsruns · 4,741 bytes
scripts/graph-navigate.mjsruns · 2,715 bytes
scripts/har-ingest.mjsruns · 12,933 bytes
scripts/lib/analyzers.mjsruns · 14,654 bytes
scripts/lib/args.mjsruns · 5,535 bytes
scripts/lib/bridge.mjsruns · 9,927 bytes
scripts/lib/client.mjsruns · 12,276 bytes
scripts/lib/corpus.mjsruns · 19,298 bytes
scripts/lib/extractors.mjsruns · 10,751 bytes
scripts/lib/providers.mjsruns · 1,508 bytes
scripts/lib/text.mjsruns · 3,858 bytes
scripts/provider-check.mjsruns · 2,173 bytes
scripts/provider-usage.mjsruns · 1,445 bytes
scripts/resource-list.mjsruns · 2,274 bytes
scripts/schema-helper.mjsruns · 2,090 bytes
scripts/schemas/graph.schema.jsonasset · 2,582 bytes
scripts/schemas/provider.schema.jsonasset · 2,352 bytes
scripts/scrapingant-check.mjsruns · 436 bytes
scripts/scrapingant-fetch.mjsruns · 418 bytes
scripts/scrapingant-usage.mjsruns · 436 bytes

Read excerpt


# Octocode Scraping

Flow: `FRAME → POLICY → ROUTE → FETCH → CORPUS → SEARCH → CITE → RECOVER`.

FRAME before the first fetch: fix target URL/domain, goal, depth, and output shape — vague ask → `references/user-inputs.md`.

Defaults: one public URL, `--mode html`, omit `--provider` (keyless `cdp`→`direct`), session `.octocode/tmp/scrape/{sessionId}`, compact stdout. Search corpus before refetch. Live interaction → chrome-devtools on one port, then `har-ingest` + `corpus-run` into the same session. Ask before auth, hosted spend, crawl widen, CAPTCHA/MFA, destructive actions. Cite paths + URL metadata — not raw dumps.

Stop when: two same-class failures (report evidence, route tried, sanitized status, next approval); hosted `403` (wrong key or credits gone — status only, no retry); CAPTCHA/MFA, auth wall, or cookie/profile transfer needed; still blocked after one `cdp` try (ask before `--provider scrapingant`); personal data, form submits, purchases, sends, deletes, or account changes in scope; the saved corpus already proves the claim (cite it, do not refetch); crawl widen before `reports/summary.md` is useful. Recovery table: `references/failure-recovery.md`.

## Route (pick one)
| Need | Do | Skip |
|---|---|---|
| Vague scrape | `--mode html`, omit `--provider` | markdown / auto hosted |