Menu

Individual skill

data-scraper

0upvotes

Use when data lives on a website with no usable API — listings, prices, public records — and the scrape must stay legal and not get blocked: legal gate, extraction path, durable selectors, pacing, resilience. NOT parsing bytes you already hold into fields (that is structured-extraction), NOT a documented API or key (that is api-connector-builder).

skills/data-scraper/SKILL.md

Read pinned source

Source commit d9d6bb338f66cc1e23998585e580a0ea45bbcb82

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
No
Repository scan complete
No
Skill scan complete
No
Observed behavior
Not checked yet
Last checked
Nov 6, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings)
573
Repository last updated
Nov 4, 2026
License
MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt

# Data scraper

You get bytes off websites you do not control — legally, and without getting blocked. You pick the cheapest extraction path that works, build selectors that survive a redesign, pace requests so the host neither bans nor sues you, and you write down the legal basis *before* the first request goes out.

One rule above all the others: **scraping is the fallback, not the default.** It is what you reach for only when no API serves the data. If the site has a documented API or you hold a key, stop — that is `../api-connector-builder/SKILL.md`. And once you have the bytes, parsing them into fields is `../structured-extraction/SKILL.md`, normalizing the rows is `../data-cleaning/SKILL.md`. This skill ends the moment you hold the bytes.

In 2025-2026 scrapers do not fail on parsing. They fail on a terms-of-service breach, on GDPR exposure, or on being blocked after hammering a host. So the work runs in this order: legal gate → extraction path → tool → selectors → politeness → resilience. Skipping the gate is how you end up in *Meta v. Bright Data*.

## The legal gate — run this before any request

Walk every item. Each ends in **proceed**, **proceed-narrowed**, or **stop**. One item at stop means the whole scrape stops until you resolve it. Depth and the case law are in `references/legal-compliance.md`.

The gates below are the ones with real legal exposure — contract, data-protection law, and anti-circumvention. **robots.txt is not one of them**: it is a voluntary convention with no statutory force, so it informs the decision and never blocks it on its own.
Back to top