Individual skill
Normalize and filter noisy URL lists before crawling or queueing
Uses Courlan to clean, normalize, de-track, and language-filter raw URL inventories before a crawler, scraper, or analyst queue touches them. Best when an agent already has too many candidate links and needs a smaller, cleaner frontier, not a full crawling stack.
skills/normalize-and-filter-noisy-url-lists-before-crawling-or-queueing/SKILL.md
Source commit fc97377adefd53e861c1d456aa1392e4a4e022a2
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- No
- Repository scan complete
- No
- Skill scan complete
- Yes
- Observed behavior
- Not checked yet
- Last checked
- Nov 6, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
Standard SKILL.md format has not been attested at this commit. No installation command is available; inspect the source.
Repository context
- Repository stars (not skill ratings)
- 62
- Repository last updated
- Nov 8, 2026
- License
- MIT
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- agent-skills Repository topic
- ai-agents Repository topic
- claude-code Repository topic
- codex Repository topic
- cursor Repository topic
- npx-skills Repository topic
- openclaw Repository topic
- ai-tools Repository topic
- mcp Repository topic
- awesome-list Repository topic
- engineering Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
SKILL.mdcore · 2,268 bytes
Read excerpt
# Normalize and filter noisy URL lists before crawling or queueing
Uses Courlan to clean, normalize, de-track, and language-filter raw URL inventories before a crawler, scraper, or analyst queue touches them. Best when an agent already has too many candidate links and needs a smaller, cleaner frontier, not a full crawling stack.
## Prerequisites
Python 3, pip, command line
## Installation
Use the upstream install or setup path that matches your environment:
- $ pip install courlan # pip3 install on systems where both Python 2 and 3 are installed
- $ pip install --upgrade courlan # to make sure you have the latest version
- $ pip install git+https://github.com/adbar/courlan.git # latest available code (see build status above)