Individual skill

crestapps-core-web-crawlers

Skill for CrestApps.Core web crawlers that scrape public websites into a Web AI data source. Covers CrestApps.Core.AI.WebCrawlers with AddCoreWebCrawlers and AddCoreWebCrawlerStoresYesSql or AddCoreWebCrawlerStoresEntityCore, WebCrawler records with SitemapWebCrawlerMetadata, the Sitemap strategy for urlset, sitemapindex, gzip, text sitemaps, feeds and robots.txt, WebCrawlerOptions, the re-index planner and background service with lastmod-based incremental updates, PartiallyDiscovered results that never delete pages, page URL citations, HtmlIngestionDocumentReader from CrestApps.Core.DataIngestion, crawlers that feed a File data source, and custom IWebCrawlerStrategy implementations.

src/CrestApps.AgentSkills/crestapps-core/crestapps-core-web-crawlers/SKILL.md

Read pinned source

Source commit d86d04f234b1b2bd8145368a2544490f9d557596

What was checked

Matches the observed repository HEAD: Yes
Scanner rules are current: Yes
Repository scan complete: No
Skill scan complete: No
Observed behavior: Instructions only
Last checked: Nov 6, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings): 13
Repository last updated: Oct 21, 2026
License: MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

agent-skills Repository topic
ai Repository topic
ai-agents-framework Repository topic
dotnet Repository topic
mcp Repository topic
mcp-server Repository topic
orchardcore Repository topic
data Derived category

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files

SKILL.mdcore · 10,264 bytes
SKILL.mdcore · 10,264 bytes
SKILL.mdcore · 10,264 bytes
SKILL.mdcore · 10,264 bytes
SKILL.mdcore · 10,264 bytes
SKILL.mdcore · 10,264 bytes
SKILL.mdcore · 10,264 bytes

Read excerpt


# CrestApps.Core Web Crawlers

A **Web** AI data source (`AIDataSourceSourceTypes.Web`) is a target bucket with no site configuration and no field mapping. A **web crawler** (`WebCrawler` record) chooses a strategy (its `Source`, for example `Sitemap`), stores that strategy's settings, and points at the data source. Many crawlers can fill one knowledge base, each re-crawled on its own interval. Folders and file servers use file sources instead (`crestapps-core-file-sources`); the data source model is in `crestapps-core-data-sources`.

## Registration

```csharp
using CrestApps.Core.AI.WebCrawlers;
using CrestApps.Core.Data.YesSql;

builder.Services.AddCrestAppsCore(crestApps => crestApps
    .AddAISuite(ai => ai.AddOpenAI())
    .AddIndexingServices(indexing => indexing
        .AddYesSqlStores()