Individual skill

data-pipeline-architect

Designs robust ETL/ELT data pipelines covering ingestion, idempotency, schema evolution, orchestration, and data quality validation. Use this skill when the user asks to design, build, or review a data pipeline, ingest data from APIs/databases/files into a warehouse or lake, set up batch or streaming ETL/ELT, choose an orchestrator (Airflow, Dagster, Prefect, dbt), make a pipeline idempotent or backfill-safe, handle late-arriving or duplicate data, manage schema drift/evolution, add data quality or freshness checks, or model incremental/CDC loads.

data-pipeline-architect/SKILL.md

Read pinned source

Source commit f9a456bf34c297bd23fb234f410ec9b239ebed14

What was checked

Matches the observed repository HEAD: Yes
Scanner rules are current: Yes
Repository scan complete: No
Skill scan complete: No
Observed behavior: Local scripts
Last checked: Nov 6, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings): 12
Repository last updated: Oct 14, 2026
License: MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

agent-skills Repository topic
ai-agents Repository topic
anthropic Repository topic
awesome Repository topic
claude Repository topic
claude-code Repository topic
codex Repository topic
cursor Repository topic
gemini-cli Repository topic
llm Repository topic
design Derived category
data Derived category

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files

SKILL.mdcore · 7,828 bytes
SKILL.mdcore · 7,828 bytes
SKILL.mdcore · 7,828 bytes
SKILL.mdcore · 7,828 bytes
examples/orders-pipeline.mddoc · 3,312 bytes
references/data-quality.mddoc · 3,425 bytes
references/etl-vs-elt.mddoc · 2,496 bytes
references/idempotency.mddoc · 2,923 bytes
references/ingestion-patterns.mddoc · 3,249 bytes
references/orchestration.mddoc · 3,132 bytes
references/schema-evolution.mddoc · 3,113 bytes
scripts/pipeline_lint.pyruns · 8,762 bytes
templates/pipeline-design.mddoc · 2,289 bytes

Read excerpt


# Data Pipeline Architect

## Overview

This skill helps you design and review production-grade data pipelines. It covers the full lifecycle: ingestion, transformation, idempotency and backfills, schema evolution, orchestration, and data quality. It is opinionated toward modern ELT (load raw, transform in-warehouse) but supports classic ETL where it fits.

Keywords: ETL, ELT, data pipeline, ingestion, idempotency, backfill, schema evolution, schema drift, CDC, incremental load, watermark, orchestration, Airflow, Dagster, Prefect, dbt, data quality, freshness, dedup, late-arriving data, partitioning, medallion, bronze silver gold.

Use this skill to produce a concrete pipeline design (a design doc), to review an existing pipeline against best practices, or to generate skeleton DAGs/models and data-quality checks.

## Decision: ETL vs ELT

Default to **ELT** when the destination is a modern columnar warehouse/lake (Snowflake, BigQuery, Redshift, Databricks, DuckDB). Land raw data first, transform with SQL/dbt. Use **ETL** when: the destination can't transform cheaply, you must mask/drop PII before it lands (compliance), or you transform in-flight for a stream. See `references/etl-vs-elt.md`.