vaquarkhan/data-engineering-agent-skills

# Airflow And Workflow Orchestration

## Overview

Use this skill when the problem is orchestration rather than transformation logic. It helps agents design `Airflow`, `MWAA`, `Cloud Composer`, `Azure Data Factory`, `Step Functions`, `Google Cloud Workflows`, or `Databricks Workflows` patterns with explicit dependencies, retries, ownership, backfills, and publish-safe cutover behavior.

## When to Use

- building or modifying `Airflow` DAGs
- choosing between scheduler-driven, event-driven, or platform-native orchestration
- designing scheduling, sensors, task dependencies, or retry policy
- coordinating ingestion, transformation, quality checks, and publish steps
- changing backfill, catchup, or SLA behavior

# Apache Beam Unified Batch And Stream

## Overview

Use this skill when `Apache Beam` is the abstraction layer for both batch and streaming data processing. It helps agents preserve portability without hiding time semantics or runner-specific constraints.

## When to Use

- building `Apache Beam` pipelines for batch, streaming, or unified workloads
- targeting multiple runners such as `Dataflow`, `Flink`, `Spark`, or `Direct Runner`
- sharing transform logic across batch and streaming modes
- managing windowing, watermarks, triggers, and late data handling
- designing portable pipelines that must run across environments

# Apache Hudi Lakehouse

## Overview

Use this skill when `Apache Hudi` is the primary table layer for incremental lakehouse workloads. It helps agents reason about mutation-heavy patterns, table type selection, compaction behavior, timeline safety, and consumer expectations across read-optimized and real-time query paths.

## When to Use

- choosing or operating `Apache Hudi` for lakehouse tables
- building record-level upsert or delete pipelines
- managing compaction, clustering, and incremental consumption
- supporting lakehouse tables with heavy mutations (CDC sinks, slowly changing dimensions)
- planning multi-engine access (Spark, Presto, Trino, Athena, Hive)

# API And SaaS Ingestion Patterns

## Overview

Use this skill when the source system is an external API or SaaS platform rather than a database or file drop. It helps agents design reliable extraction, pagination, throttling, auth handling, and backfill-safe ingestion contracts.

## When to Use

- extracting from REST or GraphQL APIs
- ingesting SaaS platform data
- handling pagination, cursors, or sync tokens
- dealing with rate limits and source-side throttling
- backfilling historical API data safely

# Avro Protobuf JSON Schema Registry

## Overview

Use this skill when schema management for events must be explicit and enforceable. It helps agents coordinate producer changes, consumer compatibility, registry policy, and versioned contracts across teams sharing event streams.

## When to Use

- adopting a schema registry for event-driven systems
- defining or modifying Avro, Protobuf, or JSON Schema definitions
- setting compatibility policies for topics or subjects
- coordinating producer schema changes across multiple consumers
- enforcing contract governance for multi-team event ecosystems

# BigQuery And Dataform Platform Engineering

## Overview

Use this skill when `BigQuery` is the center of gravity for analytics engineering and data delivery on `GCP`. It helps agents decide what should run in `BigQuery`, what belongs in `Dataform`, and when the workflow should move to `Dataflow`, `Dataproc`, or external orchestration.

## When to Use

- designing or reviewing `BigQuery` physical models
- deciding between `Dataform`, `dbt`, `Dataflow`, or `Dataproc` responsibilities
- tuning partitioning, clustering, slots, and cost behavior
- defining ingestion and transformation boundaries on `GCP`
- building platform-native analytics workflows around `BigQuery`

# CDC And Incremental Loading

## Overview

Use this skill when the system changes data over time and full reloads are not the right answer. It helps agents design safe incremental behavior with clear watermarks, merge rules, deduplication, and replay semantics.

## When to Use

- implementing CDC ingestion
- designing watermark-based or timestamp-based incremental loads
- merging inserts, updates, and deletes
- handling late-arriving data
- fixing duplicate or missing incremental records

# ClickHouse Real Time Analytics

## Overview

Use this skill when `ClickHouse` is the target for low-latency analytical serving. It helps agents design ingestion, partitioning, materialized views, and query-ready schemas for fast reads while maintaining operational safety and cost control.

## When to Use

- designing or modifying `ClickHouse` tables for real-time analytics
- building event-heavy analytical aggregation layers
- creating materialized views for pre-computed metrics
- optimizing low-latency dashboards and metric APIs
- planning ingestion patterns for high-throughput event streams

# Data Catalog And Discovery

## Overview

Use this skill when the challenge is not only building data, but making it understandable and discoverable. It helps agents treat metadata, ownership, lineage, and usage context as delivery artifacts instead of afterthoughts.

## When to Use

- publishing a new shared dataset
- improving catalog metadata quality
- curating lineage, tags, or ownership information
- reducing duplicate datasets created because teams cannot find trusted ones

Do not stop at filling in a title and description. Discovery quality requires operational context too.

# Data Contract Testing With Schema Registry

## Overview

Use this skill when contracts must be tested through registry-backed compatibility rules, not only by convention. It helps agents make event contract validation a blocking part of build and release behavior with automated testing, consumer impact analysis, and evidence-driven release gates.

## When to Use

- testing event contract changes before production release
- validating schema compatibility as part of CI/CD pipelines
- coordinating producer and consumer schema evolution across teams
- enforcing registry-backed contract quality gates before deployment
- building release evidence that proves schema safety

# Data Lake And Zone Architecture

## Overview

Use this skill when the storage platform needs structure before pipelines scale into chaos. It helps agents design clear lake zones, dataset boundaries, ownership, lifecycle rules, and publish-safe storage conventions.

## When to Use

- designing a new data lake
- reorganizing raw, staging, refined, or curated zones
- defining object storage layout and lifecycle rules
- separating landing, transformation, and publish responsibilities
- reducing data swamp behavior in shared lake storage

# Data Mesh And Domain-Oriented Design

## Overview

Use this skill when the problem is organizational scale as much as technical scale. It helps agents design domain-owned data products with explicit boundaries, interoperable contracts, and platform guardrails that do not collapse back into central bottlenecks.

## When to Use

- designing domain-oriented data products
- defining ownership boundaries across multiple teams
- introducing or refining a data mesh operating model
- clarifying platform versus domain responsibilities
- reducing central-team bottlenecks in large data organizations

# Data Migration And Platform Cutover

## Overview

Use this skill when the system is changing platforms or major architectural boundaries. It helps agents plan parallel runs, compatibility layers, validation, and safe cutover rather than treating migration as a one-day switch.

## When to Use

- migrating from one warehouse or lakehouse to another
- moving pipelines between orchestration or compute systems
- replatforming storage formats or table engines
- performing major serving-layer cutovers

Do not start cutover work without a rollback and validation path.

# Data Observability And SLA Management

## Overview

Use this skill when the pipeline must be operated as a service, not just executed as code. It helps agents define what healthy looks like and how teams know when the system drifts away from that state.

## When to Use

- launching or hardening a production data product
- defining freshness or completeness SLAs
- improving alerting and anomaly detection
- adding operational metadata and health visibility
- reducing noisy or low-signal incident response
- designing resilience drills with explicit alert and recovery evidence

# Data Platform CI CD And Release Management

## Overview

Use this skill when changes need controlled promotion across environments. It helps agents design release gates for code, contracts, infra, and datasets so teams can ship faster without losing trust.

## When to Use

- adding or changing CI pipelines for data projects
- promoting changes from dev to staging to production
- releasing `dbt`, orchestration, infra, or schema updates
- defining rollback-aware deployment behavior

Do not treat data releases like app-only deploys. Data contracts, backfills, and environment state matter too.

# Data Platform Disaster Recovery And Business Continuity

## Overview

Use this skill when the question is how the data platform survives major failure, not only how one incident is handled. It helps agents design recovery objectives, backup and restore paths, control-plane recovery, failover decisions, and repeatable restore drills for data systems.

## When to Use

- defining `RTO` and `RPO` for critical datasets or platform services
- planning region, account, or environment failover
- validating backup and restore behavior for warehouses, lakes, or stateful processors
- designing business continuity for shared data products and critical publish paths
- running recovery drills before a real outage forces them

# Data Platform Operating Model And Service Ownership

## Overview

Use this skill when the hard problem is organizational and operational, not just technical. It helps agents define who owns platform capabilities, how teams onboard safely, what the golden paths are, and how support, escalation, and service boundaries work across a shared data platform.

## When to Use

- defining central platform-team responsibilities
- clarifying handoffs between platform and domain teams
- designing golden paths for new datasets, pipelines, or consumer onboarding
- documenting support tiers, ownership, and escalation paths
- reducing operational sprawl and unclear platform accountability

# Data Quality And Contract Testing

## Overview

Data work is not complete when code runs. It is complete when source assumptions, output contracts, and quality checks prove the behavior is correct.

## When to Use

- new source ingestion
- schema changes
- transformation logic updates
- new or changed published tables
- bug fixes involving bad data or broken metrics

# Data Quality Platforms And Rule Management

## Overview

Use this skill when the question is not only what checks to write, but how the quality program should operate across tools, teams, and publish stages. It helps agents design rule ownership, severity, evidence, and enforcement across multiple quality frameworks.

## When to Use

- selecting or combining data-quality tools
- designing rule portfolios and severity models
- aligning `dbt` tests, `Great Expectations`, `Deequ`, `Cuallee`, `Soda`, or warehouse-native checks
- deciding what blocks publish versus what raises warnings
- improving long-term maintainability of data-quality controls

# Data Reconciliation And Financial Controls

## Overview

Use this skill when correctness must be proven with control evidence, not only tests. It helps agents design reconciliations, control totals, exception workflows, and audit-friendly validation for high-trust datasets.

## When to Use

- finance, billing, or revenue pipelines
- audit-sensitive operational datasets
- month-end or close-process data products
- source-to-target control validations
- exception-based review flows

# Data Resiliency Testing And Failure Injection

## Overview

Use this skill when the goal is to prove that a data system recovers safely under failure, not only when everything goes right. It helps agents design controlled drills for retries, restarts, dependency outages, state recovery, duplicate prevention, backlog catch-up, and publish protection.

## When to Use

- hardening a production pipeline before broad rollout
- testing failover, restart, replay, or checkpoint recovery behavior
- validating that retries do not duplicate or corrupt data
- proving recovery objectives for orchestrators, jobs, streams, or warehouse publishes
- converting a past incident into a repeatable resilience drill

# Data Security, Compliance, And Regulated Data

## Overview

Use this skill when the data platform handles regulated or highly sensitive data and the change must satisfy engineering, governance, and audit expectations together. It helps agents treat controls, lineage, access, retention, deletion, evidence, and publish safety as one delivery surface.

## When to Use

- handling `PII`, `PCI`, `HIPAA`, `PHI`, or other regulated data classes
- moving sensitive data between raw, curated, serving, and partner-facing layers
- designing masking, tokenization, encryption, or access controls
- proving that published data meets audit or policy requirements
- changing lineage, retention, deletion, or evidence paths for regulated assets
- preparing a system for compliance review, internal audit, or control sign-off

# Data Sharing And Publishing Contracts

## Overview

Use this skill when data leaves the producing team and becomes a product for other teams, systems, or partners. It helps agents define stable publish contracts with ownership, compatibility, and operational expectations.

## When to Use

- publishing curated tables or marts
- sharing extracts or files with other teams or partners
- exposing governed data through APIs or reverse ETL
- modifying an existing consumer-facing dataset

Do not publish widely-used data with implicit contracts.

# Data Specification

## Overview

Write the data specification before writing pipeline code. The spec should define business intent, source and destination expectations, quality rules, and success criteria so the agent is not forced to guess.

## When to Use

- new ingestion or transformation projects
- schema or contract changes
- major changes to data products, marts, or semantic models
- requests that sound simple but leave operational details unclear

Do not use this for trivial spelling fixes or non-behavioral documentation edits.

# Dataplex And BigQuery Governance

## Overview

Use this skill when governance on `GCP` is centered on `Dataplex`, `BigQuery`, and related Google Cloud metadata controls. It helps agents define governance zones, policy-tag boundaries, lineage expectations, and trusted publishing patterns for warehouse and lake platforms.

## When to Use

- designing `Dataplex` lakes, zones, or governance domains
- defining `BigQuery` policy tags and governed publish boundaries
- improving metadata quality, lineage, and discovery on `GCP`
- aligning warehouse and lake governance across `Cloud Storage`, `BigQuery`, `Dataflow`, and `Dataproc`
- making governed analytics delivery work with regional or regulated-data controls

# dbt And Analytics Engineering

## Overview

Use this skill when the job is analytics engineering rather than raw ingestion. It helps agents build trustworthy `dbt` projects with clear layering, reusable models, tests, documentation, and publish-safe business definitions.

## When to Use

- creating or changing `dbt` models
- building staging, intermediate, or mart layers
- adding tests, snapshots, or exposures
- organizing business logic for analysts and BI tools
- preparing semantic-layer-friendly outputs

# Debezium And Kafka Connect CDC

## Overview

Use this skill when database changes must be captured and delivered through `Debezium` and `Kafka Connect`. It helps agents define connector safety, snapshot behavior, schema handling, and downstream CDC contracts.

## When to Use

- setting up `Debezium` connectors for `PostgreSQL`, `MySQL`, `MongoDB`, `SQL Server`, or `Oracle`
- designing Kafka-based CDC from transactional databases
- handling initial snapshots, incremental streaming, and connector recovery
- feeding downstream stream processors, lakehouse sinks, or search indexes
- managing schema evolution when source tables change

# Delta Lake And Medallion Architecture

## Overview

Use this skill when a `Delta Lake`-based lakehouse needs more than generic table-format guidance. It helps agents design medallion-style layers, transactional update behavior, CDC merges, and publish-safe transformation paths for Delta-centric platforms.

## When to Use

- designing `bronze`, `silver`, and `gold` layering
- implementing `Delta Lake` merges, upserts, or deletes
- building `Databricks`-centered batch and streaming lakehouse flows
- deciding how raw landing data evolves into trusted publish outputs

Do not use medallion terminology as decoration if the layers do not carry distinct responsibilities.

# DuckDB Local Analytics And Dev

## Overview

Use this skill when `DuckDB` is the fastest path to local analytical iteration. It helps agents build reproducible local workflows that accelerate development without confusing prototype convenience for production architecture.

## When to Use

- prototyping data models and transformations locally before deploying to a warehouse
- reproducing production data issues with sample datasets
- running analytical queries during development without remote infrastructure
- building lightweight CLI tools, validators, or test harnesses
- validating dbt models locally with `dbt-duckdb` adapter
- creating proof-of-concept demonstrations with embedded analytics

# Enterprise ETL And Data Integration Modernization

## Overview

Use this skill when delivery depends on classic enterprise ETL tooling, not only modern code-first platforms. It helps agents reason about mapping logic, scheduler dependencies, restart behavior, metadata export, migration sequencing, and coexistence between legacy ETL platforms and newer Spark, dbt, Airflow, or lakehouse stacks.

## When to Use

- working with `Informatica`, `Talend`, `DataStage`, `SSIS`, or `Matillion`
- reverse-engineering existing ETL mappings and workflows
- modernizing or migrating legacy ETL jobs
- adding quality, lineage, and operational controls to GUI-driven pipelines
- running hybrid estates where old and new tooling must coexist

# ESG And Sustainability Regulatory Reporting

## Overview

Use this skill when sustainability reporting becomes a governed data-engineering problem instead of a spreadsheet exercise. It helps agents build traceable data pipelines, materiality and methodology records, value-chain inputs, approval paths, and assurance-ready evidence for ESG or sustainability disclosures.

## When to Use

- building data products for `CSRD`, `ESRS`, `BRSR`, climate, or investor-facing sustainability disclosures
- managing emissions, workforce, supplier, governance, or value-chain metrics
- preparing ESG datasets for audit, assurance, or board reporting
- replacing spreadsheet-driven sustainability reporting with governed pipelines
- aligning sustainability metrics with finance, ERP, procurement, and operational source systems

# ETL ELT And Modernization Strategy

## Overview

Use this skill when the hard part is not a single job, but deciding where transformations should run and how a data estate should modernize over time. It helps agents reason about `ETL` versus `ELT`, pushdown versus external compute, orchestration boundaries, migration sequencing, and proof of parity during modernization.

## When to Use

- choosing between `ETL`, `ELT`, or hybrid transformation patterns
- moving from legacy ETL tools into warehouse, dbt, Spark, or lakehouse execution
- redesigning ingestion and transformation boundaries across raw, curated, and publish layers
- reducing operational sprawl caused by duplicate transformation logic
- modernizing batch-first estates without breaking existing delivery expectations

# Feature Store And ML Data Pipelines

## Overview

Use this skill when the platform must support model training and inference safely. It helps agents design feature generation, point-in-time correctness, serving parity, and operational contracts for ML-focused data products.

## When to Use

- building training datasets
- designing feature stores or reusable features
- supporting online and offline feature access
- preventing leakage and training-serving mismatch
- publishing model-ready data products

# File And Partner Feed Ingestion

## Overview

Use this skill when ingestion depends on externally delivered files rather than APIs or CDC streams. It helps agents design landing, validation, late-file handling, checksum or manifest controls, replay safety, and publish boundaries for partner-managed feeds.

## When to Use

- onboarding `SFTP`, MFT, or shared-bucket file feeds
- ingesting recurring `CSV`, `JSON`, `XML`, or columnar extracts from external partners
- validating manifests, checksums, control totals, or arrival windows
- handling late, duplicate, partial, or corrected file deliveries
- defining replay-safe landing and publish behavior for externally supplied data

# Glue Data Catalog And Lake Formation Governance

## Overview

Use this skill when `AWS` governance is anchored in `Glue Data Catalog` and `Lake Formation`, not only in generic metadata tools. It helps agents design catalog structure, access controls, governed sharing, and publish-safe dataset access across lake and warehouse workflows.

## When to Use

- designing `Glue Data Catalog` database or table organization
- defining `Lake Formation` permissions, tag-based access, or sharing boundaries
- reviewing governed access for `S3`, `Athena`, `Glue`, `EMR`, or `Redshift`
- improving metadata quality for AWS-native data discovery
- aligning platform-native governance with regulated-data and publish controls

# Great Expectations Deequ And Cuallee

## Overview

Use this skill when the team wants structured quality enforcement through a data-quality framework instead of ad hoc checks. It helps agents align contracts with expectation suites, route failures by severity, and produce reviewable validation evidence.

## When to Use

- implementing `Great Expectations` expectation suites or checkpoints
- adding `Deequ` analyzers and constraints to Spark pipelines
- using `Cuallee` for lightweight PySpark or Pandas validation
- standardizing reusable quality checks across datasets
- integrating framework-based quality gates into CI/CD or publish workflows
- producing validation evidence for audits or release gates

# Incident Triage And Pipeline Recovery

## Overview

Use this skill when production behavior is already wrong and the team needs controlled recovery. It helps agents contain blast radius, diagnose quickly, restore trust, and avoid making the incident worse.

## When to Use

- failed production runs
- bad or partial publishes
- corrupted incremental state
- freshness SLA breaches
- emergency rollback or replay decisions

# Java Data Engineering And Integration Services

## Overview

Use this skill when `Java` is the main implementation language for data-adjacent services or processing components. It helps agents design operationally safe JVM services for ingestion, metadata, contracts, stream handling, connectors, and control-plane style data tooling with deliberate resource, dependency, and concurrency management.

## When to Use

- building ingestion or connector services in `Java`
- implementing JVM-based stream processors or integration utilities
- exposing data-platform metadata, contract, or control services
- managing `Maven` or `Gradle` builds for data-related services
- debugging resource, thread, serialization, or connection-pool behavior in JVM services

# Kafka Resilience And Schema Evolution

## Overview

Generic streaming guidance is not enough for production Kafka. Agents routinely introduce breaking schema changes, under-provisioned durability settings, and missing poison-message isolation. This skill mandates enforceable broker, producer, consumer, and registry guardrails before any production change ships.

## When to Use

- creating or modifying `Kafka` topics, producers, or consumers
- setting or changing schema registry compatibility policies
- designing dead-letter queue (DLQ) routing for poison pill messages
- hardening producer durability (`acks`, retries, idempotence)
- reviewing consumer lag, replay, or failover behavior on Kafka-backed pipelines

# lakeFS And Data Versioning

## Overview

Use this skill when datasets need branch-like safety and controlled promotion. It helps agents treat data versioning as an operational control for risky changes, validation before publish, environment isolation, and safe rollback for lakehouse operations.

## When to Use

- validating data changes in isolation before merging to the main branch
- branching lake data for experiments, migrations, or schema changes
- promoting data states between environments (dev → staging → production)
- implementing safe rollback for risky pipeline changes
- running A/B comparisons between data versions
- creating reproducible dataset snapshots for compliance or debugging

# Lakehouse Table Format Engineering

## Overview

Use this skill when the storage layer itself is an architecture decision. It helps agents reason about `Iceberg`, `Delta`, `Hudi`, table evolution, partitioning, compaction, snapshot behavior, and how batch and streaming jobs interact with shared lakehouse tables.

## When to Use

- choosing between `Iceberg`, `Delta`, and `Hudi`
- designing lakehouse datasets with update or merge behavior
- changing partition strategy, snapshot retention, or compaction behavior
- integrating `Spark`, `Flink`, or warehouse readers with shared table formats
- implementing CDC-style or incremental lakehouse pipelines

# Lineage, PII, And Governance

## Overview

Governance is an engineering concern, not a cleanup exercise. This skill ensures that every meaningful data change accounts for ownership, lineage, access, and sensitive-data handling before release.

## When to Use

- publishing a new table, model, or stream
- changing business-critical metrics
- handling personal, financial, health, or otherwise sensitive data
- modifying access controls or data-sharing patterns
- changing upstream or downstream lineage

# Lower Environment Data Masking And Obfuscation

## Overview

Use this skill when lower environments need production-like data but direct production copies would create privacy, security, or audit risk. It helps agents define masking, tokenization, subsetting, access controls, and refresh behavior for non-production use.

## When to Use

- seeding development, QA, or staging with production-like data
- preparing masked lower-environment refresh workflows
- obfuscating personal, financial, health, or otherwise sensitive values
- creating safe subsets of large production datasets
- validating lower-environment data refresh pipelines and controls

# Mainframe Modernization And Data Offload

## Overview

Use this skill when a modernization effort involves mainframe systems, batch schedules, and legacy data structures rather than only modern ETL tooling. It helps agents design offload, replication, contract mapping, coexistence, reconciliation, and cutover paths for mainframe-origin data.

## When to Use

- modernizing `COBOL`, `JCL`, `VSAM`, `IMS`, or `DB2 for z/OS` data flows
- offloading mainframe data to lakes, warehouses, or downstream APIs
- mapping copybooks and legacy record layouts into modern contracts
- designing coexistence between batch mainframe outputs and modern platforms
- planning phased cutover, replay, and reconciliation for legacy estates

# Master Data And Entity Resolution

## Overview

Use this skill when the data problem is "who or what is this really?" across multiple systems. It helps agents define canonical entities, matching logic, survivorship rules, and downstream-safe resolution behavior.

## When to Use

- building customer, product, or account golden records
- resolving duplicate identities across systems
- defining master data domains
- publishing canonical reference datasets

Do not collapse entity resolution into ad hoc joins if the result becomes a shared dependency.

# MCP Data Observability Integration

## Overview

Agents that cannot see production signals guess at fixes — scaling clusters blindly, rerunning jobs that OOM for structural reasons, or changing Kafka consumers without checking lag. This skill pairs MCP server templates with safe read-only observability workflows so diagnosis precedes mutation.

## When to Use

- diagnosing Spark OOM, shuffle skew, or stage failures
- inspecting Kafka consumer group lag before replay or consumer changes
- reviewing Airflow DAG run state, task duration drift, or failed retries
- validating warehouse publish state before reopening downstream consumers
- setting up IDE-integrated observability for data engineering agents

# Microsoft Purview And Azure Data Governance

## Overview

Use this skill when `Microsoft Purview` is the governance control plane for `Azure` data platforms. It helps agents design metadata collections, classification strategy, scan scope, lineage expectations, and governed publish behavior across Microsoft analytics surfaces.

## When to Use

- designing `Purview` collection and ownership structure
- defining scans, classifications, glossary, and lineage expectations
- governing datasets across `ADLS`, `Synapse`, `Data Factory`, `Azure Databricks`, or `Fabric`
- improving trusted discovery and certification for shared data products
- aligning Azure-native governance with privacy, security, and release controls

# Notebook To Production Hardening

## Overview

Use this skill when a notebook has outgrown exploration and needs to become a maintainable delivery artifact. It helps agents separate experimentation from production packaging, testing, configuration, and orchestration.

## When to Use

- moving notebook logic into scheduled jobs
- hardening `Databricks` or `Jupyter` notebooks for repeated use
- extracting reusable logic from cells into modules or packages
- improving testability and deployment discipline

Do not treat a manually rerun notebook as production just because it worked once.

# OpenMetadata DataHub And OpenLineage

## Overview

Use this skill when metadata and lineage must be operationalized through open tooling such as `OpenMetadata`, `DataHub`, or `OpenLineage`. It helps agents align producers with catalog expectations, ensure lineage accuracy, and make discovery trustworthy rather than decorative.

## When to Use

- integrating metadata platforms into delivery workflows
- improving dataset discovery and trust signals
- capturing and validating lineage across pipelines
- publishing governed datasets into open catalog ecosystems
- connecting `OpenLineage` events from Spark, Airflow, or dbt to a catalog
- defining metadata governance policies for multi-team environments

# Operational Datastore Selection Relational And NoSQL

## Overview

Use this skill when the real decision is not warehouse modeling, but what operational datastore should hold state, serve requests, or back data-driven application flows. It helps agents choose intentionally between relational systems such as `MySQL` and `PostgreSQL` versus `NoSQL` families such as document, key-value, wide-column, or cache-oriented stores.

## When to Use

- choosing `MySQL` versus `NoSQL` for application-facing or service-facing data
- selecting stores for metadata, idempotency keys, session state, event state, or operational APIs
- deciding whether flexible schema or strict relational constraints matter more
- evaluating consistency, scale, and access-pattern trade-offs
- preventing analytics or pipeline systems from being misused as OLTP stores

# Orchestration And Backfills

## Overview

Reliable data systems are shaped as much by rerun behavior as by transformation logic. This skill ensures schedules, retries, and backfills are deliberate, safe, and reversible.

## When to Use

- adding or changing a scheduled pipeline
- designing retries or failure handling
- introducing a new dependency chain
- reprocessing historical data
- modifying publish windows or cutover behavior

# Pipeline Planning And Task Breakdown

## Overview

Turn a validated data specification into a sequence of small tasks that can be implemented and verified independently. A good plan protects the team from wide, risky pipeline edits.

## When to Use

- multi-step ingestion or modeling work
- changes involving orchestration, infrastructure, and transformations
- work that needs coordination across contracts, checks, and deployment

Do not use this when the task is a single isolated change with obvious verification.

# Privacy, Retention, And Right To Delete

## Overview

Use this skill when the data platform must respect privacy obligations as an engineering behavior, not a policy slide. It helps agents make retention, deletion, masking, and minimization explicit across storage, transformations, and publishes.

## When to Use

- handling personal or regulated data
- defining retention rules
- implementing deletion or erasure requests
- changing how sensitive fields are stored, copied, or published
- validating that downstream systems do not retain data longer than allowed

# Python Data Engineering And Pipeline Packaging

## Overview

Use this skill when `Python` is the main implementation language for data pipelines or operational data tooling. It helps agents structure jobs as maintainable packages instead of loose scripts, choose the right execution boundary, manage dependencies explicitly, and keep runtime behavior testable and production-safe.

## When to Use

- building or modifying `Python` data pipelines
- packaging `PySpark`, ingestion, validation, or orchestration helper code
- moving from notebooks or scripts into production-ready modules
- managing dependency, environment, and runtime issues in `Python`
- adding CLI entry points, test harnesses, or local development workflows

# Regional Data Compliance And Sovereignty

## Overview

Use this skill when data obligations differ by jurisdiction, sector, or supervisory authority. It helps agents translate legal and policy requirements into engineering decisions for storage location, transfer boundaries, access, retention, deletion, and audit evidence. This skill is not legal advice; it is the engineering planning layer that should work with legal, privacy, and compliance owners.

## When to Use

- operating across `Europe`, the `USA`, `India`, `Saudi Arabia`, or other multi-jurisdiction footprints
- handling data residency, sovereignty, or cross-border transfer constraints
- designing controls for `GDPR`, `PDPL`, `DPDP`, state privacy, or sector-specific obligations
- working in regulated sectors where local supervisors such as `SAMA` add extra controls
- preparing localized retention, deletion, consent, or breach-evidence paths

# Reverse ETL And Operational Data Serving

## Overview

Use this skill when trusted analytical data needs to move back into operational systems. It helps agents design stable outbound contracts, idempotent syncs, destination-aware quality rules, and failure-safe activation pipelines.

## When to Use

- syncing warehouse outputs into SaaS tools
- publishing segments, scores, or entity attributes to operational systems
- designing outbound APIs or activation datasets
- changing operationally consumed data contracts

Do not treat reverse ETL like a simple export job. Operational destinations have side effects and user-facing impact.

# Safe Backfill And Replay Orchestration

## Overview

Backfills are among the highest-risk data engineering operations. Agents that rerun jobs without bounded windows, publish gates, or reconciliation checks routinely cause double-counting, corrupted metrics, and downstream trust loss. This skill blocks execution until a written replay plan, rollback path, and validation gates exist.

## When to Use

- running `/backfill` or any historical reprocessing request
- repairing a failed or partial publish window
- replaying events or partitions after an incident
- changing transformation logic that requires recomputation
- cutting over to new pipeline behavior with overlapping data windows

# Scala Data Engineering On JVM Runtimes

## Overview

Use this skill when `Scala` is the implementation language for distributed or streaming data systems. It helps agents manage build compatibility, packaging, JVM dependency issues, typed data models, serialization behavior, and runtime assumptions that often cause production failures long after code compiles.

## When to Use

- building or modifying `Scala` `Spark` jobs
- implementing `Flink`, `Kafka Streams`, or JVM-native data processors in `Scala`
- managing `sbt` builds, shaded JARs, or runtime compatibility
- choosing typed datasets, encoders, or JVM serialization strategies
- debugging classpath, version, or packaging problems in distributed runtimes

# Schema Evolution And Contract Migrations

## Overview

Use this skill when a schema change can ripple through pipelines, models, dashboards, or external consumers. It helps agents make schema evolution deliberate, compatible, and recoverable.

## When to Use

- changing input or output schemas
- renaming or dropping columns
- changing data types or nullability
- adding keys or changing grain
- migrating consumers from an old contract to a new one

# Semantic Layer And Metric Governance

## Overview

Use this skill when the challenge is consistency of meaning, not just correctness of rows. It helps agents define shared metrics and dimensions so teams stop rebuilding conflicting logic in every downstream tool.

## When to Use

- defining or changing business metrics
- introducing a semantic or metric layer
- standardizing dimensions and filters across teams
- governing high-value shared KPIs

Do not rely on ad hoc BI formulas when a metric needs cross-team trust.

# Snowflake Native Pipelines And Governance

## Overview

Use this skill when `Snowflake` is not just the storage target but the primary pipeline and governance surface. It helps agents reason about native ingestion, incremental patterns, task orchestration, governance controls, warehouse sizing, and secure publish paths inside Snowflake.

## When to Use

- building warehouse-native incremental pipelines in `Snowflake`
- designing with `Streams`, `Tasks`, or `Dynamic Tables`
- evaluating `Snowpipe`, `Snowpark`, or external orchestration boundaries
- defining masking, row access, tags, and secure sharing in the same workflow
- reviewing publish readiness for Snowflake-native data products

# Source Reliability And Extraction Resilience

## Overview

Use this skill when the real risk is upstream instability rather than downstream modeling. It helps agents design safe ingestion behavior around outages, late data, flaky responses, inconsistent source states, and operational dependencies.

## When to Use

- unstable upstream systems
- intermittent extraction failures
- delayed source availability
- rate-limited or timeout-prone sources
- ingestion designs that must survive source incidents

# Spark And Distributed Processing

## Overview

Use this skill for batch workloads that exceed single-node processing and need distributed execution discipline. It helps agents reason about `Spark`, managed Spark services such as `Glue` and `EMR`, partitioning, joins, storage layout, and failure-safe processing.

## When to Use

- building or changing `Spark` jobs
- choosing between `Spark`, `Glue`, `EMR`, or smaller engines
- debugging expensive joins, skew, shuffle, or memory issues
- designing batch pipelines over large files or partitioned tables
- implementing transformations against lakehouse tables such as `Iceberg`, `Delta`, or `Hudi`

# Spark Serverless Reliability And State Management

## Overview

Serverless Spark hides cluster failures behind short-lived runtimes and opaque retries. Silent partial writes, orphaned checkpoints, and timeout-induced corruption are common when agents treat serverless Spark like a long-running cluster job. This skill forces explicit state boundaries, resumable progress, and cleanup before any publish path opens.

## When to Use

- implementing or reviewing Spark on `AWS Lambda`, `AWS Glue` serverless, or comparable short-lived runtimes
- designing S3-backed checkpoints, progress markers, or staged write paths
- recovering from timeout, OOM, or mid-batch partial failure without double-counting
- translating IceGuard-style checkpoint and rollback patterns into agent workflows
- hardening batch Spark that must survive cold starts, memory limits, and hard execution ceilings

# Streaming And Messaging Systems

## Overview

Use this skill for continuous data movement and real-time processing. It helps agents design safe event pipelines around brokers and stream processors such as `Kafka`, `Kinesis`, and `Flink`, with attention to ordering, state, replay, schema evolution, and delivery guarantees.

## When to Use

- designing or modifying `Kafka` topics, consumers, or stream processors
- using `Kinesis` for event ingestion or streaming delivery
- implementing `Flink` jobs or stateful stream transformations
- defining near-real-time pipelines, windows, watermarks, and replay behavior
- handling schemas and contracts for event-driven systems

# Superset And Metrics Serving

## Overview

Use this skill when `Apache Superset` or a similar BI serving surface is the final consumer layer. It helps agents keep chart-ready data aligned with governed metrics, prevent dashboard drift, and manage the boundary between analytical modeling and visualization safely.

## When to Use

- publishing governed datasets into `Superset` for dashboard consumption
- aligning dashboard metrics with centralized semantic definitions
- preventing BI-layer drift from governed metric definitions
- managing access control and row-level security for dashboard datasets
- designing chart-ready schemas that optimize Superset query performance
- operating Superset as part of a broader data platform

# Terraform And Data Platform Infrastructure

## Overview

Use this skill when data engineering work depends on platform infrastructure, not just SQL or code. It helps agents make environment setup reproducible, reviewable, and promotion-safe.

## When to Use

- provisioning buckets, warehouses, jobs, roles, or secrets
- changing network or access boundaries
- creating environment-specific data platform resources
- standardizing dev, staging, and production infrastructure

Do not treat manual console changes as equivalent to managed infrastructure.

# Test Data Preparation And Synthetic Data

## Overview

Use this skill when data work needs realistic test inputs without depending on unsafe copies of production. It helps agents decide when to use masked subsets, synthetic data, seeded fixtures, or contract-shaped test datasets.

## When to Use

- building lower-environment validation datasets
- preparing integration-test or QA data
- creating representative demo or training datasets
- generating synthetic data that preserves shape and edge cases
- defining seeded fixtures for pipelines, dbt, or Spark jobs

# Trino Presto Federated Query

## Overview

Use this skill when `Trino` or `Presto` sits across multiple data systems as a federated query layer. It helps agents reason about federation boundaries, predicate pushdown limits, consistency trade-offs, performance planning, and governed consumption patterns.

## When to Use

- querying across multiple heterogeneous data stores with `Trino` or `Presto`
- defining governed federated analytics access for users or applications
- managing performance, cost, and semantic consistency across systems
- building a query layer over lakehouse tables, relational databases, and object stores
- planning catalog and connector configuration for multi-source environments

# Unity Catalog And Lakehouse Governance

## Overview

Use this skill when `Unity Catalog` is the core governance surface for a `Databricks` platform. It helps agents design governed boundaries across catalogs, schemas, storage locations, shares, and access paths for lakehouse data products.

## When to Use

- designing `Unity Catalog` hierarchy and ownership
- defining external locations, credentials, and storage boundaries
- governing shared `Delta` tables, views, volumes, or models
- reviewing cross-workspace or multi-team lakehouse governance
- planning secure sharing, publish approvals, or governed consumer access

# Using Data Agent Skills

## Overview

This legacy entry point remains for compatibility with older setups. The preferred start-here skill is `using-data-engineering-agent-skills`, which provides the current classification flow, preset mapping, example routing, and next-command guidance.

## When to Use

- loading an older project that still points to this skill
- triaging a request before switching to the current entry skill
- preserving compatibility for previously installed rule bundles

Do not use this as the long-term default. Switch to `skills/using-data-engineering-agent-skills/SKILL.md` once the session starts.

# Using Data Engineering Agent Skills

## Overview

Start here before changing code, SQL, orchestration, contracts, or infrastructure. This skill maps the user request to the right data engineering workflow so the agent does not skip specification, quality, governance, replay, or rollback thinking.

## When to Use

- starting a new session
- deciding which skill should lead execution
- translating a vague request into the right workflow
- choosing the best preset, starter pack, or runnable example
- deciding the safest next command

# Warehouse And Schema Design

## Overview

Use this skill when the primary challenge is how data should be modeled for reliable analytics consumption. It helps agents choose good grain, keys, relationships, and serving patterns so downstream work stays understandable and performant.

## When to Use

- designing marts, warehouse schemas, or curated serving tables
- choosing fact and dimension boundaries
- deciding grain, surrogate keys, or relationship strategy
- balancing normalization and denormalization
- restructuring analytics-facing datasets for usability

# Warehouse Performance And Cost Optimization

## Overview

Use this skill when data is correct but too slow or too expensive. It helps agents treat performance and cost as measurable design concerns rather than guesswork.

## When to Use

- slow warehouse queries
- runaway compute or scan cost
- poor partitioning or clustering choices
- overloaded workloads competing on shared compute
- repeated complaints about expensive marts or dashboards