Individual skill
senior-data-engineer
World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.
tasks/flink-query/environment/skills/senior-data-engineer/SKILL.md
Source commit f860ecf02ef53412a030f8661c83d644b739ff01
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- No
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Not checked yet
- Last checked
- Nov 3, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 1,906
- Repository last updated
- Nov 4, 2026
- License
- Apache-2.0
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- benchmark Repository topic
- skillsbench Repository topic
- architecture Declared skill tag
- data Declared skill tag
- design Declared skill tag
- engineer Declared skill tag
- engineering Declared skill tag
- senior Declared skill tag
- streaming Declared skill tag
- kafka Declared skill tag
- flink Declared skill tag
- real-time Declared skill tag
- engineering Derived category
- data Derived category
- design Derived category
- devops Derived category
- data-databases Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
HOW_TO_USE.mddoc · 2,500 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesSKILL.mdcore · 23,665 bytesassets/.gitkeepasset · 0 bytesassets/data-quality-checklist.mddoc · 0 bytesassets/pipeline-design-template.mddoc · 0 bytesreferences/frameworks.mddoc · 42,313 bytesreferences/templates.mddoc · 67,136 bytesreferences/tools.mddoc · 39,140 bytesscripts/data_quality_validator.pyruns · 3,489 bytesscripts/etl_performance_optimizer.pyruns · 3,504 bytesscripts/kafka_config_generator.pyruns · 41,177 bytesscripts/pipeline_orchestrator.pyruns · 3,487 bytesscripts/stream_processor.pyruns · 60,553 bytesscripts/streaming_quality_validator.pyruns · 40,005 bytes
Read excerpt
# Senior Data Engineer
## Core Capabilities
- **Batch Pipeline Orchestration** - Design and implement production-ready ETL/ELT pipelines with Airflow, intelligent dependency resolution, retry logic, and comprehensive monitoring
- **Real-Time Streaming** - Build event-driven streaming pipelines with Kafka, Flink, Kinesis, and Spark Streaming with exactly-once semantics and sub-second latency
- **Data Quality Management** - Comprehensive batch and streaming data quality validation covering completeness, accuracy, consistency, timeliness, and validity
- **Streaming Quality Monitoring** - Track consumer lag, data freshness, schema drift, throughput, and dead letter queue rates for streaming pipelines
- **Performance Optimization** - Analyze and optimize pipeline performance with query optimization, Spark tuning, and cost analysis recommendations
## Key Workflows
### Workflow 1: Build ETL Pipeline
