Individual skill
spark-engineer
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
skills/spark-engineer/SKILL.md
Source commit 078a25cc5bfd8011459be7b91d004e900c1d02d3
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- Yes
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Network or broad shell
- Last checked
- Nov 9, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 12,327
- Repository last updated
- Oct 30, 2026
- License
- MIT
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- ai-agents Repository topic
- claude Repository topic
- claude-code Repository topic
- claude-marketplace Repository topic
- claude-skills Repository topic
- data Derived category
- operations Derived category
- writing Derived category
- testing Derived category
- data-databases Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
SKILL.mdcore · 6,714 bytesSKILL.mdcore · 6,714 bytesSKILL.mdcore · 6,714 bytesSKILL.mdcore · 6,714 bytesSKILL.mdcore · 6,714 bytesreferences/partitioning-caching.mddoc · 14,840 bytesreferences/performance-tuning.mddoc · 15,875 bytesreferences/rdd-operations.mddoc · 16,382 bytesreferences/spark-sql-dataframes.mddoc · 13,842 bytesreferences/streaming-patterns.mddoc · 20,501 bytes
Read excerpt
# Spark Engineer
Senior Apache Spark engineer specializing in high-performance distributed data processing, optimizing large-scale ETL pipelines, and building production-grade Spark applications.
## Core Workflow
1. **Analyze requirements** - Understand data volume, transformations, latency requirements, cluster resources
2. **Design pipeline** - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities
3. **Implement** - Write Spark code with optimized transformations, appropriate caching, proper error handling
4. **Optimize** - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations
5. **Validate** - Check Spark UI for shuffle spill before proceeding; verify partition count with `df.rdd.getNumPartitions()`; if spill or skew detected, return to step 4; test with production-scale data, monitor resource usage, verify performance targets
## Reference Guide
