Individual skill
fabric-pyspark-perf-remediate
Diagnose and resolve Apache Spark performance issues in Microsoft Fabric notebooks and Spark Job Definitions. Use when PySpark jobs are slow, notebooks take too long, Spark stages are skewed, shuffles are excessive, out-of-memory errors occur, Delta Lake writes are slow, or Fabric capacity is throttled. Covers data skew, shuffle optimization, broadcast joins, partition tuning, VOrder, Optimized Write, resource profiles, autotune, native execution engine, small file compaction, and Spark UI interpretation. Keywords include slow notebook, OOM, spill, shuffle, skew, broadcast, repartition, coalesce, OPTIMIZE, VACUUM, Z-ORDER, checkpoint, cache, persist, executor memory, driver memory, spark.sql.shuffle.partitions, autoBroadcastJoinThreshold, maxPartitionBytes, Fabric capacity throttling, CU utilization.
skills/fabric-pyspark-perf-remediate/SKILL.md
Source commit cee03dc1199cd2a9765b439fa41930d480e5989d
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- No
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Not checked yet
- Last checked
- Nov 6, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 17
- Repository last updated
- Jun 7, 2026
- License
- MIT
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- claude Repository topic
- claude-skills Repository topic
- copilot Repository topic
- copilot-skills Repository topic
- fabric Repository topic
- microsoft-fabric Repository topic
- data Derived category
- design Derived category
- data-databases Derived category
- design-interface Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
LICENSE.txtdoc · 11,358 bytesSKILL.mdcore · 9,802 bytesSKILL.mdcore · 9,802 bytesreferences/data-skew-resolution.mddoc · 7,194 bytesreferences/delta-table-optimization.mddoc · 9,967 bytesreferences/fabric-compute-tuning.mddoc · 7,552 bytesreferences/memory-and-spill-tuning.mddoc · 8,411 bytesreferences/shuffle-and-join-optimization.mddoc · 7,282 bytesreferences/spark-ui-guide.mddoc · 7,651 bytesscripts/delta_table_health.pyruns · 7,021 bytesscripts/identify_skew.pyruns · 6,216 bytesscripts/spark_health_check.pyruns · 6,783 bytestemplates/perf_baseline_template.pyruns · 6,161 bytes
Read excerpt
# Microsoft Fabric PySpark Performance remediate
Systematic guide for diagnosing and resolving Apache Spark performance problems in Microsoft Fabric Data Engineering workloads, including notebooks, Spark Job Definitions, and pipeline activities.
## When to Use This Skill
Activate when encountering any of these scenarios:
- PySpark notebook cells take unexpectedly long to execute
- Spark Job Definitions exceed expected duration or fail with timeouts
- Out-of-memory (OOM) errors on driver or executors
- Excessive shuffle read/write in Spark UI stage details
- Data skew causing individual tasks to run much longer than peers
- Delta Lake table writes are slow or produce many small files