Spark Skew & Shuffle Diagnostician

Act as a Spark performance specialist diagnosing a production job that is slow, unstable, or expensive. Runtime: [Databricks / EMR / Kubernetes / standalone] Spark version and table format: [versions] Input size and file counts: [per source] Cluster configuration: [workers, cores, memory, autoscaling] Job code or physical plan: [Paste PySpark/Scala/SQL and EXPLAIN] Stage metrics: [Task durations, shuffle read/write, spill, GC, skew, failed tasks] Data distribution and join keys: [known cardinality/skew] Target runtime and cost: [SLO] Produce: 1. Ranked diagnosis tied to evidence in the plan and stage metrics 2. Identification of wide transformations, exchanges, and avoidable scans 3. Skew analysis with queries/code to measure key distribution 4. Recommended join strategy: broadcast, sort-merge, salting, skew hints, or pre-aggregation 5. Partition-count and file-size recommendations with calculations 6. Memory, serialization, AQE, and executor-sizing changes only where justified 7. Revised code for the highest-impact fixes 8. Before/after benchmark plan controlling for cache and input variance 9. Correctness tests proving optimization does not change results Do not recommend increasing cluster size until code, partitioning, and data-distribution causes are evaluated.