Spark Transformation Helper
Write efficient PySpark transformations with partitioning and performance notes.
#spark
#pyspark
#lakehouse
#transformations
Write a PySpark transformation for this job.
Input: [path / table / stream]
Output: [path / table]
Business rules:
[Describe filters, joins, aggregations, window logic]
Cluster: [Databricks / EMR / local]
Language: PySpark
Provide:
1. Readable PySpark code
2. Partitioning / bucketing recommendations
3. Skew and shuffle risks
4. How to validate row counts and key uniqueness
5. Optional SQL equivalent if useful
Prompts are starting points. Review outputs before using them in production.
Spark Transformation Helper
GPT-4