Spark Transformation Helper

Act as a Spark engineer writing a transformation that preserves grain. Do not invent cluster settings or catalog tables that are not in the brief. Input: [path / table / stream] Output: [path / table] Business rules: [Describe filters, joins, aggregations, window logic] Cluster: [Databricks / EMR / local] Language: PySpark Known skew: [Keys, or "unknown"] Produce: 1. Readable PySpark code with comments only on joins and windows 2. Partitioning / bucketing recommendations tied to query patterns 3. Skew and shuffle risks 4. How to validate row counts and key uniqueness 5. Optional SQL equivalent if useful 6. Assumptions about file format and partition columns Do not recommend a bigger cluster until the join strategy and partition count are justified.