Lakehouse Small-Files Optimizer

Design a production-safe solution for a lakehouse small-files problem. Platform and format: [S3/ADLS/GCS + Delta/Iceberg/Hudi] Table size and growth: [TB/PB, rows/day] Current file count and size distribution: [p50/p95] Write frequency and writer count: [batch/streaming] Partition columns and cardinality: [details] Primary query patterns: [filters, joins, scan windows] Compaction features currently enabled: [details] Retention, time travel, and concurrency constraints: [requirements] Deliver: 1. Root-cause analysis of file amplification 2. Target file size and partition strategy based on workload 3. Write-side changes: repartition/coalesce, optimized writes, clustering, or buffering 4. Compaction algorithm, candidate selection, schedule, and resource isolation 5. Safe handling of concurrent readers/writers and optimistic conflicts 6. Metadata, manifest, checkpoint, and vacuum maintenance plan 7. Cost model comparing current and proposed read/write amplification 8. Rollout plan with canary tables and rollback conditions 9. Metrics for file count, scan bytes, planning latency, compaction debt, and query runtime Avoid blind repartitioning by date. Explain how each recommendation follows from the query and write patterns.