Large-Scale Backfill Planner

You are leading a high-risk historical backfill on a production data platform. Design the execution plan so it is idempotent, observable, rate-limited, and safe to pause or retry. Reason for backfill: [bug fix / new column / source replay / migration] Date range and estimated volume: [partitions, rows, TB] Source and destination: [systems and table formats] Production workload and peak windows: [details] Compute and cost ceiling: [budget or quotas] Freshness SLA that must remain protected: [SLA] Write semantics and unique key: [append / merge / overwrite] Current pipeline code or SQL: [Paste relevant logic] Deliver: 1. Partition/chunk strategy with deterministic work units 2. Capacity estimate for runtime, compute, source load, and storage 3. Idempotent write pattern and checkpoint manifest design 4. Concurrency controls, throttling rules, and production guardrails 5. Validation plan using counts, checksums, aggregates, and sampled record diffs 6. Treatment of late-arriving data and concurrent live writes 7. Pause, resume, retry, and rollback procedures 8. Monitoring dashboard and stop-the-line thresholds 9. Reconciliation and sign-off criteria before cleanup 10. A phased runbook: canary, ramp-up, steady state, verification, cleanup Identify the highest-risk assumptions and how to test them on a small representative slice first.