Design ingestion from an object-store landing zone into a lakehouse or warehouse. Files are not transactions. Partial objects, overwritten keys, and vendor re-drops will happen.
Cloud store: [S3 / GCS / ADLS]
Drop pattern: [partner SFTP dump, app export, CDC files, logs]
File formats and compression: [CSV, JSON, Parquet, mixed]
Arrival SLA and late-file window: [details]
Partitioning in the prefix: [date, account, or chaotic]
Downstream: [Spark, COPY INTO, Snowpipe, BigQuery load]
Current prefix layout and job:
[Paste]
Pain: [duplicates, missing days, schema mix, or none]
Deliver:
1. Landing contract: naming, completeness marker, checksum, and immutability rules
2. Detection of complete vs partial objects (multipart, zero-byte, overwritten keys)
3. Inventory and manifest design so a run is a set of object versions, not "whatever is in the folder"
4. Dedup of re-dropped files using etag/generation plus business keys
5. Schema-on-read vs enforced schema, and quarantine for mixed schemas
6. Exactly-once-enough load into the next layer (idempotent COPY/MERGE)
7. Late-file and restated-day handling without rewriting unrelated partitions blindly
8. Security: bucket policy, encryption, and no public ACL surprises
9. Observability: expected vs arrived files, lag, poison objects
10. Replay procedure that names object versions, not "re-run yesterday"
If partners overwrite the same key, require versioning or a content-addressed prefix. A last-writer-wins folder is not a pipeline.