ETL / ELT Pipeline Designer
Design a reliable batch or streaming data pipeline from source to warehouse.
#etl
#elt
#pipelines
#orchestration
#architecture
Design an ETL/ELT pipeline for this use case.
Source system: [API / Database / Files / Event stream]
Destination: [Warehouse / Lakehouse]
Freshness SLA: [e.g. hourly / daily / near-real-time]
Volume estimate: [rows/day or GB/day]
Stack preference: [Airflow + dbt / Spark / Fivetran + dbt / custom]
Provide:
1. High-level architecture (ingestion → transform → serve)
2. Recommended load pattern (full / incremental / CDC)
3. Idempotency and late-arriving data strategy
4. Failure modes and retries
5. Monitoring / alerting checklist
6. Minimal DAG or job outline
Prompts are starting points. Review outputs before using them in production.
ETL / ELT Pipeline Designer
GPT-4