ETL / ELT Pipeline Designer

Customizable · 6 variables

Design a reliable batch or streaming pipeline from source to warehouse, including load pattern, idempotency, and failure handling.

Prompt

Act as a data platform engineer designing an ETL/ELT pipeline. Prefer the smallest operable design. Do not invent source APIs, SLAs, or vendor features that are not in the brief. Source system: [API / Database / Files / Event stream] Destination: [Warehouse / Lakehouse] Freshness SLA: [e.g. hourly / daily / near-real-time] Volume estimate: [rows/day or GB/day] Stack preference: [Airflow + dbt / Spark / Fivetran + dbt / custom] Constraints: [PII, region, budget, or "none"] Produce: 1. High-level architecture (ingestion → transform → serve) 2. Recommended load pattern (full / incremental / CDC) and why it fits this source 3. Idempotency and late-arriving data strategy 4. Failure modes, retries, and what must not double-write 5. Monitoring / alerting checklist 6. Minimal DAG or job outline 7. Assumptions and facts you still need If volume or SLA is missing, state a default and label it as an assumption.

LLM

Customize this prompt

6 variables

Progress: 0 of 6 variables filled0%

1Fill Variables

API / Database / Files / Event stream*

Warehouse / Lakehouse*

e.g. hourly / daily / near-real-time*

rows/day or GB/day*

Airflow + dbt / Spark / Fivetran + dbt / custom*

PII, region, budget, or "none"*

2Generated Prompt

Act as a data platform engineer designing an ETL/ELT pipeline. Prefer the smallest operable design. Do not invent source APIs, SLAs, or vendor features that are not in the brief. Source system: [API / Database / Files / Event stream] Destination: [Warehouse / Lakehouse] Freshness SLA: [e.g. hourly / daily / near-real-time] Volume estimate: [rows/day or GB/day] Stack preference: [Airflow + dbt / Spark / Fivetran + dbt / custom] Constraints: [PII, region, budget, or "none"] Produce: 1. High-level architecture (ingestion → transform → serve) 2. Recommended load pattern (full / incremental / CDC) and why it fits this source 3. Idempotency and late-arriving data strategy 4. Failure modes, retries, and what must not double-write 5. Monitoring / alerting checklist 6. Minimal DAG or job outline 7. Assumptions and facts you still need If volume or SLA is missing, state a default and label it as an assumption.

ETL / ELT Pipeline Designer

Customize this prompt

1Fill Variables

2Generated Prompt

Comments

Details