Data Engineering AI prompts

Data Engineering

Audit a change-data-capture pipeline for gaps, duplicates, reordering, and recovery risks.

Data Engineering

Plan a backward-compatible schema change across producers, pipelines, and consumers.

Data Engineering

Design a safe, restartable backfill without overwhelming production systems.

Data Engineering

Size a streaming platform for peak traffic, ordering, lag, and failure recovery.

Data Engineering

Determine whether a streaming system delivers real end-to-end exactly-once outcomes.

Data Engineering

Diagnose stragglers, spills, OOMs, and expensive shuffles in large Spark jobs.

Data Engineering

Fix small-file amplification while preserving ingestion latency and safe concurrency.

Data Engineering

Diagnose why a warehouse query suddenly got slower or more expensive, with likely causes, checks to run, and practical remediations.

Data Engineering

Design auditable reconciliation controls for revenue, payments, or ledger pipelines.

Data Engineering

Turn vague data reliability expectations into measurable SLOs and error budgets.

Data Engineering

Find sensitive-data exposure and design least-privilege controls across a data platform.

Data Engineering

Build and test recovery plans for regional outages, corruption, and credential compromise.

Data Engineering

Review an Airflow DAG for idempotency, scheduler pressure, backfills, and safe recovery.

Data Engineering

Prove a dbt incremental model handles updates, late data, deletes, and full refreshes.

Data Engineering

Detect point-in-time leakage and training-serving skew in feature pipelines.

Data Engineering

Design globally resilient event ingestion with clear consistency and failover semantics.

Data Engineering

Plan a low-risk warehouse migration with measurable parity and rollback gates.

Data Engineering

Turn legal, business, and cost constraints into enforceable lifecycle policies.

Data Engineering

Build a layered test suite that catches transformation, contract, and replay defects before production.

Data Engineering

Create a low-ambiguity runbook for diagnosing and restoring critical data pipelines.

Data Engineering

Find expensive queries and recommend practical warehouse cost reductions.

Data Engineering

Run a structured root-cause analysis for broken pipelines or bad dashboards.

Data Engineering

Design event topics, keys, and schemas for reliable streaming pipelines.

Data Engineering

Document end-to-end lineage from source systems to dashboards or ML features.

Data Engineering

Choose an incremental load pattern—merge, append, or CDC—and spell out keys, late data handling, and warehouse-safe implementation steps.

Data Engineering

Draft a producer–consumer data contract covering schema, freshness, ownership, and failure expectations for a shared dataset.

Data Engineering

Write efficient PySpark transformations with partitioning and performance notes.

Data Engineering

Design a star or snowflake schema with fact grain, measures, dimensions, and SCD recommendations for a defined analytics process.

Data Engineering

Scaffold a clean Airflow DAG with retries, sensors, and clear task boundaries.

Data Engineering

Define practical data quality checks and thresholds for a critical table.

Data Engineering

Design a reliable batch or streaming data pipeline from source to warehouse.

Data Engineering

Generate production-ready dbt models with tests, docs, and naming conventions.