Design watermarks, allowed lateness, and side outputs so windowed streaming jobs stay correct when events arrive late, out of order, or after idle gaps.

streamingwatermarks+2

SaaS API Ingestion Reliability Designer

Data Engineering

LLM

Design a restartable SaaS API extractor that survives pagination, rate limits, partial pages, and vendor cursor resets without duplicating or skipping records.

ingestionapis+2

Iceberg Snapshot Isolation Reviewer

Data Engineering

LLM

Review Iceberg or Delta snapshot, isolation, and planning behavior so concurrent writers, time travel, and compaction cannot serve torn or expired data.

iceberglakehouse+2

Semantic Metrics Layer Auditor

Data Engineering

LLM

Audit a metrics or semantic layer so revenue, conversion, and other executive numbers have one grain, one join path, and no silent double-counting.

metricssemantic-layer+2

Data Platform FinOps Attribution

Data Engineering

LLM

Attribute warehouse, lake, and streaming spend to teams and data products with guardrails that cut cost without silently dropping SLAs.

finopscost+2

Entity Resolution Pipeline Designer

Data Engineering

LLM

Design a deterministic-then-probabilistic identity pipeline with reversible merges, audit trails, and replay when source keys collide or split.

identityentity-resolution+2

Schema Registry Compatibility Gate

Data Engineering

LLM

Build CI and runtime gates so Avro, Protobuf, or JSON Schema changes cannot break consumers, sinks, or warehouse loaders after they ship.

schema-registrycompatibility+2

Object-Store Landing Zone Ingestor

Data Engineering

LLM

Design a durable S3/GCS/ADLS landing zone that handles late files, partial uploads, duplicate drops, and schema-mixed folders without poisoning downstream tables.

ingestions3+2

CDC Pipeline Correctness Auditor

Data Engineering

LLM

Audit a change-data-capture pipeline for gaps, duplicates, reordering, and recovery risks.

cdccorrectness+2

Zero-Downtime Schema Evolution Planner

Data Engineering

LLM

Plan a backward-compatible schema change across producers, pipelines, and consumers, with an expand-migrate-contract order and a rollback path.

schema-evolutiondata-contracts+2

Large-Scale Backfill Planner

Data Engineering

LLM

Design a safe, restartable backfill that does not overwhelm production systems or violate SLAs.

backfillidempotency+2

Streaming Capacity & Partition Planner

Data Engineering

LLM

Size a streaming platform for peak traffic, ordering, lag, and failure recovery, with formulas you can defend and a load-test plan.

streamingkafka+2

Exactly-Once Semantics Auditor

Data Engineering

LLM

Determine whether a streaming system delivers real end-to-end exactly-once outcomes, or only a framework marketing claim.

streamingexactly-once+2

Spark Skew & Shuffle Diagnostician

Data Engineering

LLM

Diagnose stragglers, spills, OOMs, and expensive shuffles in large Spark jobs with concrete next checks.

sparkperformance+2

Lakehouse Small-Files Optimizer

Data Engineering

LLM

Fix lakehouse small-file amplification while preserving ingestion latency, query scan cost, and safe concurrent writers.

lakehouseiceberg+2

Warehouse Query Regression Investigator

Data Engineering

LLM

Diagnose why a warehouse query suddenly got slower or more expensive, with likely causes, checks to run, and practical remediations.

sqlwarehouse+2

Financial Data Reconciliation Designer

Data Engineering

LLM

Design auditable reconciliation controls for revenue, payments, or ledger pipelines, without silently netting unexplained gaps.

reconciliationfinance+2

Data SLO & Error Budget Designer

Data Engineering

LLM

Turn vague data reliability expectations into measurable SLOs, error budgets, and burn-rate alerts consumers can use.

sloobservability+2

PII Governance & Access Control Reviewer

Data Engineering

LLM

Find sensitive-data exposure and design least-privilege controls across a data platform.

piigovernance+2

Data Platform Disaster Recovery Planner

Data Engineering

LLM

Build and test recovery plans for regional outages, corruption, and credential compromise.

disaster-recoveryrpo+2

Airflow Production Reliability Review

Data Engineering

LLM

Review an Airflow DAG for idempotency, scheduler pressure, backfills, and safe recovery.

airflowdag-review+2

dbt Incremental Model Correctness Review

Data Engineering

LLM

Prove a dbt incremental model handles updates, late data, deletes, and full refreshes, with fixtures that would catch silent drift.

dbtincremental+2

ML Feature Pipeline Leakage Auditor

Data Engineering

LLM

Detect point-in-time leakage and training-serving skew in feature pipelines before they ship to models.

feature-engineeringml+2

Multi-Region Event Ingestion Architect

Data Engineering

LLM

Design globally resilient event ingestion with clear consistency and failover semantics.

multi-regionstreaming+2

Legacy Warehouse Migration Planner

Data Engineering

LLM

Plan a low-risk warehouse migration with measurable parity checks and explicit rollback gates.

migrationwarehouse+2

Data Retention & Lifecycle Architect

Data Engineering

LLM

Turn legal, business, and cost constraints into enforceable data lifecycle and retention policies.

retentionlifecycle+2

Data Pipeline Test Strategy

Data Engineering

LLM

Build a layered test suite that catches transformation, contract, and replay defects before production.

testingdata-quality+2

Pipeline On-Call Runbook Builder

Data Engineering

LLM

Create a low-ambiguity on-call runbook for diagnosing and restoring critical data pipelines, including what not to retry blindly.

on-callrunbook+2

Warehouse Cost Optimizer

Data Engineering

LLM

Find expensive warehouse queries and recommend practical cost reductions, from rewrites to materializations, without changing results.

costperformance+2

Data Incident RCA Assistant

Data Engineering

LLM

Run a structured root-cause analysis for broken pipelines or bad dashboards, with checks, mitigations, and what not to guess.

incidentrca+2

Kafka Event Schema Designer

Data Engineering

LLM

Design event topics, keys, and schemas for reliable streaming pipelines, including compatibility and dead letters.

kafkastreaming+2

Data Lineage Documenter

Data Engineering

LLM

Document end-to-end lineage from source systems to dashboards or features, including owners and blast radius.

lineagedocumentation+2

Incremental Load Strategist

Data Engineering

LLM

Choose an incremental load pattern (merge, append, or CDC) and spell out keys, late data handling, and warehouse-safe implementation steps.

incrementalcdc+3

Data Contract Writer

Data Engineering

LLM

Draft a producer-consumer data contract covering schema, freshness, ownership, and failure expectations for a shared dataset.

data-contractsgovernance+2

Spark Transformation Helper

Data Engineering

LLM

Write efficient PySpark transformations with partitioning, skew notes, and a way to validate row counts without changing grain.

sparkpyspark+2

Dimensional Model Designer

Data Engineering

LLM

Design a star or snowflake schema with fact grain, measures, dimensions, and SCD recommendations for a defined analytics process.

dimensional-modelingstar-schema+2

Airflow DAG Scaffolder

Data Engineering

LLM

Scaffold a clean Airflow DAG with retries, timeouts, and task boundaries an operator can run without guessing hidden state.

airfloworchestration+2

Data Quality Test Suite

Data Engineering

LLM

Define practical data-quality checks and thresholds for a critical table, including fail vs warn severity and owner next steps.

data-qualitytesting+2

ETL / ELT Pipeline Designer

Data Engineering

LLM

Design a reliable batch or streaming pipeline from source to warehouse, including load pattern, idempotency, and failure handling.

etlelt+3

dbt Model Generator

Data Engineering

LLM

Generate a production-ready dbt model with tests, docs, and naming that match the stated warehouse grain, without inventing columns.

dbtanalytics-engineering+2