Determine whether a streaming system delivers real end-to-end exactly-once outcomes.
#streaming
#exactly-once
#idempotency
#failure-testing
Prompt
Audit the following streaming pipeline's claim of "exactly once." Treat framework marketing claims as unproven until the source, processor, side effects, and sink are analyzed together.
Architecture: [source → broker → processor → sinks]
Event identity and keys: [details]
Checkpoint and state backend: [configuration]
Sink write behavior: [transactions / upserts / append]
External side effects: [APIs, notifications, secondary stores]
Failure and replay behavior observed: [details]
Business correctness requirement: [what duplication or loss means]
Return:
1. A delivery-semantics matrix for every boundary
2. Counterexamples showing where duplicates or loss can still occur
3. Assessment of source replayability and deterministic processing
4. Transaction, checkpoint, and sink-commit coordination analysis
5. Strategy for idempotent external side effects
6. Stable deduplication key and state-retention recommendation
7. Failure-injection tests for crashes before/after each commit boundary
8. Honest final classification: at-most-once, at-least-once, effectively-once, or end-to-end exactly-once
9. Minimal changes required to meet the stated business guarantee
Use a timeline for at least two difficult crash scenarios. State which guarantees fail after the deduplication retention window expires.
Customize
Fill Variables
Details
Model
Claude 3
Category
Data Engineering
Added On
Jul 12, 2026
Prompts are starting points. Review outputs before using them in production.