Streaming Watermark & Late-Event Designer

Design event-time handling for a production streaming job. Treat processing time as a last resort. Idle sources and sparse keys must not freeze windows forever or drop money-relevant events silently. Processor: [Flink / Spark Structured Streaming / Kafka Streams / Beam] Event time field and clock source: [producer, broker, ingestion] Expected disorder: [p95/p99 delay, max observed lateness] Window type and size: [tumbling / sliding / session] Allowed lateness vs business cutoff: [duration] Key cardinality and idle keys: [details] Downstream sink semantics: [retractable, upsert, append-only] Current job config or code: [Paste watermark, window, and sink settings] Provide: 1. Event-time vs processing-time decision with the distortion each creates 2. Watermark formula tied to measured disorder, not a copied default 3. Allowed lateness, side-output/late-data path, and what happens after the cutoff 4. Idle-source and sparse-key watermark progress strategy 5. Out-of-order, duplicate, and timestamp-skew counterexamples 6. Retract vs overwrite vs append implications for the sink 7. Checkpoint alignment so a restart does not reopen already-emitted windows incorrectly 8. Metrics and alerts for watermark lag, dropped late events, and stalled progress 9. Load-test cases: burst, idle gap, clock jump, and replay 10. Explicit statement of which events are unrecoverable after the watermark advances If max lateness is unknown, design a measurement job first. Do not invent a 5-minute watermark to look complete.