How do you handle late arriving events in a streaming pipeline?

Use event time rather than processing time, and define a watermark, which is the engine's heuristic estimate that no events older than a given timestamp will still arrive. Windows close when the watermark passes their end. Set an allowed lateness so slightly late events still update results, and route anything later to a side output or a reconciliation table rather than discarding it silently.

First, everything is keyed on event time, because processing time results change every time you replay and that makes them useless for reconciliation. Then the watermark is the engine's claim that it does not expect events older than this timestamp anymore, and windows finalize when it passes them. Choosing that lag is the actual design decision: a short watermark gives you fast results that are sometimes incomplete, a long one gives you correct results late. I set it from data rather than intuition. On a mobile app pipeline we measured the gap between event time and ingestion time and found the ninety ninth percentile was about four minutes, driven by phones being offline, so we set allowed lateness at ten minutes. Anything later went to a side output that fed a daily reconciliation job which corrected the affected windows in the warehouse. The rule I hold to is that late data never gets dropped quietly. If we discard it, it gets counted first.