What does it mean for a pipeline to be idempotent, and how do you build one?

An idempotent pipeline produces the same result whether you run it once or five times for the same input window. Build it by making each run own a deterministic slice of data, then replacing that slice rather than appending to it. Use partition overwrite or a merge on a stable key, derive the window from the scheduled interval rather than the wall clock, and avoid side effects that cannot be repeated.

It means I can rerun yesterday's job right now and end up with exactly the same table as if it had run once cleanly. The pattern I use is that every run owns a partition, and the write is a replace of that partition rather than an append. So the job for 2026-08-25 deletes and rewrites that date's partition, or merges on a unique key, and running it four times changes nothing after the first. The detail that catches people is time. If your job filters on now minus one day, then a rerun three days later processes the wrong window, so the interval has to come from the orchestrator's data interval, not from the clock inside the task. I also watch for side effects, because we had a pipeline that sent a summary email at the end and a retry meant finance got the same report twice. We moved the notification into a separate task with its own dedupe key. Once everything is idempotent, backfilling is just running a range of dates.