What is idempotency in data processing systems?

Idempotency ensures that repeated execution of an operation produces the same result.

In distributed systems, retries are common due to failures. Without idempotency, retries can cause duplicate records or inconsistent states. Idempotency is achieved using unique keys, deduplication logic, or transactional writes. It is essential for exactly-once semantics in pipelines.