What is shuffle operation in distributed data processing?

Shuffle is the redistribution of data across partitions during operations like joins or aggregations.

Shuffle is one of the most expensive operations in distributed systems. It involves network I/O, disk spill, and serialization overhead. It occurs when data must be regrouped based on keys, such as groupBy or join operations.