What is a distributed join and why is it expensive in large-scale systems?

A distributed join combines datasets across multiple nodes and is expensive due to data movement (shuffle).

In distributed systems like Spark, joins often require shuffling data across the network so that matching keys are colocated. This introduces network I/O, disk spill, and serialization overhead. Broadcast joins can reduce cost when one dataset is small. Partitioned joins are more efficient when both datasets share the same partitioning strategy. Poor join strategy selection is one of the biggest performance bottlenecks in data pipelines.