Optimizing Data Shuffles: Sort-Based vs Bypass.

Spark uses different Shuffle Managers; understanding when 'BypassMergeSort' is triggered is key to performance.

BypassMergeSort is used when there are few partitions and no map-side combine. It avoids the overhead of sorting by writing to separate files per partition. For large shuffles, Spark uses Sort-based shuffle which spills to disk.