What is speculative execution in distributed data processing systems?

Speculative execution runs duplicate tasks to mitigate slow nodes in a distributed system.

In large clusters, some tasks run slower due to hardware issues or network delays (stragglers). Speculative execution launches duplicate copies of slow tasks on other nodes. The first completed task wins, and others are discarded. This improves job completion time but increases resource usage.