What is kernel launch overhead and why does it matter?

Kernel launch overhead is the CPU cost of dispatching GPU kernels.

Each CUDA kernel launch involves driver communication, scheduling, and setup overhead. When models contain many small ops, this overhead becomes significant. Fusion and torch.compile reduce number of launches.