How does PyTorch CUDA kernel execution pipeline work end-to-end?

PyTorch schedules operations on CUDA streams which are then executed asynchronously by the GPU driver through kernel launches.

When a tensor operation is called, PyTorch converts it into a CUDA kernel call via ATen/THC backend. The kernel is enqueued into a CUDA stream. The GPU driver schedules kernels onto SMs (Streaming Multiprocessors). Execution is asynchronous, and CPU continues dispatching operations unless synchronization is forced. Memory transfers may overlap with compute if streams allow.