What is the role of Python interpreter overhead in PyTorch performance?

Python interpreter overhead can dominate execution time in small tensor operations.

Each PyTorch operation involves Python dispatch before reaching C++/CUDA kernels. In small models or fine-grained ops, this overhead reduces performance. torch.compile mitigates this by fusing operations.