How does PyTorch autograd engine work internally?

PyTorch autograd builds a dynamic DAG and uses reverse-mode differentiation for gradient computation.

The autograd engine records operations as Function nodes in a dynamic computation graph. Each tensor stores a grad_fn pointing to its creator. During backward(), PyTorch performs a topological traversal in reverse order, applying chain rule and accumulating gradients in leaf tensors.