What is Fisher-Rao geometry and why is it important in cost functions?

Fisher-Rao geometry defines a natural distance metric on probability distributions used in optimization.

Instead of measuring distances in Euclidean parameter space, Fisher-Rao geometry defines distances based on how distributions change under parameter perturbations. The Fisher Information Matrix acts as a metric tensor, enabling natural gradient descent. This leads to parameter updates that are invariant to reparameterization and more aligned with the true geometry of probability space.