What is information geometry in the context of cost functions?

Information geometry studies cost functions as geometric objects on probability manifolds.

In information geometry, model parameters define a manifold where each point represents a probability distribution. The cost function defines distances on this manifold using metrics like Fisher Information. This allows optimization to be interpreted as geodesic movement rather than simple Euclidean updates. Natural gradient descent is derived from this framework and improves convergence by respecting the curvature of probability space.