How does cost function shape the loss landscape in deep neural networks?

The cost function defines a high-dimensional surface that determines optimization difficulty and convergence behavior.

In deep neural networks, the cost function induces a loss landscape in parameter space with millions or billions of dimensions. This landscape is highly non-convex, containing flat regions, sharp minima, saddle points, and plateaus. The geometry of this surface directly affects optimization stability. Modern research shows that many local minima are not equally bad; many are functionally equivalent due to over-parameterization. The optimizer's trajectory is heavily influenced by curvature, gradient noise, and initialization scale.