What is implicit bias of optimization algorithms in cost minimization?

Optimization algorithms prefer certain solutions even without explicit regularization.

Even when multiple global minima exist, optimization algorithms like SGD tend to converge toward specific solutions due to their dynamics. This is called implicit bias. It arises from gradient noise, initialization, and step-size dynamics. In deep learning, this bias often favors flat minima, which correlate with better generalization.