seniorGradient Descent
What is implicit bias of Gradient Descent?
Updated May 16, 2026
Short answer
Implicit bias refers to the tendency of Gradient Descent to prefer certain solutions even without explicit regularization.
Deep explanation
Even when multiple solutions minimize loss equally, Gradient Descent tends to converge to specific types (e.g., minimum norm solutions). This bias depends on initialization, architecture, and optimization dynamics.
Real-world example
Neural networks favoring flatter minima that generalize better.
Common mistakes
- Assuming all minima are equally likely.
Follow-up questions
- Why does implicit bias matter?
- Does SGD have bias?