seniorGradient Descent
What is double descent phenomenon in Gradient Descent?
Updated May 16, 2026
Short answer
Double descent describes test error decreasing, increasing, then decreasing again as model complexity grows.
Deep explanation
Traditional bias-variance tradeoff predicts a single U-shaped curve. However, in modern deep learning, test error exhibits double descent: after interpolation threshold, increasing complexity improves generalization again. Gradient Descent interacts with this phenomenon via implicit bias and optimization path.
Real-world example
Large neural networks outperform medium-sized ones even after overfitting regime.
Common mistakes
- Assuming more complexity always worsens generalization.
Follow-up questions
- What is interpolation threshold?
- Why does second descent happen?