seniorGradient Descent
What is training dynamics bifurcation in Gradient Descent?
Updated May 16, 2026
Short answer
Bifurcation refers to sudden qualitative changes in optimization behavior due to parameter changes.
Deep explanation
In Gradient Descent systems, small changes in learning rate, initialization, or momentum can lead to drastically different trajectories. This nonlinear behavior is called bifurcation and is studied using dynamical systems theory.
Real-world example
Training instability in deep neural networks when tuning hyperparameters.
Common mistakes
- Assuming smooth behavior across hyperparameter changes.
Follow-up questions
- What causes bifurcation?
- How to detect it?