What is an optimizer in neural networks?
Updated May 17, 2026
Short answer
An optimizer updates model parameters to minimize loss.
Deep explanation
Optimizers use gradients to adjust weights. Variants include SGD, Adam, RMSprop, each with different update rules and convergence properties.
Real-world example
Adam is widely used in transformer models.
Common mistakes
- Using same learning rate for all optimizers.
Follow-up questions
- Why is Adam popular?
- What is momentum?