juniorDeep Learning
What is gradient descent in Deep Learning?
Updated May 16, 2026
Short answer
Gradient descent is an optimization algorithm that minimizes loss by iteratively updating model parameters.
Deep explanation
Gradient descent computes gradients of the loss function with respect to model weights and updates them in the opposite direction of the gradient. Variants include batch gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent. Optimizers like Adam and RMSProp improve convergence speed and stability using momentum and adaptive learning rates.
Real-world example
Used in recommendation engines and NLP models to optimize neural network weights.
Common mistakes
- Using a learning rate that is too high, causing unstable training.
Follow-up questions
- What is a learning rate?
- Why use mini-batches?
- What happens if learning rate is too low?