What is Gradient Clipping and why is it important?
Updated May 16, 2026
Short answer
Gradient clipping limits gradient magnitude to stabilize training and prevent exploding gradients.
Deep explanation
During backpropagation, gradients can sometimes grow exponentially, especially in deep networks and recurrent architectures. Extremely large gradients cause unstable parameter updates, numerical overflow, and divergence.
Gradient clipping prevents this by constraining gradient magnitude.
Common approaches:
- Clip by Value:
- Restrict each gradient component to a range.
- Clip by Norm:
- Scale gradients if overall norm exceeds threshold.
Formula:
if ||g|| > threshold: g = g * threshold / ||g||
Benefits:
- Stabilizes optimization.
- Prevents weight explosion.
- Enables training deeper networks.
- Improves recurrent model stability.
Gradient clipping is especially important in:
- LSTMs.
- GRUs.
- Transformers.
- Reinforcement learning systems.
Real-world example
Large language models use gradient clipping extensively during distributed training to maintain numerical stability.
Common mistakes
- Using clipping thresholds that are too small, preventing effective learning.
Follow-up questions
- What causes exploding gradients?
- Does clipping solve vanishing gradients?
- Why is clipping important for RNNs?