What is weight decay in neural networks?

Weight decay penalizes large weights to reduce overfitting.

It adds a regularization term that shrinks weights during training, encouraging simpler models and better generalization.