What is gradient boosting and how is it different from other ensemble methods?

Gradient boosting builds models sequentially where each new model corrects errors of the previous one using gradients of the loss function.

Gradient boosting is an ensemble technique that optimizes a loss function by adding weak learners (usually decision trees) in a stage-wise fashion. Each new model is trained to predict the residual errors (negative gradients) of the current ensemble. Unlike bagging, which reduces variance via averaging, boosting reduces bias by focusing on hard-to-learn samples. Algorithms like XGBoost, LightGBM, and CatBoost are optimized implementations with regularization and efficient tree-building strategies.