What is feature importance in ensemble models and how is it computed?

Feature importance measures how much each feature contributes to predictions in ensemble models like Random Forest and Gradient Boosting.

Ensemble models compute feature importance differently depending on the method. In tree-based ensembles, importance is often calculated using impurity reduction (Gini or entropy decrease) or split gain. In Random Forest, importance is averaged over all trees. In boosting models like XGBoost or LightGBM, importance is computed using gain, cover, or frequency of feature usage. However, impurity-based importance can be biased toward high-cardinality features, so permutation importance is often preferred for more reliable interpretation.