How does ensemble diversity impact bias and variance reduction?

Ensemble diversity reduces variance effectively; higher diversity among models leads to stronger error cancellation and better generalization.

In ensemble learning, performance is not only determined by individual model accuracy but also by diversity among models. If all models make similar errors, averaging does not reduce variance effectively. Diversity can be introduced through data sampling (bagging), feature sampling (random forests), different algorithms, or stochastic training. The key principle is that uncorrelated errors cancel out, reducing variance while maintaining or slightly improving bias.

Mathematically, ensemble error depends on both average model error and correlation between models.…