What is Mixture of Experts (MoE) in large language models?

MoE activates only a subset of model parameters per input to improve efficiency at scale.

Mixture of Experts architectures route each token through a learned gating network that selects a small number of expert sub-networks. This allows scaling model capacity to trillions of parameters while keeping compute cost relatively constant. Sparse activation improves efficiency but introduces challenges in routing stability, load balancing, and training convergence.