What is Mixture of Experts (MoE) architecture?

MoE uses multiple specialized subnetworks (experts) with a gating network to route inputs dynamically.

Mixture of Experts activates only a subset of experts per input using a learned gating function, enabling massive model capacity with sparse computation.