What is Mixture of Experts (MoE) in vision models and how does it scale architectures?

MoE activates only a subset of model 'experts' per input, enabling massive scaling with controlled compute.

Mixture of Experts (MoE) replaces a dense feed-forward network with multiple expert sub-networks and a routing mechanism (gating network). For each input token or image patch, a router selects top-k experts (e.g., 1 or 2) to process it. This allows models to scale to billions of parameters while keeping per-inference compute relatively low. In vision transformers, MoE is often applied in FFN blocks, improving capacity without proportional computational cost increase.