How do Mixture of Experts (MoE) architectures work in modern LLMs?

Mixture of Experts architectures improve scalability by activating only a subset of model parameters for each token instead of the entire network.

Traditional dense transformers activate all model parameters during inference, which becomes extremely expensive as models scale into hundreds of billions or trillions of parameters.

Mixture of Experts (MoE) architectures solve this problem by dividing the network into multiple specialized expert subnetworks. Instead of routing tokens through every expert, a gating network dynamically selects only a small subset of experts for each token.

The architecture typically includes:

Basic transformer computations common to all inputs.

2.…