How does Mixture of Experts routing collapse happen and how is it prevented?

Routing collapse happens when only a few experts are selected repeatedly; it is prevented using load balancing losses and stochastic routing.

In MoE systems, a gating network assigns tokens to experts. Without constraints, optimization drives the router to overuse a subset of experts, causing underutilization and capacity bottlenecks. Load balancing loss encourages uniform expert usage, while techniques like noisy top-k gating, entropy regularization, and auxiliary routing losses stabilize distribution. This is critical in large-scale sparse transformers where imbalance leads to degraded representation learning.