How does dynamic model selection at inference time influence bias and variance in large-scale systems?

Dynamic model selection reduces bias by choosing specialized models but can increase variance due to inconsistent routing decisions.

Dynamic model selection systems choose different models at inference time based on input features, context, or confidence scores. This allows specialization, reducing bias by using models tailored to specific sub-populations.

However, if the selection mechanism is unstable or sensitive to small input changes, it introduces variance. Two nearly identical inputs may be routed to different models, producing inconsistent outputs.

Architecturally, this is implemented using gating networks, rule-based routers, or reinforcement learning-based policies.…