Why do normalization layers sometimes hurt model performance?

Normalization can remove useful signal or destabilize small-batch training.

BatchNorm assumes stable batch statistics, which fails for small batches or non-i.i.d data. It may also reduce model expressiveness by forcing distributions into constrained forms.