How does concentration of measure affect neural network training stability?

It makes gradients and activations more predictable but also more sensitive to small perturbations.

In high-dimensional neural networks, concentration of measure implies that activations and gradients tend to cluster around expected values. While this stabilizes forward propagation, it also means small parameter changes can have globally consistent effects, increasing sensitivity during optimization. This dual effect explains why normalization layers (BatchNorm, LayerNorm) are critical in deep learning.