What is the role of activation functions in transformer expressivity?

Activation functions introduce non-linearity, enabling transformers to model complex patterns.

Without nonlinear activations, transformers would collapse into linear transformations. GELU and SiLU improve gradient flow and expressivity compared to ReLU. They allow smoother optimization and better performance in deep networks.