How do transformer feed-forward layers contribute to representation learning?

Feed-forward networks transform token-wise representations non-linearly to increase model capacity.

While attention mixes token information, FFN layers apply nonlinear transformations independently per token, enabling feature extraction and higher-dimensional projection. They often contain more parameters than attention layers and are critical for expressive power.