What is activation memory bottleneck in transformer models?

Activations consume more memory than parameters in large transformers.

During training, intermediate activations for each layer and token must be stored for backpropagation, scaling with sequence length and depth. This becomes the dominant memory cost.