What is activation function choice impact in deep networks?

Activation functions affect gradient flow, convergence speed, and representational capacity.

ReLU avoids vanishing gradients but can cause dead neurons. GELU smooths gradients and is preferred in transformers. Sigmoid/tanh suffer from saturation leading to vanishing gradients.