How do positional encoding methods impact transformer generalization?

Positional encoding determines how well transformers generalize to unseen sequence lengths.

Fixed sinusoidal encodings generalize better to longer sequences than learned embeddings. Rotary Positional Embedding (RoPE) encodes relative position information, improving extrapolation. ALiBi introduces linear bias for attention decay, enhancing long-context robustness.