How do LLMs internally approximate probability distributions over sequences?

They factorize sequence probability using autoregressive decomposition.

LLMs model joint probability P(x1,...,xn) as product of conditional probabilities P(x_t | x_<t). Transformer layers approximate these conditionals using contextual embeddings and attention. This factorization enables scalable sequence modeling.