What is KL divergence in cost functions?

KL divergence measures how one probability distribution differs from another.

KL divergence is widely used in variational inference and probabilistic modeling. It acts as a cost function that penalizes divergence between predicted and true distributions. It is asymmetric and closely tied to entropy and information theory.