What is CKA (Centered Kernel Alignment) in model evaluation?

CKA measures similarity between neural network representations in a scale-invariant way.

CKA compares two sets of representations by computing kernel similarities after centering. Unlike cosine similarity, it is invariant to orthogonal transformations and isotropic scaling, making it ideal for comparing neural layers. It is widely used in interpretability research to compare architectures and training runs.