What is knowledge bottleneck in deep vision models?

Information bottleneck theory explains how models compress input while preserving task-relevant information.

The Information Bottleneck principle states that a good representation should maximize mutual information with output labels while minimizing information about the input. In vision models, intermediate layers act as compression stages, discarding irrelevant pixel-level details and retaining semantic features like shapes and objects.