What is Expected Calibration Error (ECE) in model evaluation?

ECE measures how well predicted probabilities align with actual outcomes.

ECE quantifies calibration by grouping predictions into bins and comparing predicted confidence vs observed accuracy. A perfectly calibrated model predicts probabilities that match real-world frequencies. For example, among all predictions with 0.8 confidence, ~80% should be correct. It is widely used in classification models, especially deep learning systems where overconfidence is common.