What is Huber loss?

Updated May 15, 2026

Short answer

Huber loss is a regression cost function that combines the advantages of Mean Squared Error (MSE) and Mean Absolute Error (MAE). It behaves like MSE for small prediction errors and like MAE for large errors, making it less sensitive to outliers while still being smooth and easy to optimize. Huber loss is commonly used when a dataset contains some noisy or extreme values.

Deep explanation

In supervised learning, a model makes predictions and compares them with the actual values using a loss function.

For regression problems, the error for a prediction is:

[ e = y - \hat{y} ]

Where:

(y) = actual value.
(\hat{y}) = predicted value.
(e) = prediction error (residual).

A good loss function should:

Penalize incorrect predictions.
Encourage the model to improve.
Handle noise appropriately.

Two common regression losses are:

Mean Squared Error (MSE).
Mean Absolute Error (MAE).

Huber loss combines their strengths.

---

Problem with Mean Squared Error (MSE)

MSE calculates:

[ MSE = (y-\hat{y})^2 ]

Example:

TEXT

1Error = 2
2
3Squared error:
4
52² = 4
6
7
8Error = 10
9
10Squared error:
11
1210² = 100

Large errors become extremely large after squaring.

Advantages:

Smooth and differentiable.
Works well with gradient-based optimization.

Disadvantage:

Very sensitive to outliers.

Example:

TEXT

Dataset:

[10, 11, 12, 13, 1000]

The value 1000 creates a huge error and can dominate training.

---

Problem with Mean Absolute Error (MAE)

MAE calculates:

[ MAE = |y-\hat{y}| ]

Example:

TEXT

1Error = 2
2
3Absolute error:
4
5|2| = 2
6
7
8Error = 10
9
10Absolute error:
11
12|10| = 10

Advantages:

Less affected by outliers.
Treats all errors more proportionally.

Disadvantage:

The absolute value function has a sharp corner at zero, making optimization less smooth.

Graphically:

TEXT

1MSE:
2
3       /
4      /
5_____/____
6
7
8MAE:
9
10     /
11    /
12___/____

MSE has a smooth curve, while MAE has a sharp point.

---

How Huber Loss Works

Huber loss uses two behaviors depending on the size of the error.

For small errors:

[ L_\delta(e)=\frac{1}{2}e^2 ]

It behaves like MSE.

For large errors:

[ L_\delta(e)=\delta(|e|-\frac{1}{2}\delta) ]

It behaves like MAE.

The complete formula is:

[ L_\delta(e)= \begin{cases} \frac{1}{2}e^2, & |e|\leq\delta\ \delta(|e|-\frac{1}{2}\delta), & |e|>\delta \end{cases} ]

Where:

(e) = prediction error.
(\delta) = threshold that decides when to switch behavior.

---

Intuition Behind Huber Loss

The threshold (\delta) controls how the model treats errors.

Example:

TEXT

1δ = 1
2
3
4Error = 0.5
5
6Use MSE behavior
7
8Small error → square it
9
10
11Error = 10
12
13Use MAE behavior
14
15Large error → reduce penalty

The model gets:

Precise optimization for normal errors.
Protection against extreme outliers.

---

Why Huber Loss Is Useful

1. Robust to outliers

Unlike MSE:

TEXT

1Large error
2
3MSE:
4error² → very large penalty
5
6
7Huber:
8linear penalty → controlled penalty

This prevents a few unusual examples from dominating training.

---

2. Smooth optimization

Unlike MAE, Huber loss is differentiable around the transition point.

This makes it easier for optimization algorithms such as gradient descent to minimize.

---

3. Good balance between MSE and MAE

Huber loss provides:

TEXT

1Small errors:
2
3Accuracy of MSE
4
5
6Large errors:
7
8Robustness of MAE

---

Choosing the Delta Value

The parameter (\delta) determines when Huber loss changes behavior.

Small (\delta):

TEXT

More MAE-like

More resistant to outliers

Large (\delta):

TEXT

More MSE-like

More sensitive to errors

The correct value depends on the dataset.

Common practice:

Use validation data to tune (\delta).
Scale features and targets before choosing thresholds.

---

Huber Loss Compared with Other Regression Losses

Loss Function	Small Errors	Large Errors	Outlier Sensitivity
MSE	Squared penalty	Very large squared penalty	High
MAE	Linear penalty	Linear penalty	Low
Huber	Squared penalty	Linear penalty	Medium

---

Huber Loss and Gradient Descent

Machine learning models often minimize Huber loss using gradient-based optimization.

Training process:

TEXT

1Input data
2
3    ↓
4
5Model prediction
6
7    ↓
8
9Calculate Huber loss
10
11    ↓
12
13Compute gradients
14
15    ↓
16
17Update model parameters
18
19    ↓
20
21Repeat

Because Huber loss is smooth, the gradients are more stable than with MAE.

---

Real-world example

Suppose we build a model to predict house prices.

Training data:

TEXT

1House size → Actual price
2
31000 sq ft → $200,000
41500 sq ft → $300,000
52000 sq ft → $400,000

Most houses follow a normal pattern, but one record is incorrect:

TEXT

5000 sq ft → $50,000,000

Using MSE:

TEXT

1Prediction error:
2
3$10,000,000
4
5
6Squared error:
7
8Huge penalty

The model may adjust too much because of this one outlier.

Using Huber loss:

TEXT

1Small errors:
2
3treated like MSE
4
5
6Large errors:
7
8treated more like MAE

Python example:

Python

1from sklearn.linear_model import HuberRegressor
2
3model = HuberRegressor()
4
5model.fit(
6    X_train,
7    y_train
8)
9
10predictions = model.predict(X_test)

The model learns from normal examples while reducing the influence of extreme values.

Common mistakes

* Thinking Huber loss completely removes the effect of outliers.
* Assuming Huber loss is only for classification problems.
* Confusing Huber loss with MAE because both reduce outlier sensitivity.
* Forgetting that the delta parameter controls the transition between MSE and MAE behavior.
* Using MSE automatically even when the dataset contains many extreme noisy values.
* Assuming Huber loss always performs better than MSE or MAE for every regression problem.

Follow-up questions

How is Huber loss different from Mean Squared Error?
Why is Huber loss preferred over MAE in some cases?
What does the delta parameter in Huber loss control?
When should you use Huber loss instead of MSE?
Is Huber loss differentiable?
Can Huber loss be used for neural networks?

Short answer

Deep explanation

Problem with Mean Squared Error (MSE)

Problem with Mean Absolute Error (MAE)

How Huber Loss Works

Intuition Behind Huber Loss

Why Huber Loss Is Useful

1. Robust to outliers

2. Smooth optimization

3. Good balance between MSE and MAE

Choosing the Delta Value

Huber Loss Compared with Other Regression Losses

Huber Loss and Gradient Descent

Real-world example

Common mistakes

Follow-up questions

More Cost Function interview questions