What are decision trees, and why are they used as building blocks in Random Forest?

Updated Feb 20, 2026

Short answer

A decision tree is a model that makes predictions by repeatedly splitting data using simple rules, creating a tree-like structure of decisions. Random Forest uses many decision trees as building blocks because combining their different predictions reduces overfitting and produces a more accurate, stable model.

Deep explanation

What is a decision tree?

A decision tree is a supervised machine learning algorithm that predicts an output by learning a sequence of questions from training data. It looks similar to a flowchart:

The root node is the first question the model asks.
Internal nodes are additional decision rules.
Branches represent possible outcomes.
Leaf nodes contain the final prediction.

For example, a bank might predict whether a customer will repay a loan:

TEXT

1Is income > $50,000?
2├── Yes → Is credit score > 700?
3│   ├── Yes → Approve loan
4│   └── No  → Review manually
5└── No → Reject loan

During training, the tree automatically discovers useful questions by finding feature splits that best separate the target outcomes.

How a decision tree learns

A tree does not start with rules. It creates them by repeatedly choosing the best split.

For classification problems, common split measures include:

Gini impurity: Measures how mixed the classes are in a node.
Entropy / information gain: Measures how much uncertainty a split removes.

For regression problems, trees usually minimize:

Mean squared error (MSE)
Variance within each resulting group

A simplified training process:

Python

1tree = DecisionTreeClassifier(
2    max_depth=5,
3    criterion="gini"
4)
5
6tree.fit(training_features, training_labels)
7
8prediction = tree.predict(new_data)

The goal is to create groups where examples inside each group are as similar as possible.

Why a single decision tree has limitations

Decision trees are powerful because they can capture complex patterns without requiring feature scaling. However, they have a major weakness: high variance.

A small change in training data can create a completely different tree.

For example:

One tree may split on age first.
Another tree trained on slightly different data may split on income first.
Both may make very different predictions.

A single decision tree can memorize training data instead of learning patterns that generalize.

This is called overfitting.

Why Random Forest uses decision trees

Random Forest solves this problem by creating an ensemble of many decision trees and combining their outputs.

The main idea:

Create many different training subsets using bootstrap sampling.
Train a separate decision tree on each subset.
At each split, consider only a random subset of features.
Combine all tree predictions.

For classification:

Each tree votes for a class.
The majority vote becomes the final prediction.

For regression:

Predictions from all trees are averaged.

The relationship looks like this:

Rendering diagram…

The randomness is important because it makes trees less similar. If every tree learned exactly the same patterns, averaging would provide little benefit.

⚠️ Rule of thumb: many weakly correlated trees usually outperform one highly optimized tree.

Bias vs variance trade-off

Random Forest improves accuracy mainly by reducing variance.

Model	Bias	Variance	Typical behavior
Single decision tree	Low	High	Can overfit easily
Random Forest	Slightly higher	Lower	More stable predictions
Linear model	Higher for complex patterns	Low	Simpler but less flexible

A Random Forest may not perfectly capture every detail of training data, but it usually generalizes better to unseen data.

Why trees are good building blocks

Decision trees are especially useful inside Random Forest because they:

Handle numerical and categorical features well.
Capture nonlinear relationships.
Require little data preprocessing.
Can model feature interactions naturally.

For example, a tree can easily learn:

"High income matters only when credit history is poor."

Traditional linear models often need manually created interaction features to represent this kind of relationship.

Random Forest works because each tree learns a different view of the problem, and the final prediction combines those views.

Prediction flow inside a Random Forest

For a new data point:

TEXT

1Customer information
2        |
3        v
4+----------------+
5| Decision Tree 1 | ---> Approved
6+----------------+
7
8+----------------+
9| Decision Tree 2 | ---> Rejected
10+----------------+
11
12+----------------+
13| Decision Tree 3 | ---> Approved
14+----------------+
15
16Final result: Approved (majority vote)

The forest is not trusting one tree completely. It trusts the collective decision.

Real-world example

Imagine an e-commerce company predicting whether a visitor will purchase a product.

A single decision tree might learn rules like:

Python

1if previous_purchases > 3:
2    return "Buy"
3elif visit_duration > 5:
4    return "Buy"
5else:
6    return "No Buy"

This tree may work well on historical customers but fail on new visitors.

A Random Forest trains hundreds of trees:

One tree may focus on browsing behavior.
Another may focus on purchase history.
Another may focus on device type and location.

The final prediction combines all trees:

Python

1from sklearn.ensemble import RandomForestClassifier
2
3model = RandomForestClassifier(
4    n_estimators=200,
5    random_state=42
6)
7
8model.fit(X_train, y_train)
9prediction = model.predict(X_new)

Rendering diagram…

The company gets a more reliable prediction because it uses the combined judgment of many different trees.

Common mistakes

* **Tree equals Random Forest** - A single decision tree is only one model
Random Forest is an ensemble of many trees working together.
* **More trees always fixes everything** - Increasing `n_estimators` can improve stability, but it does not fix poor features or bad training data.
* **Random means inaccurate** - The randomness creates diversity between trees, which helps reduce overfitting.
* **Trees need feature scaling** - Decision trees generally do not require normalization because they split using feature thresholds.
* **Random Forest never overfits** - It reduces overfitting compared with one tree, but extremely complex forests can still memorize noisy data.

Follow-up questions

Why does Random Forest use random subsets of features at each split?
How does Random Forest reduce overfitting compared with a single decision tree?
What is the difference between bagging and boosting?
What hyperparameters are commonly tuned in Random Forest?
When would you choose a Random Forest over a single decision tree?

Short answer

Deep explanation

What is a decision tree?

How a decision tree learns

Why a single decision tree has limitations

Why Random Forest uses decision trees

Bias vs variance trade-off

Why trees are good building blocks

Prediction flow inside a Random Forest

Real-world example

Common mistakes

Follow-up questions

More Random Forest interview questions