Random Forest: An Ensemble Learning Method

A single decision tree is easy to read and easy to break. Change a few training rows and the root split can flip, taking the whole tree with it. A random forest accepts that instability and puts it to work: it grows hundreds of deliberately different trees and averages their answers. The individual trees are worse than one carefully pruned tree. The average is better than any of them.

Random forests are one of the most dependable models for tabular data. They need almost no preprocessing, give a sensible result with default settings, and come with a free validation score. This article explains why averaging works, the two sources of randomness that make it work, which settings matter, and where forests fall short.

It follows the article on decision trees, and everything here builds on it: splits, Gini impurity, overfitting and pruning. If train and test splits or ROC AUC are unfamiliar, start with model training and evaluation.

Why one tree is not enough

The decision tree article ended with two problems. An unconstrained tree memorizes its training data, and a tree’s structure depends heavily on exactly which rows it saw. Both are symptoms of high variance: the model changes a lot when the data changes a little.

Pruning reduces variance by making the tree smaller, but that also makes it simpler, so it can miss real structure. You trade one kind of error for another. Ensembles take a different route. They keep each tree large and flexible, so each one has low bias, and then reduce variance by averaging many of them.

The intuition is the same as averaging repeated measurements. One reading from a noisy sensor can be far off. The mean of a hundred independent readings is much closer to the truth, because the errors point in different directions and cancel. If you average B models whose errors each have variance σ² and are completely independent, the variance of the average is σ² / B.

The catch is the word independent. Trees trained on the same data make the same mistakes. If the average pairwise correlation between trees is ρ, the variance of the average becomes:

Var(average) = ρ · σ² + (1 − ρ) · σ² / B

As you add trees, the second term shrinks toward zero, but the first term stays. Adding trees can never remove the error the trees share. So the design goal of a random forest is to make trees that are individually decent and as uncorrelated as possible. It does that with two kinds of randomness.

How a random forest is built

Leo Breiman described the method in 2001, combining his earlier work on bagging with random feature selection. The recipe:

1. Draw a bootstrap sample: pick n rows from the n training rows at random, with replacement.
2. Grow a decision tree on that sample. At every split, consider only a random subset of the features, and choose the best split among those.
3. Grow the tree deep, usually without pruning.
4. Repeat for B trees, each with its own sample and its own feature draws.
5. To predict, average the trees’ predicted probabilities (classification) or values (regression).

flowchart TD
    D["Training data: n rows"] --> S1["Bootstrap sample 1"]
    D --> S2["Bootstrap sample 2"]
    D --> S3["Bootstrap sample B"]
    S1 --> T1["Tree 1: random features per split"]
    S2 --> T2["Tree 2: random features per split"]
    S3 --> T3["Tree B: random features per split"]
    T1 --> A["Average the predictions"]
    T2 --> A
    T3 --> A
    A --> P["Forest prediction"]

Each tree is trained independently of the others, so the work parallelizes across CPU cores with no coordination. That is a practical difference from gradient boosting, where each tree depends on the ones before it.

Bootstrap sampling and bagging

Sampling with replacement means some rows are drawn two or three times and others not at all. Training each model on its own bootstrap sample and averaging them is called bagging, short for bootstrap aggregating.

On average, a bootstrap sample contains about 63.2 percent of the distinct original rows. The chance a given row is never picked in n draws is (1 − 1/n)ⁿ, which approaches 1/e, or about 36.8 percent, as n grows. You can check it directly:

import numpy as np

rng = np.random.default_rng(0)
n = 3750
for i in range(3):
    sample = rng.integers(0, n, n)        # draw n row numbers with replacement
    unique = len(np.unique(sample))
    print(f"bootstrap {i + 1}: {unique} distinct rows ({unique / n:.1%}), "
          f"{n - unique} left out of bag")

bootstrap 1: 2382 distinct rows (63.5%), 1368 left out of bag
bootstrap 2: 2364 distinct rows (63.0%), 1386 left out of bag
bootstrap 3: 2370 distinct rows (63.2%), 1380 left out of bag

The rows a tree never saw are its out-of-bag rows. They come back later as a free validation set.

Random feature subsets

Bagging alone leaves the trees more alike than you would hope. If one feature is much stronger than the rest, nearly every bootstrap tree puts it at the root, and the trees end up with similar shapes and similar errors.

A random forest breaks this up by letting each split choose from only a random subset of the features, controlled by max_features. A strong feature is simply unavailable at some splits, so other features get a chance and the trees diverge. Each tree becomes a little worse on its own, and the forest becomes better because the trees disagree more usefully.

In scikit-learn the default for RandomForestClassifier is max_features="sqrt", the square root of the feature count. For RandomForestRegressor the default is 1.0, meaning all features, which makes the regressor plain bagged trees unless you change it. The trees themselves default to max_depth=None and min_samples_leaf=1, so they grow until their leaves are pure.

A random forest in scikit-learn

The decision tree article used an alert dataset with three features and a rule simple enough that a four-leaf tree recovered it exactly. Real problems are messier, so this article uses a larger synthetic set of 5,000 monitoring alerts with eight features, saved as alerts.py. The risk of a real incident rises smoothly with some features, interacts between others, and does not depend on the hour of day at all.

import numpy as np

rng = np.random.default_rng(7)
n = 5000

error_rate = rng.uniform(0, 5, n)        # percent of requests failing
latency_ms = rng.uniform(80, 900, n)     # p95 latency in milliseconds
since_deploy = rng.uniform(0, 240, n)    # minutes since the last deploy
cpu_util = rng.uniform(5, 100, n)        # percent
queue_depth = rng.poisson(20, n)         # messages waiting
hour = rng.integers(0, 24, n)            # hour of day, UTC
retries = rng.poisson(2, n)              # client retries per second
replicas = rng.integers(2, 12, n)        # healthy replicas

# Made-up, smooth risk score with curves and interactions
score = (
    -2.2
    + 1.2 * np.tanh(error_rate - 2.5)
    + 1.5 * (latency_ms > 600) * np.exp(-since_deploy / 40)
    + 0.03 * np.maximum(cpu_util - 70, 0)
    + 0.6 * np.log1p(queue_depth) * (replicas < 5)
    + 0.25 * retries
)
p = 1 / (1 + np.exp(-2 * score))
incident = rng.binomial(1, p)

X = np.column_stack(
    [error_rate, latency_ms, since_deploy, cpu_util, queue_depth, hour, retries, replicas]
)
y = incident
feature_names = [
    "error_rate", "latency_ms", "since_deploy", "cpu_util",
    "queue_depth", "hour", "retries", "replicas",
]

About 28 percent of the alerts are real incidents. Now compare a forest with the models from earlier in the series, plus gradient boosting, on the same split:

from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from alerts import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

pruned = GridSearchCV(
    DecisionTreeClassifier(random_state=0),
    {"ccp_alpha": [0.0005, 0.001, 0.002, 0.005, 0.01]},
    cv=5,
    scoring="roc_auc",
)
models = {
    "logistic regression": make_pipeline(StandardScaler(), LogisticRegression()),
    "unpruned tree": DecisionTreeClassifier(random_state=0),
    "pruned tree": pruned,
    "random forest": RandomForestClassifier(n_estimators=300, n_jobs=-1, random_state=0),
    "gradient boosting": HistGradientBoostingClassifier(random_state=0),
}
for name, model in models.items():
    model.fit(X_train, y_train)
    proba = model.predict_proba(X_test)[:, 1]
    acc = accuracy_score(y_test, model.predict(X_test))
    print(f"{name:<20} accuracy={acc:.3f}  ROC AUC={roc_auc_score(y_test, proba):.3f}")

logistic regression  accuracy=0.821  ROC AUC=0.873
unpruned tree        accuracy=0.781  ROC AUC=0.735
pruned tree          accuracy=0.844  ROC AUC=0.889
random forest        accuracy=0.854  ROC AUC=0.902
gradient boosting    accuracy=0.839  ROC AUC=0.897

The forest is the best model here, with no tuning at all. The unpruned tree is the worst, and the forest is made entirely of unpruned trees. That is the whole idea in one table: 300 trees that each overfit, averaged together, beat one tree that was carefully pruned with cross-validation.

Gradient boosting with default settings lands close to the forest. With tuning it often pulls ahead on tabular data, but it usually needs that tuning to get there, while the forest’s defaults are already near its best. On a dataset this small and noisy, the gap between the good models is small either way.

How the trees vote

predict_proba on a forest returns the average of the trees’ probabilities. With fully grown trees, each leaf is usually pure, so each tree outputs 0 or 1 and the average is effectively the share of trees voting for the class. Here is a single alert with a moderate error rate, high latency shortly after a deploy, and a short-staffed service with only four replicas:

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from alerts import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)
forest = RandomForestClassifier(n_estimators=300, n_jobs=-1, random_state=0)
forest.fit(X_train, y_train)

# error_rate, latency_ms, since_deploy, cpu_util, queue_depth, hour, retries, replicas
alert = np.array([[1.5, 780, 12, 55, 18, 14, 2, 4]])
votes = np.array([t.predict_proba(alert)[0, 1] for t in forest.estimators_])

print(f"trees voting incident: {(votes > 0.5).sum()} of {len(votes)}")
print(f"forest probability:    {forest.predict_proba(alert)[0, 1]:.2f}")
print(f"mean of tree outputs:  {votes.mean():.2f}")

trees voting incident: 170 of 300
forest probability:    0.57
mean of tree outputs:  0.57

Any one of these trees would have given a confident yes or no. The forest says 0.57, and the true probability from the generating formula is 0.63. A single tree can only tell you which side of the line an alert falls on. The forest’s disagreement between trees gives you a usable estimate of how close to the line it is.

Out-of-bag evaluation

Every tree skipped roughly a third of the training rows. For each training row, you can collect predictions from only the trees that did not see it, and score those. That is the out-of-bag (OOB) estimate, and it behaves much like cross-validation without training anything extra.

flowchart LR
    R["Training row i"] --> Q{"Was row i in this tree's sample?"}
    Q -->|"Yes"| X["Skip this tree"]
    Q -->|"No"| V["Use this tree's prediction"]
    V --> M["Average the out-of-bag predictions for row i"]
    M --> C["Compare with the true label"]
    C --> O["OOB score over all rows"]

Turn it on with oob_score=True. The loop below grows forests of increasing size and compares the OOB score with the held-out test score:

import warnings
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from alerts import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

for n_trees in [1, 5, 10, 25, 50, 100, 300, 1000]:
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")  # very small forests warn about OOB rows
        forest = RandomForestClassifier(
            n_estimators=n_trees, oob_score=True, n_jobs=-1, random_state=0
        ).fit(X_train, y_train)
    print(f"trees={n_trees:<5} OOB={forest.oob_score_:.3f}  "
          f"test={forest.score(X_test, y_test):.3f}")

trees=1     OOB=0.737  test=0.778
trees=5     OOB=0.786  test=0.826
trees=10    OOB=0.814  test=0.842
trees=25    OOB=0.841  test=0.848
trees=50    OOB=0.856  test=0.849
trees=100   OOB=0.855  test=0.854
trees=300   OOB=0.858  test=0.854
trees=1000  OOB=0.857  test=0.853

Two things stand out. Accuracy climbs quickly for the first few dozen trees and then flattens: after about 100 trees, more trees cost time and memory and buy nothing. It never gets worse, though. More trees do not cause overfitting, because each tree is trained independently and the average only becomes more stable.

The OOB score is pessimistic for small forests, because each row is judged by only the handful of trees that skipped it. Once the forest has a hundred or so trees, OOB tracks the test score closely. On a large dataset, where cross-validation means training the forest five times, OOB gives you a reasonable estimate from a single fit. By default it reports accuracy; oob_score also accepts a metric function with the signature metric(y_true, y_pred).

The settings that matter

A random forest has many parameters, but most of the effect comes from a few.

max_features and tree correlation

To see the trade-off from the variance formula in real numbers, fit forests with different max_features and measure both the quality of individual trees and how correlated their predictions are. With eight features, "sqrt" means two features per split, and None means all eight, which is plain bagging.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from alerts import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

for mf in [1, "sqrt", 4, None]:
    forest = RandomForestClassifier(
        n_estimators=300, max_features=mf, n_jobs=-1, random_state=0
    ).fit(X_train, y_train)
    # Each tree's predictions on the test set, one row per tree
    per_tree = np.array([t.predict_proba(X_test)[:, 1] for t in forest.estimators_])
    tree_auc = np.mean([roc_auc_score(y_test, p) for p in per_tree[:50]])
    corr = np.corrcoef(per_tree[:50])[np.triu_indices(50, 1)].mean()
    forest_auc = roc_auc_score(y_test, forest.predict_proba(X_test)[:, 1])
    print(f"max_features={str(mf):<5} one tree AUC={tree_auc:.3f}  "
          f"tree correlation={corr:.2f}  forest AUC={forest_auc:.3f}")

max_features=1     one tree AUC=0.689  tree correlation=0.31  forest AUC=0.904
max_features=sqrt  one tree AUC=0.726  tree correlation=0.43  forest AUC=0.902
max_features=4     one tree AUC=0.742  tree correlation=0.50  forest AUC=0.897
max_features=None  one tree AUC=0.745  tree correlation=0.52  forest AUC=0.893

Read the table from the bottom up. Giving each split more features makes each tree better, from 0.689 to 0.745 AUC, and also makes the trees more alike. The forest moves the other way: the weakest, least correlated trees produce the best forest. Here the differences are small and the default is close to the best. On datasets with many irrelevant features, a very low max_features can hurt, because most splits only get to choose among noise. Tune it with cross-validation or the OOB score rather than assuming.

min_samples_leaf and probabilities

Fully grown trees have tiny leaves. That is fine for ranking, but it makes model files large and can make probabilities noisy. Raising min_samples_leaf trades a little training fit for smoother trees:

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import brier_score_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from alerts import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)
for leaf in [1, 5, 10, 20, 50]:
    forest = RandomForestClassifier(
        n_estimators=300, min_samples_leaf=leaf, n_jobs=-1, random_state=0
    ).fit(X_train, y_train)
    proba = forest.predict_proba(X_test)[:, 1]
    print(f"min_samples_leaf={leaf:<3} train={forest.score(X_train, y_train):.3f}  "
          f"test={forest.score(X_test, y_test):.3f}  "
          f"AUC={roc_auc_score(y_test, proba):.3f}  Brier={brier_score_loss(y_test, proba):.3f}")

min_samples_leaf=1   train=1.000  test=0.854  AUC=0.902  Brier=0.108
min_samples_leaf=5   train=0.929  test=0.855  AUC=0.906  Brier=0.107
min_samples_leaf=10  train=0.902  test=0.854  AUC=0.908  Brier=0.107
min_samples_leaf=20  train=0.881  test=0.850  AUC=0.908  Brier=0.108
min_samples_leaf=50  train=0.853  test=0.838  AUC=0.908  Brier=0.114

The Brier score is the mean squared error of the predicted probabilities, so lower is better. Moderate leaf sizes improve AUC slightly and keep probabilities as good as before, with far fewer leaves to store. At 50 the trees start to underfit and accuracy drops. Notice that the training accuracy of 1.000 at the default is not a warning sign for a forest the way it is for a single tree: the forest still generalizes, because no single tree’s memorization survives the average.

If you need well-calibrated probabilities, for example to set alert thresholds by expected cost, check calibration on held-out data and wrap the forest in CalibratedClassifierCV if it is off. The logistic regression article explains why calibrated probabilities matter.

Reading a random forest

You give up the readable rules of a single tree. Three hundred deep trees are not something anyone can inspect. What you can still get is which features the model relies on, and that comes with the same trap as before.

Feature importance

feature_importances_ averages the impurity decrease from every split on each feature across all trees. It is computed on training data and favors features with many distinct values. In this dataset, hour has no effect on incidents at all, which makes it a good test:

from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
from alerts import X, y, feature_names

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)
forest = RandomForestClassifier(n_estimators=300, n_jobs=-1, random_state=0)
forest.fit(X_train, y_train)

perm = permutation_importance(
    forest, X_test, y_test, scoring="roc_auc", n_repeats=10, random_state=0, n_jobs=-1
)
rows = sorted(
    zip(feature_names, forest.feature_importances_, perm.importances_mean, perm.importances_std),
    key=lambda r: -r[2],
)
for name, imp, p, s in rows:
    print(f"{name:<13} impurity={imp:.3f}  permutation={p:+.3f} (+/- {s:.3f})")

replicas      impurity=0.241  permutation=+0.174 (+/- 0.007)
error_rate    impurity=0.285  permutation=+0.170 (+/- 0.012)
retries       impurity=0.059  permutation=+0.017 (+/- 0.003)
cpu_util      impurity=0.109  permutation=+0.014 (+/- 0.002)
since_deploy  impurity=0.097  permutation=+0.011 (+/- 0.002)
latency_ms    impurity=0.091  permutation=+0.006 (+/- 0.002)
queue_depth   impurity=0.057  permutation=+0.002 (+/- 0.002)
hour          impurity=0.061  permutation=-0.001 (+/- 0.001)

By impurity, the irrelevant hour scores 0.061, above both retries and queue_depth, which genuinely drive incidents. Deep trees split on it anyway to separate a few noisy training rows. Permutation importance on the test set, which shuffles one column at a time and measures how much AUC falls, puts hour at zero and last.

Permutation importance has a weakness of its own: when two features are strongly correlated, shuffling one barely hurts because the model can lean on the other, so both can look unimportant. If your features are correlated, group them or drop one before reading too much into the ranking.

Other limits

Forests inherit the regression tree’s inability to extrapolate. A RandomForestRegressor predicts an average of leaf values seen in training, so it can never predict above the highest target or below the lowest one in the training data. For forecasting growth, use a model that continues trends, such as linear regression, or model the change rather than the level.

They are also larger and slower to serve than one tree. A forest of 300 fully grown trees on a big dataset can take hundreds of megabytes, and prediction has to walk every tree. min_samples_leaf, max_depth and a smaller n_estimators (chosen by watching where OOB flattens) keep that in check.

Random forests compared with other tree models

A close relative, extremely randomized trees (ExtraTreesClassifier in scikit-learn), goes one step further. Instead of searching for the best threshold on each candidate feature, it draws thresholds at random and picks the best of those, and by default it trains each tree on the full dataset rather than a bootstrap sample. It is faster to train and sometimes as accurate, so it is worth trying alongside a forest.

Random forests in the cloud

Random forests train comfortably on a single machine for most datasets, so a notebook or a script with scikit-learn is usually all you need; the simple ML project article covers that setup. The managed platforms offer them in a few forms:

• BigQuery ML trains random forests in SQL with model_type = 'RANDOM_FOREST_CLASSIFIER' or 'RANDOM_FOREST_REGRESSOR', using the XGBoost library. Options such as NUM_PARALLEL_TREE, SUBSAMPLE and COLSAMPLE_BYNODE map to the number of trees, the row sample and the per-split feature sample.
• Amazon SageMaker AI does not have a built-in supervised random forest algorithm. You run scikit-learn through its prebuilt scikit-learn framework container. The built-in Random Cut Forest algorithm is a different thing: an unsupervised method for anomaly detection, not a classifier.
• Azure Machine Learning includes RandomForest and ExtremeRandomTrees among the models that automated ML tries for classification and regression.

A BigQuery ML version of the alert model looks like this, assuming a table ops.alerts with the features and an incident column:

CREATE OR REPLACE MODEL `ops.incident_forest`
OPTIONS (
  model_type = 'RANDOM_FOREST_CLASSIFIER',
  input_label_cols = ['incident'],
  num_parallel_tree = 300,
  colsample_bynode = 0.35,
  subsample = 0.8
) AS
SELECT
  error_rate, latency_ms, since_deploy, cpu_util,
  queue_depth, retries, replicas, incident
FROM `ops.alerts`;

Here colsample_bynode = 0.35 plays the role of max_features, and subsample sets the fraction of rows each tree sees. Check the current option list and defaults in the BigQuery ML reference before relying on them.

When to use a random forest

A random forest is a strong first choice when:

• the data is tabular, with a mix of numeric and categorical features and possibly interactions you have not identified
• you want a good result quickly, without a long tuning process
• you need a validation estimate but cross-validation is expensive
• you want a baseline that a more complex model must clearly beat to be worth its cost

Look elsewhere when you need readable rules (a pruned tree or a linear model), when predictions must extend beyond the training range, when the model must be very small or very fast to serve, or when you are willing to tune for the last point of accuracy, where gradient boosting often wins. For images, audio and free text, neural networks are the better fit.

For further reading, see the scikit-learn guide to forests of randomized trees, its example on out-of-bag error, and the permutation importance guide.

Key takeaways

• A random forest averages many deep decision trees, each trained on a bootstrap sample with a random subset of features considered at every split.
• Averaging reduces variance, but only the part of the error the trees do not share, so the method works by making trees as uncorrelated as possible.
• max_features controls that correlation and is the most important setting. The classifier defaults to the square root of the feature count; the scikit-learn regressor defaults to all features.
• Adding trees never causes overfitting; it only stabilizes the average. Stop when the out-of-bag score levels off.
• The out-of-bag score uses the roughly 37 percent of rows each tree never saw, giving a validation estimate from a single fit.
• Impurity-based importance can rank an irrelevant feature above useful ones; use permutation importance on held-out data.
• Forests cannot extrapolate beyond the training range and are larger to serve than one tree, but with default settings they are one of the most reliable models for tabular data.