Decision Trees: Making Decisions Using Data

A decision tree is a model you can read. It is a sequence of yes-or-no questions about the input, each one chosen because it separates the data well, ending in an answer. Is the error rate above 3 percent? If not, was there a deploy in the last half hour? If so, is latency above 600 ms? That is a complete decision tree, and you could run it in your head.

That readability is why trees still matter even though a single tree is rarely the most accurate model available. They handle interactions between features without you having to engineer them, they need almost no preprocessing, and they are the building block of random forests and gradient boosting, which win a large share of real problems on tabular data. Understand one tree, and those ensembles stop being black boxes.

This article continues the series after logistic regression and uses the same on-call alert scenario, so you can compare the two directly. If train and test splits, overfitting or ROC AUC are unfamiliar, model training and evaluation covers them.

How a tree makes a prediction

A trained tree is made of nodes. Each internal node holds a test on one feature, such as error_rate <= 3.0. Each leaf holds a prediction. To classify an input you start at the root, follow the branch that matches the input’s value, and repeat until you reach a leaf.

Here is a tree learned from alert data later in this article. Each leaf shows the share of training alerts in it that were real incidents:

flowchart TD
    A{"error_rate <= 3.0?"} -->|"No"| B["Incident: 84% of 878"]
    A -->|"Yes"| C{"since_deploy <= 32.3?"}
    C -->|"No"| D["Noise: 8% of 1,175"]
    C -->|"Yes"| E{"latency_ms <= 608.1?"}
    E -->|"Yes"| F["Noise: 14% of 131"]
    E -->|"No"| G["Incident: 91% of 66"]

Three things about this structure explain most of how trees behave.

First, every test compares a single feature with a threshold. The tree carves the feature space into rectangular boxes with edges parallel to the axes, and every input that falls in the same box gets the same prediction.

Second, the predicted probability is just the class mix in the leaf. An alert that lands in the bottom-right leaf gets a probability of 0.91 because 60 of the 66 training alerts there were incidents.

Third, the tests depend on the path. Latency is only checked for alerts with a low error rate and a recent deploy. That is an interaction, and the tree found it without being told to look for one.

How a tree learns to split

Training a tree means choosing the questions. The standard method, called CART (classification and regression trees), is greedy and recursive:

1. Start with all the training data in one node.
2. Try every feature and every useful threshold, and measure how much each split would purify the two resulting groups.
3. Keep the best split, and divide the data.
4. Repeat for each child, until a stopping rule says to make it a leaf.

flowchart TD
    A["Node with training rows"] --> B{"Stopping rule met?"}
    B -->|"Yes"| L["Make a leaf: majority class or mean"]
    B -->|"No"| C["Score every feature and threshold"]
    C --> D["Pick the split with the largest impurity decrease"]
    D --> E["Left child: rows that pass"]
    D --> F["Right child: rows that fail"]
    E --> A
    F --> A

Greedy means each split is chosen for its immediate benefit, without looking ahead. Finding the best possible tree overall is computationally hopeless for any real dataset, so every practical library does it this way. The result is usually good, but a split that looks weak on its own and would enable two excellent splits below it can be missed.

Measuring impurity with Gini

To compare splits you need a number for how mixed a group is. For classification, the default in scikit-learn is Gini impurity:

Gini = 1 − Σ pₖ²

where pₖ is the share of class k in the node. A pure node (all one class) has a Gini of 0. A 50/50 mix of two classes has the maximum for two classes, 0.5. You can read Gini as the chance that two rows drawn at random from the node, with replacement, belong to different classes.

A split is scored by the weighted average Gini of its two children. The root node in the tree above held 2,250 training alerts: 1,339 noise and 911 incidents. Splitting on error_rate <= 3.0 sent 1,372 alerts left (1,199 noise, 173 incidents) and 878 right (140 noise, 738 incidents).

def gini(neg, pos):
    n = neg + pos
    return 1 - (neg / n) ** 2 - (pos / n) ** 2

root = gini(1339, 911)
left, right = gini(1199, 173), gini(140, 738)
weighted = (1372 * left + 878 * right) / 2250

print(f"root:  {root:.3f}")
print(f"left:  {left:.3f}   right: {right:.3f}")
print(f"after split: {weighted:.3f}   decrease: {root - weighted:.3f}")

root:  0.482
left:  0.220   right: 0.268
after split: 0.239   decrease: 0.243

The split halves the impurity. The algorithm ran the same calculation for every candidate threshold on every feature, and this one gave the largest decrease, so it became the root.

Entropy and the other criteria

The alternative is entropy, from information theory:

Entropy = − Σ pₖ · log₂(pₖ)

It is also 0 for a pure node, and 1 bit for a 50/50 split of two classes. The decrease in entropy from a split is called information gain. For the same root split, entropy falls from 0.974 to 0.580, a gain of 0.393 bits.

In scikit-learn you choose with criterion="gini", "entropy" or "log_loss"; the last two are the same thing under different names. Gini and entropy pick the same split most of the time, and the choice rarely changes accuracy much. Gini is slightly cheaper to compute because it avoids logarithms, which is one reason it is the default.

Fitting a tree with scikit-learn

The data is a synthetic set of 3,000 monitoring alerts, saved as data.py. It deliberately contains an interaction: an alert is likely to be real if the error rate is high, or if latency is high shortly after a deploy.

import numpy as np

rng = np.random.default_rng(11)
n = 3000

error_rate = rng.uniform(0, 5, n)          # percent of requests failing
latency_ms = rng.uniform(80, 900, n)       # p95 latency in milliseconds
since_deploy = rng.uniform(0, 240, n)      # minutes since the last deploy

# Made-up rules: a real incident is likely when errors are high,
# or when latency is high shortly after a deploy
risky = (error_rate > 3.0) | ((latency_ms > 600) & (since_deploy < 30))
p = np.where(risky, 0.85, 0.08)
incident = rng.binomial(1, p)

X = np.column_stack([error_rate, latency_ms, since_deploy])
y = incident
feature_names = ["error_rate", "latency_ms", "since_deploy"]

Now fit a small tree, print it, and compare it with logistic regression on the same split:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier, export_text
from data import X, y, feature_names

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

tree = DecisionTreeClassifier(max_leaf_nodes=4, random_state=0)
tree.fit(X_train, y_train)
print(export_text(tree, feature_names=feature_names, decimals=1, show_weights=True))

logreg = make_pipeline(StandardScaler(), LogisticRegression())
logreg.fit(X_train, y_train)

for name, model in [("decision tree", tree), ("logistic regression", logreg)]:
    proba = model.predict_proba(X_test)[:, 1]
    acc = accuracy_score(y_test, model.predict(X_test))
    auc = roc_auc_score(y_test, proba)
    print(f"{name:<20} accuracy={acc:.3f}  ROC AUC={auc:.3f}")

# Three alerts: (error rate, p95 latency, minutes since deploy)
alerts = [[1.0, 750, 10], [1.0, 750, 120], [4.0, 200, 120]]
for alert, pt, pl in zip(
    alerts, tree.predict_proba(alerts)[:, 1], logreg.predict_proba(alerts)[:, 1]
):
    print(f"{alert}  tree={pt:.2f}  logistic={pl:.2f}")

|--- error_rate <= 3.0
|   |--- since_deploy <= 32.3
|   |   |--- latency_ms <= 608.1
|   |   |   |--- weights: [113.0, 18.0] class: 0
|   |   |--- latency_ms >  608.1
|   |   |   |--- weights: [6.0, 60.0] class: 1
|   |--- since_deploy >  32.3
|   |   |--- weights: [1080.0, 95.0] class: 0
|--- error_rate >  3.0
|   |--- weights: [140.0, 738.0] class: 1

decision tree        accuracy=0.885  ROC AUC=0.883
logistic regression  accuracy=0.856  ROC AUC=0.863

[1.0, 750, 10]  tree=0.91  logistic=0.15
[1.0, 750, 120]  tree=0.08  logistic=0.10
[4.0, 200, 120]  tree=0.84  logistic=0.72

The four-leaf tree recovered the rules almost exactly: the thresholds of 3.0, 32.3 and 608.1 are close to the true 3.0, 30 and 600. The weights are the noise and incident counts in each leaf.

The three example alerts show where the difference comes from. The first has a low error rate, high latency and a deploy ten minutes ago. The tree gives it 0.91 because it learned the post-deploy rule. Logistic regression gives it 0.15, because it can only add up one effect per feature and has no way to say “latency matters, but only right after a deploy”. You could fix that by hand with an interaction feature, but you would have to know to add it.

The tree does not need a StandardScaler, either. A split on latency_ms <= 608.1 works the same whether latency is in milliseconds or seconds, so scaling a feature never changes which rows go left and which go right.

Overfitting, and how to stop it

Left alone, a tree keeps splitting until every leaf is pure or holds a single row. It will then score perfectly on the training data, because it has memorized it, noise included. Watch what happens as the depth limit is relaxed:

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from data import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

for depth in [2, 3, 4, 6, 10, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0)
    tree.fit(X_train, y_train)
    print(
        f"max_depth={str(depth):<4}  leaves={tree.get_n_leaves():>3}  "
        f"train={tree.score(X_train, y_train):.3f}  "
        f"test={tree.score(X_test, y_test):.3f}"
    )

max_depth=2     leaves=  4  train=0.861  test=0.872
max_depth=3     leaves=  8  train=0.886  test=0.884
max_depth=4     leaves= 15  train=0.889  test=0.887
max_depth=6     leaves= 36  train=0.895  test=0.879
max_depth=10    leaves= 96  train=0.910  test=0.865
max_depth=None  leaves=373  train=1.000  test=0.797

The unlimited tree grows 373 leaves, scores 100 percent on training data and drops to 79.7 percent on the test set, worse than the four-leaf tree. Since about 8 to 15 percent of the labels in this data are random by construction, any training accuracy above roughly 89 percent is the tree learning noise.

The settings that control tree size

Every one of these limits growth in a different way. In practice you tune one or two of them with cross-validation.

Cost-complexity pruning

The first five parameters stop growth early, which is called pre-pruning. Its weakness is the greedy one: a split that looks useless may be the one that makes the next split valuable, and an early stop never finds out.

Cost-complexity pruning works the other way. It grows the full tree, then removes branches whose improvement in training impurity is too small to justify their extra leaves. ccp_alpha sets the price of each leaf: at 0 nothing is pruned, and as it rises, more of the tree is cut back. Choose it with cross-validation:

from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeClassifier
from data import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

search = GridSearchCV(
    DecisionTreeClassifier(random_state=0),
    {"ccp_alpha": [0, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02]},
    cv=5,
    scoring="roc_auc",
)
search.fit(X_train, y_train)

best = search.best_estimator_
print("best ccp_alpha:", search.best_params_["ccp_alpha"])
print(f"leaves: {best.get_n_leaves()}  depth: {best.get_depth()}")
print(f"test accuracy: {best.score(X_test, y_test):.3f}")

best ccp_alpha: 0.002
leaves: 4  depth: 3
test accuracy: 0.885

Starting from the 373-leaf tree, pruning arrives at the same four-leaf structure as the hand-picked tree earlier. On real data you rarely land on the true structure so neatly, but the pattern holds: a pruned tree is smaller, generalizes better and is easier to explain. If you want the exact alphas at which branches would be removed, cost_complexity_pruning_path returns them, though a simple grid like the one above is usually enough.

Regression trees

Trees predict numbers too. A regression tree uses the same splitting procedure, but measures impurity as the squared error around the mean of each node, and each leaf predicts the mean of its training targets. The result is a step function: flat within each leaf, jumping between them.

That has one consequence you need to know before using a tree for forecasting. A tree cannot predict outside the range of values it saw in training. Here a tree and a straight line are fitted to latency measured at loads between 0 and 1,000 requests per second, then asked about higher loads:

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(3)
rps = rng.uniform(0, 1000, 500).reshape(-1, 1)            # requests per second
latency = 120 + 0.3 * rps[:, 0] + rng.normal(0, 15, 500)  # p95 latency in ms

tree = DecisionTreeRegressor(max_depth=3).fit(rps, latency)
line = LinearRegression().fit(rps, latency)

for load in [200, 800, 1000, 1500, 3000]:
    t = tree.predict([[load]])[0]
    l = line.predict([[load]])[0]
    print(f"{load:>5} rps  tree={t:6.1f} ms  linear={l:6.1f} ms")

  200 rps  tree= 180.2 ms  linear= 180.9 ms
  800 rps  tree= 358.9 ms  linear= 360.1 ms
 1000 rps  tree= 396.8 ms  linear= 419.9 ms
 1500 rps  tree= 396.8 ms  linear= 569.3 ms
 3000 rps  tree= 396.8 ms  linear=1017.4 ms

Inside the training range the two agree closely. Beyond it the tree returns the value of its last leaf forever: it predicts the same latency at 3,000 requests per second as at 1,000. For capacity planning, sales growth or anything else where the future lies outside the past, that is a serious limitation, and it applies to random forests and gradient boosting as well. Linear regression extrapolates, for better or worse.

Reading a tree, and where it misleads

Trees are often chosen because they are interpretable, and a shallow tree genuinely is: the printed rules are the model. Two habits will keep you from reading too much into one.

Feature importance

feature_importances_ reports how much each feature reduced impurity, summed over every split that used it. It is free to compute, but it is measured on training data, and it favors features with many distinct values, because they offer more thresholds to overfit with. To see this, add a column of pure random noise, standing in for something like a request ID, and fit an unpruned tree:

import numpy as np
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from data import X, y, feature_names, rng

# Add a column of random noise, a stand-in for something like a request ID
X_id = np.column_stack([X, rng.uniform(0, 1, len(X))])
names = feature_names + ["request_id"]

X_train, X_test, y_train, y_test = train_test_split(
    X_id, y, test_size=0.25, stratify=y, random_state=0
)
tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
perm = permutation_importance(
    tree, X_test, y_test, n_repeats=10, random_state=0
)

for name, imp, p in zip(names, tree.feature_importances_, perm.importances_mean):
    print(f"{name:<13} impurity={imp:.3f}  permutation={p:+.3f}")

error_rate    impurity=0.608  permutation=+0.261
latency_ms    impurity=0.146  permutation=+0.018
since_deploy  impurity=0.118  permutation=+0.028
request_id    impurity=0.128  permutation=+0.007

By impurity, the meaningless ID ranks above since_deploy. Permutation importance, which shuffles one column of the test set at a time and measures how much accuracy drops, puts it last and close to zero. When you need to explain which features matter, use permutation importance on held-out data, and prune the tree first.

Instability

A tree’s structure depends heavily on the exact training rows. Because each split is chosen greedily and everything below depends on it, a small change in the data can swap the root split and produce a tree that looks entirely different while predicting about as well. Treat the printed rules as one plausible description of the data, not the description. This instability is exactly what random forests exploit: average many trees built on different samples and the variance cancels out.

Practical details

A few behaviors of scikit-learn’s trees that matter in real projects:

• Missing values. Current versions of DecisionTreeClassifier and DecisionTreeRegressor accept NaN in dense input. During training, each split learns which side the missing values should go to.
• Categorical features. scikit-learn’s trees split on numbers only, so encode categories first. One-hot encoding works; for ordered categories, or for a tree on its own, ordinal encoding is often fine because the tree can still isolate any value with two splits.
• Imbalanced classes. class_weight="balanced" weights rare classes more heavily when scoring splits. As with logistic regression, moving the decision threshold is an alternative that keeps the probabilities honest.
• Monotonic constraints. monotonic_cst forces predictions to only rise (or only fall) with a feature, which helps when a domain rule must hold, such as risk never decreasing as the error rate climbs. It supports regression and binary classification, not multiclass.
• Probabilities. Leaf probabilities are coarse: every input in a leaf gets the same value, and small pure leaves produce overconfident 0 and 1 predictions. min_samples_leaf softens this.

Decision trees compared with the linear models

The two families fail in opposite directions, which is why it is worth fitting both as baselines before reaching for anything bigger.

Decision trees in the cloud

Managed platforms tend to offer tree ensembles rather than single trees, since a lone tree is rarely the final model. BigQuery ML trains gradient-boosted trees with model_type = 'BOOSTED_TREE_CLASSIFIER' or 'BOOSTED_TREE_REGRESSOR', and random forests with 'RANDOM_FOREST_CLASSIFIER' or 'RANDOM_FOREST_REGRESSOR', both built on the XGBoost library. Amazon SageMaker AI offers XGBoost as a built-in algorithm, and Azure Machine Learning includes tree models in automated ML.

A single tree is cheap enough that the usual place to train it is a notebook or a small script, on a laptop or a free notebook service; the simple ML project article covers that setup. Once trained, a shallow tree can be exported as rules with export_text and reimplemented almost anywhere, including as a SQL CASE expression.

When to use a decision tree

A single decision tree is a good choice when:

• people need to see and check the exact rules, as in eligibility, triage or compliance decisions
• you suspect interactions or thresholds that a linear model would miss
• you want a quick, preprocessing-free baseline on tabular data
• the model must run somewhere simple, as a handful of if statements

Reach for something else when accuracy matters more than readability, since an ensemble of trees will almost always beat one tree on the same data, and when you need to predict beyond the range of the training data. The next article in this series covers random forests, which keep most of a tree’s strengths and remove much of its instability.

For more depth, see the scikit-learn decision trees guide, its example on cost-complexity pruning, and the guide to permutation importance.

Key takeaways

• A decision tree predicts by asking a sequence of single-feature threshold questions, and each leaf predicts the class mix or mean value of its training rows.
• Trees are trained greedily with CART: at each node, pick the split that most reduces Gini impurity or entropy, then repeat on the children.
• Trees find interactions on their own and never need feature scaling, which is where they beat linear models.
• An unconstrained tree memorizes its training data. Limit it with max_depth, min_samples_leaf or max_leaf_nodes, or prune it with ccp_alpha chosen by cross-validation.
• Regression trees produce step functions and cannot extrapolate beyond the training range.
• Impurity-based feature importance is biased toward high-cardinality features; use permutation importance on held-out data instead.
• Single trees are unstable, which is the problem random forests and gradient boosting solve by combining many of them.