Logistic Regression: Understanding Probabilities

Logistic regression has “regression” in its name and is almost always used for classification. It answers yes-or-no questions: will this customer churn, is this transaction fraud, is this alert a real incident. What makes it more useful than a plain yes or no is that it gives you a probability, and a probability lets you decide how much risk you are willing to accept.

It is also the model that a lot of more complicated systems quietly reduce to. The last layer of most neural network classifiers is logistic regression in disguise. If you understand how it turns a weighted sum into a probability, how it learns, and how to pick a decision threshold, you understand the core of binary classification.

This article builds directly on linear regression, so read that first if weights, intercepts and gradient descent are new to you. The vocabulary of features, labels and train and test splits comes from key concepts and terminologies in ML.

Why linear regression is the wrong tool for yes or no

Suppose you run an on-call rotation and your monitoring fires a few hundred alerts a week. Most are noise. You want a model that looks at an alert and estimates whether it is a real incident worth waking someone for. Each alert comes with three numbers:

• the error rate at the time, as a percentage of requests
• the p95 latency in milliseconds
• the minutes since the last deploy

The label is 1 if the alert turned out to be a real incident and 0 if it was noise.

You could fit a linear regression to that 0 and 1 label and treat the output as a score. The trouble is that a straight line has no bounds. For an alert with a very high error rate it will happily predict 1.7, and for a quiet one it will predict -0.4. Neither is a probability, and the line gets bent out of shape by extreme points that are obviously in one class anyway.

What you want is a function that takes any number and squeezes it into the range 0 to 1, rising smoothly in between. That is the whole idea of logistic regression.

The model: a weighted sum through a sigmoid

Logistic regression starts exactly like linear regression. It computes a weighted sum of the features plus an intercept, called z:

z = w₁·x₁ + w₂·x₂ + w₃·x₃ + b

Then, instead of using z as the answer, it passes it through the sigmoid (also called the logistic function):

p = 1 / (1 + e^(−z))

The output p is the predicted probability that the label is 1. A few values show the shape:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

for z in [-4, -2, 0, 2, 4]:
    print(f"z={z:+d}  p={sigmoid(z):.3f}")

z=-4  p=0.018
z=-2  p=0.119
z=+0  p=0.500
z=+2  p=0.881
z=+4  p=0.982

A score of zero means even odds. Large positive scores approach 1 without reaching it, and large negative scores approach 0. Because the curve is steepest around zero, changes to the inputs matter most for the cases the model is unsure about.

flowchart LR
    X1["Error rate"] -->|"× w1"| S["Sum + b = z"]
    X2["p95 latency"] -->|"× w2"| S
    X3["Minutes since deploy"] -->|"× w3"| S
    S --> G["Sigmoid"]
    G --> P["P(incident)"]
    P --> T{"p ≥ threshold?"}
    T -->|"Yes"| A["Page on-call"]
    T -->|"No"| B["Log and ignore"]

The diagram makes one design point explicit: the model’s job ends at the probability. Turning that probability into an action needs a threshold, and choosing it is a separate decision that belongs to whoever owns the consequences. More on that later.

Log-odds: what the weighted sum really means

There is a second way to read the same model that makes the weights interpretable. The odds of an event are p / (1 − p): a probability of 0.8 is odds of 4 to 1. If you take the natural log of the odds and rearrange the sigmoid, you get:

log(p / (1 − p)) = w₁·x₁ + w₂·x₂ + w₃·x₃ + b

So logistic regression is a linear model, just not of the probability. It is linear in the log-odds. Each weight says how much the log-odds change when that feature rises by one unit and the others stay fixed. That is also why the decision boundary is a straight line (or a flat plane, with more features): the set of points where z = 0, and so p = 0.5, is defined by a linear equation.

How it learns: log loss instead of squared error

Linear regression picks weights that minimize squared error. For probabilities, that is the wrong yardstick. A model that says 0.01 for an alert that turned out to be a real incident is not slightly wrong, it is confidently wrong, and the loss should reflect that.

Logistic regression uses log loss, also called binary cross-entropy:

loss = −(1/n) · Σ [ yᵢ · log(pᵢ) + (1 − yᵢ) · log(1 − pᵢ) ]

Only one of the two terms is active for each example. For a real incident (y = 1) the loss is −log(p), which is near zero when p is close to 1 and grows without limit as p approaches 0. For noise (y = 0) it is −log(1 − p), the mirror image. Predicting 0.5 for everything costs about 0.69 per example; a useful model has to beat that, and on imbalanced data it also has to beat a constant guess at the base rate.

Minimizing log loss is the same as finding the weights that make the observed labels most likely, which is why you will also see it called maximum likelihood estimation. Unlike least squares, there is no closed-form answer, so the weights are always found iteratively. The good news is that log loss is convex for this model: it has a single minimum, so the optimizer will not get stuck in a bad local one.

Gradient descent, again

The gradient of log loss has the same tidy form as the gradient of mean squared error in linear regression: prediction minus label, multiplied by the features. The training loop is nearly identical; only the prediction step changes.

flowchart TD
    A["Start with zero weights"] --> B["z = X·w + b"]
    B --> C["p = sigmoid(z)"]
    C --> D["Gradient: features × (p − y)"]
    D --> E["Step weights against the gradient"]
    E --> F{"Log loss still falling?"}
    F -->|"Yes"| B
    F -->|"No"| G["Stop: weights have converged"]

Here is the synthetic alert data, saved as data.py so the later examples can import it. The true relationship is set on the log-odds scale, so you can check whether the model recovers it.

import numpy as np

rng = np.random.default_rng(7)
n = 2000

error_rate = rng.uniform(0, 5, n)          # percent of requests failing
latency_ms = rng.uniform(80, 900, n)       # p95 latency in milliseconds
since_deploy = rng.uniform(0, 240, n)      # minutes since the last deploy

# Made-up true relationship on the log-odds scale
log_odds = (
    -6.0
    + 1.1 * error_rate
    + 0.004 * latency_ms
    - 0.01 * since_deploy
)
p = 1 / (1 + np.exp(-log_odds))
incident = rng.binomial(1, p)

X = np.column_stack([error_rate, latency_ms, since_deploy])
y = incident

About 18 percent of these alerts are real incidents, which is a realistic kind of imbalance. Gradient descent from scratch:

import numpy as np
from data import X, y

# Standardize so one learning rate suits every weight
mean, std = X.mean(axis=0), X.std(axis=0)
X_s = (X - mean) / std

w = np.zeros(X_s.shape[1])
b = 0.0
lr = 0.5

for step in range(2000):
    p = 1 / (1 + np.exp(-(X_s @ w + b)))
    error = p - y
    w -= lr * X_s.T @ error / len(y)
    b -= lr * error.mean()

# Convert back to the original units
w_orig = w / std
b_orig = b - (w_orig * mean).sum()
print("intercept:", round(b_orig, 3))
print("weights:  ", np.round(w_orig, 4))

intercept: -5.896
weights:   [ 1.1198  0.0038 -0.0099]

The true values were -6.0, 1.1, 0.004 and -0.01. With 2,000 noisy yes-or-no labels, that is about as close as you can expect. Fitting scikit-learn’s LogisticRegression with C=np.inf (no penalty) on the same data gives the same numbers, which confirms the loop converged.

Training and evaluating with scikit-learn

In practice you use a library. scikit-learn’s LogisticRegression applies an L2 penalty by default and uses the lbfgs solver, which works well for most small and medium tabular problems. Because of the penalty, put a scaler in front of it in a pipeline so every feature is penalized on the same scale.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, log_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from data import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)

proba = model.predict_proba(X_test)[:, 1]
print("accuracy:", round(accuracy_score(y_test, model.predict(X_test)), 3))
print("log loss:", round(log_loss(y_test, proba), 3))
print("ROC AUC: ", round(roc_auc_score(y_test, proba), 3))

# One new alert: 3.5% errors, 650 ms p95, 10 minutes after a deploy
print("P(incident):", round(model.predict_proba([[3.5, 650, 10]])[0, 1], 3))

accuracy: 0.844
log loss: 0.334
ROC AUC:  0.862
P(incident): 0.543

Two details in that code are worth copying. stratify=y keeps the share of incidents the same in the training and test sets, which matters when one class is rare. And predict_proba returns one column per class; column 1 is the probability of the positive class. predict just applies a 0.5 cut-off to that column.

If you are on scikit-learn 1.8 or later, note that the penalty argument is deprecated. You now choose the penalty type with l1_ratio (0 for L2, 1 for L1, anything in between for elastic net) and its strength with C. Older tutorials that pass penalty="l1" will warn, and the argument is scheduled for removal in 1.10.

Accuracy is the wrong headline number

An accuracy of 84.4 percent sounds respectable until you notice that a model which always answers “noise” scores 81.6 percent on this test set, because that is the share of alerts that are noise. Accuracy rewards the model for agreeing with the majority class.

The other metrics tell you more:

ROC AUC and log loss judge the model itself. Precision and recall judge a model plus a threshold, which is what actually runs in production. The model training and evaluation article covers these metrics in more general terms.

Choosing a threshold

The default threshold of 0.5 is a convention, not a recommendation. It is the right cut-off only when a false alarm and a missed incident cost the same, and they rarely do. Waking an engineer for noise is annoying; missing an outage can be expensive.

Because the model gives probabilities, you can move the threshold without retraining:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from data import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]

for t in [0.5, 0.3, 0.2, 0.1]:
    pred = (proba >= t).astype(int)
    print(
        f"threshold={t:.1f}  flagged={pred.sum():>3}  "
        f"precision={precision_score(y_test, pred):.2f}  "
        f"recall={recall_score(y_test, pred):.2f}"
    )

threshold=0.5  flagged= 40  precision=0.68  recall=0.29
threshold=0.3  flagged=103  precision=0.57  recall=0.64
threshold=0.2  flagged=151  precision=0.46  recall=0.75
threshold=0.1  flagged=219  precision=0.37  recall=0.88

At 0.5 the model pages someone 40 times out of 500 alerts and is right about two thirds of the time, but it misses 71 percent of real incidents. That is a bad on-call tool. At 0.2 it catches three quarters of incidents, at the price of more than half the pages being noise. Neither is “correct”; the right row depends on what a page costs your team and what a missed incident costs the business.

A practical way to pick: decide the minimum recall you can live with, then take the highest threshold that achieves it on a validation set (not the test set, which you keep for the final check). scikit-learn’s TunedThresholdClassifierCV automates this search with cross-validation if you give it a scoring metric.

Class weights versus thresholds

Another common fix for imbalance is to weight the rare class more heavily during training. In scikit-learn that is class_weight="balanced", which weights each class in inverse proportion to its frequency. On this data it moves the default 0.5 cut-off from precision 0.68 and recall 0.29 to precision 0.45 and recall 0.77, close to the threshold-0.2 row above.

The ranking barely changes; what changes is where the probabilities sit. That has a cost. With balanced weights, the example alert’s predicted probability jumps from 0.54 to 0.84, which no longer matches how often such alerts really are incidents. If anything downstream uses the probability as a probability, such as an expected-cost calculation or a risk score shown to people, keep the unweighted model and move the threshold instead.

Reading the weights as odds ratios

Raw logistic regression weights are changes in log-odds, which nobody thinks in. Exponentiate a weight and you get an odds ratio: the factor by which the odds multiply when that feature rises by one unit. Scale the step to something meaningful first.

import numpy as np
from sklearn.linear_model import LogisticRegression
from data import X, y

model = LogisticRegression(C=np.inf, max_iter=1000).fit(X, y)
names = ["error_rate (+1 pt)", "latency_ms (+100 ms)", "since_deploy (+60 min)"]
steps = np.array([1, 100, 60])
for name, w, s in zip(names, model.coef_[0], steps):
    print(f"{name:<24} odds x {np.exp(w * s):.2f}")

error_rate (+1 pt)       odds x 3.06
latency_ms (+100 ms)     odds x 1.46
since_deploy (+60 min)   odds x 0.55

Each extra percentage point of errors roughly triples the odds that an alert is real. Every hour since the last deploy roughly halves them, which matches the intuition that fresh deploys cause incidents.

Be careful with the wording. An odds ratio of 3 does not mean the probability triples. Going from odds of 1 to 9 to odds of 3 to 9 moves the probability from 10 percent to 25 percent; at higher probabilities the same odds ratio moves it far less, because the sigmoid flattens out. The same cautions as in linear regression also apply: weights are per unit, and correlated features make them unstable.

Regularization and feature selection

Logistic regression overfits in the same ways linear regression does, and the same penalties help. In scikit-learn the strength is set by C, the inverse of the penalty strength, so smaller C means a stronger penalty. With an L1 penalty, a strong enough setting pushes useless weights to exactly zero.

To see it, add five columns of pure noise to the alert data and fit with an L1 penalty at three strengths:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from data import X, y, rng

# Add five columns of pure noise
X_noisy = np.column_stack([X, rng.normal(size=(len(X), 5))])

for C in [0.01, 0.1, 1.0]:
    model = make_pipeline(
        StandardScaler(),
        LogisticRegression(C=C, l1_ratio=1.0, solver="liblinear"),
    )
    model.fit(X_noisy, y)
    coefs = model[-1].coef_[0]
    auc = cross_val_score(model, X_noisy, y, cv=5, scoring="roc_auc").mean()
    print(f"C={C:<5} nonzero weights={np.count_nonzero(coefs)}  mean AUC={auc:.3f}")

C=0.01  nonzero weights=3  mean AUC=0.851
C=0.1   nonzero weights=5  mean AUC=0.875
C=1.0   nonzero weights=8  mean AUC=0.873

At C=0.01 the model keeps exactly the three real features and drops all five noise columns, at a small cost in AUC because it also shrinks the real weights hard. At C=1.0 every noise column gets a small non-zero weight. The middle setting ranks best here. In practice you search over C with cross-validation, which LogisticRegressionCV does for you. Not every solver supports every penalty: liblinear and saga handle L1, while the default lbfgs handles only L2 or no penalty.

More than two classes

The same model extends to several classes. Instead of one weighted sum, it computes one per class and passes them through the softmax function, which turns a list of scores into probabilities that add up to 1. This is multinomial logistic regression, also called softmax regression.

In scikit-learn you do not need to change anything: give LogisticRegression a label with three or more classes and every solver except liblinear fits the multinomial model. predict_proba then returns one column per class. The older multi_class argument you may see in tutorials has been removed.

Logistic regression in the cloud

As with linear regression, the main platforms let you train this model next to your data. In BigQuery ML it is a CREATE MODEL statement with model_type = 'logistic_reg'. BigQuery ML decides between binary and multiclass training from the number of distinct labels, and auto_class_weights applies inverse-frequency weights like scikit-learn’s balanced option.

CREATE OR REPLACE MODEL `my_dataset.alert_model`
OPTIONS (
  model_type = 'logistic_reg',
  input_label_cols = ['incident']
) AS
SELECT error_rate, latency_ms, since_deploy, incident
FROM `my_dataset.alert_history`;

-- Overall metrics, including log loss and ROC AUC
SELECT * FROM ML.EVALUATE(MODEL `my_dataset.alert_model`);

-- Recall, false positive rate and counts at a range of thresholds
SELECT * FROM ML.ROC_CURVE(MODEL `my_dataset.alert_model`);

ML.ROC_CURVE is the SQL equivalent of the threshold table above, and it works only for binary models. Amazon SageMaker AI’s built-in Linear Learner covers binary and multiclass classification for large datasets, and Azure Machine Learning offers logistic regression through its designer and automated ML. If you are choosing where to start, the simple ML project article covers local tooling and the free tiers.

When to use it, and when not to

Logistic regression is a strong first choice when:

• the label is a category, especially a yes or no
• you need probabilities, not just labels, to weigh costs or rank cases
• people need to understand and audit the decision, as in credit or medical settings
• training and prediction must be cheap, or run inside SQL

It struggles when the boundary between classes is strongly curved or depends on interactions you cannot engineer by hand, and with raw images, audio or text. Tree-based models, starting with decision trees in the next article in this series, handle those interactions without manual features. Even then, fit a logistic regression first: it is the baseline that tells you whether the more complex model is earning its keep.

For more depth, see the scikit-learn logistic regression guide, its section on tuning the decision threshold, and the BigQuery ML reference for generalized linear models.

Key takeaways

• Logistic regression computes a weighted sum like linear regression, then passes it through the sigmoid to produce a probability between 0 and 1.
• The model is linear in the log-odds, so its decision boundary is a straight line or flat plane, and each weight is a change in log-odds per unit of its feature.
• It learns by minimizing log loss, which punishes confident mistakes; the loss is convex, so iterative solvers find the single best fit.
• On imbalanced data, accuracy misleads. Judge the model with log loss and ROC AUC, and the deployed decision with precision and recall.
• The 0.5 threshold is a default, not a rule. Choose the threshold from the cost of each kind of error, on validation data.
• Class weights raise recall at the default threshold but distort the probabilities; move the threshold instead when the probabilities are used directly.
• Exponentiated weights are odds ratios, and an L1 penalty with a small C can drop irrelevant features entirely.