Naive Bayes: Applying Probability in ML

How naive Bayes classifies with Bayes' theorem and word counts: the independence assumption, smoothing, the main variants, calibration and streaming training.

AI & ML

Naive Bayes is the oldest trick in text classification that still earns its place. It counts how often each word appears in each class, multiplies a few probabilities together and picks the class with the largest result. Training is a single pass over the data, there is almost nothing to tune, and with a few hundred labeled examples it often beats models that are far more sophisticated.

The price is an assumption that is almost never true: that every feature is independent of every other once you know the class. This article shows where that assumption comes from, why the classifier works anyway, and where it breaks: mostly in its probabilities, rarely in its decisions.

It follows k-nearest neighbors in the series and reuses the synthetic alert data from the random forest article, so the scores line up with the earlier models. It assumes you know train and test splits and ROC AUC; model training and evaluation covers them.

Bayes’ theorem in one paragraph

Bayes’ theorem turns a probability around. You usually know how likely the evidence is given a cause, and you want how likely the cause is given the evidence:

P(class | features) = P(features | class) × P(class) / P(features)
  • P(class) is the prior: how common the class is before you look at anything. If 28 percent of alerts are real incidents, that is the prior for “incident”.
  • P(features | class) is the likelihood: how typical these features are for that class.
  • P(features) is the same for every class, so for picking a winner you can drop it and normalize at the end.

The hard part is the likelihood. A ticket with 20 words, from a vocabulary of 10,000, has more possible word combinations than you could ever observe. You can’t count how often that exact ticket appeared among billing tickets.

The naive assumption

Naive Bayes solves this by assuming that, within a class, each feature is independent of the others. The joint likelihood then splits into a product of one-feature terms, each of which is easy to estimate by counting:

P(class | x1, x2, ..., xn)  ∝  P(class) × P(x1 | class) × P(x2 | class) × ... × P(xn | class)

For text, that means a document is treated as a bag of words drawn one at a time from a class-specific distribution. Word order is ignored, and so is the fact that “connection” and “refused” tend to appear together. That is plainly false, which is where the name comes from.

flowchart LR
    C["Class: incident or noise"] --> W1["word 1"]
    C --> W2["word 2"]
    C --> W3["word 3"]
    C --> Wn["word n"]

In this picture the class generates every word, and the words have no arrows between them. Training estimates one prior per class and one probability per feature per class. Prediction multiplies them.

A worked example by hand

Take six alert messages, three incidents and three noise:

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

messages = [
    "disk full on db primary",          # incident
    "db primary connection refused",    # incident
    "api latency high error spike",     # incident
    "nightly backup completed",         # noise
    "disk usage high on build agent",   # noise
    "certificate renewed on api",       # noise
]
labels = np.array([1, 1, 1, 0, 0, 0])

vec = CountVectorizer()
X = vec.fit_transform(messages)
nb = MultinomialNB(alpha=1.0).fit(X, labels)
print("P(incident):", nb.predict_proba(vec.transform(["disk full on api"]))[0, 1].round(3))
P(incident): 0.542

You can reproduce that number with a pencil. The vocabulary has 20 distinct words. The incident messages contain 14 words in total and the noise messages 13. Each word probability is estimated as its count in the class plus one, divided by the class total plus the vocabulary size. The “plus one” is smoothing, explained in the next section.

WordCount in incidentsP(word given incident)Count in noiseP(word given noise)
disk12/3412/33
full12/3401/33
on12/3423/33
api12/3412/33

Both priors are 0.5, so they cancel. Multiply the four likelihoods for each class:

incident: (2/34)^4                      = 16 / 34^4 ≈ 1.197e-5
noise:    (2/33)(1/33)(3/33)(2/33)      = 12 / 33^4 ≈ 1.012e-5
P(incident) = 1.197 / (1.197 + 1.012)   ≈ 0.542

“full” pushes toward incident, because it never appeared in a noise message. “on” pushes the other way, because it appeared twice in noise. The model has no idea that “on” is meaningless; it only sees counts. On a corpus this small that matters, and on a real one the counts for filler words even out across classes so they stop influencing the result.

What does the naive independence assumption let naive Bayes do?

  • Estimate the likelihood of a whole document as a product of single-word probabilities
  • Learn which pairs of words tend to appear together
  • Skip the class prior when the classes are unbalanced

Correct! Assuming independence within each class turns one impossible joint probability into many simple counts.

Smoothing and log probabilities

Two practical details make naive Bayes work on real data.

Smoothing

Without smoothing, a word that never appeared in a class gets probability zero, and one zero wipes out the whole product. A single unseen word would veto a class no matter how strong the rest of the evidence. Additive smoothing adds alpha to every count: alpha=1 is Laplace smoothing, smaller values are often called Lidstone smoothing. scikit-learn’s default is alpha=1.0.

alpha is the one setting worth tuning. Here it is on a synthetic set of support tickets in three classes (billing, outage and access), where most words are shared filler and each class has a dozen characteristic words that also leak into the other classes. With 100 training tickets and with 3,000:

alpha=1e-10  n=100: 0.744  n=3000: 0.892
alpha=0.01   n=100: 0.782  n=3000: 0.892
alpha=0.1    n=100: 0.812  n=3000: 0.892
alpha=0.5    n=100: 0.829  n=3000: 0.892
alpha=1      n=100: 0.834  n=3000: 0.893
alpha=5      n=100: 0.831  n=3000: 0.893
alpha=20     n=100: 0.746  n=3000: 0.892

With little data, smoothing matters a lot: too little and the model trusts accidental zeros, too much and it flattens every distribution toward uniform. With 3,000 tickets the counts are large enough that alpha barely moves the result. Avoid alpha=0: since scikit-learn 1.4, force_alpha defaults to True, so the value is used as given, and every unseen word gets a log probability of minus infinity.

Log space

Multiplying hundreds of probabilities, each well below one, underflows to zero in floating point. Every implementation works in log space instead and adds log probabilities:

log P(class | x)  =  log P(class) + Σ log P(xi | class) + constant

That sum is a linear function of the feature counts. Multinomial naive Bayes is therefore a linear classifier, like logistic regression, with one weight per word per class. The difference is how the weights are found: naive Bayes reads them off from counts, logistic regression optimizes them to separate the classes. In scikit-learn the learned log probabilities are in feature_log_prob_ and the log priors in class_log_prior_.

The naive Bayes family

The independence assumption is shared; what changes is the distribution assumed for each feature.

VariantFeature typePer-feature modelTypical use
MultinomialNBCounts (or TF-IDF)Word frequencies per classText classification, the usual default
ComplementNBCounts (or TF-IDF)Frequencies in all other classesText with imbalanced classes
BernoulliNBBinaryWord present or absentShort texts, binary flags
GaussianNBContinuousNormal distribution per classNumeric measurements
CategoricalNBCategorical codesCategory frequencies per classColumns like region or instance type

A few notes on the differences:

  • BernoulliNB explicitly penalizes the absence of a word that is typical for a class. MultinomialNB ignores absent words entirely. For short messages where absence is informative, Bernoulli can win; for longer documents, multinomial usually does.
  • ComplementNB estimates each class’s weights from the documents not in that class. Because the complement of a small class is large, its estimates are more stable, which helps when one class dominates the data.
  • GaussianNB fits a mean and variance per feature per class. Its var_smoothing setting adds a small fraction of the largest feature variance to every variance, for numerical stability.
  • Multinomial and complement models reject negative inputs. scikit-learn raises ValueError: Negative values in data passed to MultinomialNB (input X), which you will hit if you feed them standardized features.

Naive Bayes on text

The ticket data is generated so that each class has its own words but they overlap heavily. A typical ticket looks like this:

for restart role since sso down billed urgent since since sso tax on a invite since billed on    -> billing

Half of the 6,000 tickets are kept for testing. Here are the four text models, including logistic regression for reference, trained on growing slices of the training half:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import BernoulliNB, ComplementNB, MultinomialNB
from sklearn.pipeline import make_pipeline
from tickets import texts, y   # the synthetic ticket generator

X_train, X_test, y_train, y_test = train_test_split(
    texts, y, test_size=0.5, stratify=y, random_state=0
)
models = {
    "MultinomialNB": MultinomialNB(),
    "ComplementNB": ComplementNB(),
    "BernoulliNB": BernoulliNB(),
    "LogisticRegression": LogisticRegression(max_iter=1000),
}
for n in [30, 100, 300, 3000]:
    scores = []
    for name, clf in models.items():
        model = make_pipeline(CountVectorizer(), clf).fit(X_train[:n], y_train[:n])
        scores.append(f"{name}={model.score(X_test, y_test):.3f}")
    print(n, " ".join(scores))
Training ticketsMultinomialNBComplementNBBernoulliNBLogistic regression
300.7660.7710.7180.664
1000.8340.8560.8320.789
3000.8660.8740.8570.836
3,0000.8930.8930.8780.887

This is the classic pattern. With 30 examples, naive Bayes is ten points ahead of logistic regression, and at 100 it still leads by five. Its strong assumption acts as a constraint that keeps it from fitting noise, so it reaches its best accuracy quickly. Logistic regression has to learn every weight from the data, needs more examples to do it, and catches up once it has them. On real text with richer word interactions, a discriminative model usually overtakes naive Bayes as the data grows; on this generated data, where words really are drawn independently, it only draws level. Complement NB is slightly ahead on the smaller samples, where the minority classes have the fewest examples.

Reading the model

Because the weights are counts, the model explains itself. The words with the largest gap in log probability between one class and the next most likely class:

import numpy as np

model = make_pipeline(CountVectorizer(), MultinomialNB()).fit(X_train, y_train)
vec, nb = model[0], model[1]
words = vec.get_feature_names_out()
for k, label in enumerate(nb.classes_):
    others = np.delete(nb.feature_log_prob_, k, axis=0).max(axis=0)
    lift = nb.feature_log_prob_[k] - others
    print(label, words[np.argsort(lift)[::-1][:5]].tolist())
access ['denied', 'invite', 'password', 'locked', 'permission']
billing ['credit', 'receipt', 'refund', 'tax', 'charge']
outage ['outage', 'unavailable', 'slow', 'restart', 'crash']

Predicting three new tickets with model.predict_proba gives:

refund for card charged twice       -> billing 0.942, outage 0.040, access 0.018
sso login denied after mfa reset    -> access 0.996, billing 0.003, outage 0.001
latency high please help            -> outage 0.591, billing 0.289, access 0.120

The first ticket shows a quirk worth knowing. “charged” and “twice” never appeared in training, and CountVectorizer drops words outside its vocabulary, so the prediction rests on “refund” and “card”. The third ticket has only one topic word, “latency”, so the billing prior, 50 percent of the data, still carries real weight. Normalizing word forms, or adding word pairs with ngram_range=(1, 2), is often a bigger improvement than changing the classifier.

Why does naive Bayes beat logistic regression with 100 training tickets but not with 3,000?

  • Naive Bayes stops learning after a few hundred examples
  • Logistic regression cannot handle word counts until the data is large
  • The independence assumption limits what naive Bayes can fit, so it converges quickly, while logistic regression needs more data to estimate its weights

Correct! A strong assumption costs accuracy at the limit but buys fast learning from small samples.

Gaussian naive Bayes on the alert data

For numeric features, GaussianNB models each feature as a bell curve per class. On the 5,000 alerts with eight features, using the same split as the rest of the series (save the data-generating code from the random forest article as alerts.py):

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, brier_score_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from alerts import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)
models = {
    "GaussianNB": GaussianNB(),
    "logistic regression": make_pipeline(StandardScaler(), LogisticRegression()),
}
for name, model in models.items():
    p = model.fit(X_train, y_train).predict_proba(X_test)[:, 1]
    print(f"{name:<20} accuracy={accuracy_score(y_test, model.predict(X_test)):.3f}  "
          f"ROC AUC={roc_auc_score(y_test, p):.3f}  Brier={brier_score_loss(y_test, p):.3f}")
GaussianNB           accuracy=0.818  ROC AUC=0.869  Brier=0.125
logistic regression  accuracy=0.821  ROC AUC=0.873  Brier=0.125

With no scaling and no tuning, naive Bayes lands within half a point of logistic regression. Scaling makes no difference to it, because each feature gets its own mean and variance per class. It is behind the tuned models from earlier articles: the tuned KNN reached an AUC of 0.894 and the random forest 0.902. Those models can represent the interactions in this data, such as latency mattering only shortly after a deploy, and naive Bayes, by construction, can’t.

The fitted parameters show what it learned. The mean error rate is 2.1 percent for noise and 3.4 for incidents; the mean replica count is 7.2 for noise and 4.8 for incidents. The hour of day has nearly the same mean in both classes, 11.3 and 11.8, so it contributes almost nothing. Unlike KNN, naive Bayes is not badly hurt by an irrelevant feature: a feature with the same distribution in every class adds roughly the same log probability to each class and mostly cancels out.

Where the assumption bites: probabilities

Naive Bayes usually ranks well even when its independence assumption is wrong. Its probabilities are less forgiving. In the alert data the features were generated independently, and the probabilities are honest: of the test alerts it scored between 0 and 0.05, 1.8 percent were incidents.

Now break the assumption on purpose. Add three exact copies of the error rate column, so the model sees the same evidence four times and multiplies it in four times:

import numpy as np

for copies in [0, 3]:
    X_tr = np.hstack([X_train] + [X_train[:, [0]]] * copies)
    X_te = np.hstack([X_test] + [X_test[:, [0]]] * copies)
    p = GaussianNB().fit(X_tr, y_train).predict_proba(X_te)[:, 1]
    print(f"copies={copies}: {((p < 0.05) | (p > 0.95)).mean():.0%} of predictions below 0.05 or above 0.95")
    for lo, hi in [(0, 0.05), (0.95, 1.0)]:
        sel = (p >= lo) & (p <= hi)
        print(f"  {lo:.2f}-{hi:.2f}: n={sel.sum():4d}  mean predicted={p[sel].mean():.3f}  "
              f"actual rate={y_test[sel].mean():.3f}")
copies=0: 23% of predictions below 0.05 or above 0.95
  0.00-0.05: n= 282  mean predicted=0.023  actual rate=0.018
  0.95-1.00: n=   2  mean predicted=0.953  actual rate=1.000
copies=3: 66% of predictions below 0.05 or above 0.95
  0.00-0.05: n= 551  mean predicted=0.006  actual rate=0.078
  0.95-1.00: n= 274  mean predicted=0.981  actual rate=0.650

With the duplicated feature, two thirds of the predictions become near-certain. Alerts scored at 98 percent are incidents only 65 percent of the time. Accuracy drops from 0.818 to 0.750 and the Brier score from 0.125 to 0.205. Logistic regression on the same four copies scores almost exactly as before (accuracy 0.819, AUC 0.873, Brier 0.125), because it spreads one weight across the copies instead of counting the evidence four times.

flowchart LR
    E["One piece of evidence"] --> F1["feature A"]
    E --> F2["correlated feature B"]
    F1 --> M["Naive Bayes multiplies both"]
    F2 --> M
    M --> O["Evidence counted twice: overconfident probability"]

Text is full of this: “connection” and “refused”, “out” and “of” and “memory”. That is why raw naive Bayes probabilities on text are usually pushed toward 0 and 1, even when the predicted class is right. If you only need the top class, it rarely matters. If you threshold on probabilities, rank alerts for an on-call queue or combine scores across models, calibrate first with CalibratedClassifierCV, as in the support vector machines article, and remove features you know are near-duplicates.

Training on a stream

Because training is just counting, naive Bayes can learn incrementally. Every scikit-learn variant supports partial_fit, which updates the counts with each batch. Paired with HashingVectorizer, which needs no fitted vocabulary, you can train on data that never fits in memory or keeps arriving:

import numpy as np
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.naive_bayes import MultinomialNB
from tickets import texts, y

vec = HashingVectorizer(n_features=2**10, alternate_sign=False, norm=None)
nb = MultinomialNB()
classes = np.array(["access", "billing", "outage"])

test_X, test_y = vec.transform(texts[3000:]), y[3000:]
for start in range(0, 3000, 1000):
    batch = vec.transform(texts[start:start + 1000])
    nb.partial_fit(batch, y[start:start + 1000], classes=classes)
    print(f"after {start + 1000} tickets: accuracy={nb.score(test_X, test_y):.3f}")
after 1000 tickets: accuracy=0.850
after 2000 tickets: accuracy=0.877
after 3000 tickets: accuracy=0.885

Three settings matter here. alternate_sign=False keeps every hashed value non-negative, which multinomial NB requires. norm=None keeps raw counts instead of unit-length vectors. And the first partial_fit call must list every class, because a later batch may contain a class the first one lacked.

The hash width interacts with smoothing in a way that is easy to miss. Smoothing adds alpha to every one of the n_features columns, so a very wide hash space adds a huge number of pseudo-counts. On this small vocabulary:

Hash widthalpha=1.0alpha=0.01
2^100.8850.889
2^140.6930.899
2^180.5340.869

With 2**18 columns and the default alpha, the pseudo-counts swamp the real ones. The denominators no longer reflect how many words each class contains, the biggest class wins nearly every time, and accuracy falls to the share of billing tickets. If you widen the hash space to limit collisions on a large vocabulary, lower alpha and tune it.

Naive Bayes in the cloud

The model is tiny and trains in seconds, so for most workloads the cloud question is about data volume and serving, not compute. A scikit-learn naive Bayes model fits comfortably in the simple ML project setup and deploys anywhere a Python container runs.

When the training data lives in a data lake, Apache Spark’s pyspark.ml.classification.NaiveBayes trains in a distributed job on Amazon EMR, Google Cloud Dataproc, Azure Databricks or any other Spark service. Its modelType is multinomial (the default), complement, bernoulli or gaussian; complement and gaussian arrived in Spark 3.0:

from pyspark.ml import Pipeline
from pyspark.ml.classification import NaiveBayes
from pyspark.ml.feature import CountVectorizer, StringIndexer, Tokenizer

pipeline = Pipeline(stages=[
    StringIndexer(inputCol="queue", outputCol="label"),
    Tokenizer(inputCol="text", outputCol="words"),
    CountVectorizer(inputCol="words", outputCol="features"),
    NaiveBayes(modelType="multinomial", smoothing=1.0),
])
model = pipeline.fit(tickets_df)   # a DataFrame with "text" and "queue" columns

If you want text classification without owning a model at all, managed options such as Amazon Comprehend custom classification or Azure AI Language custom text classification train from labeled examples you upload, and a prompted or tuned foundation model on Vertex AI or Bedrock can classify with few examples. They use stronger models than naive Bayes and cost more per prediction; a naive Bayes baseline tells you whether that extra accuracy is worth paying for.

For further reading, see the scikit-learn naive Bayes guide and the Spark NaiveBayes API.

Naive Bayes compared with other classifiers

Naive BayesLogistic regressionKNNRandom forest
TrainingCount onceIterative optimizationStore the dataGrow trees
Training data neededSmallModerateModerateModerate to large
Feature interactionsNoneOnly if you add themCaptured locallyCaptured
Feature scalingNot neededNeeded for regularizationEssentialNot needed
ProbabilitiesOften overconfidentUsually well calibratedVote sharesReasonable
Incremental learningpartial_fitSGDClassifier insteadAppend dataRetrain
ExplanationPer-class word probabilitiesCoefficientsThe neighborsFeature importance

When to use naive Bayes

Naive Bayes is a good choice when:

  • you are classifying text and want a strong baseline in minutes
  • you have little labeled data, from tens to a few hundred examples
  • training must be cheap, frequent or incremental, as with spam filters and ticket routing that learn from every correction
  • you need many classes or a very wide feature space, where the cost of other models grows quickly

Look elsewhere when features interact in ways that decide the outcome (a random forest or gradient boosting), when you have plenty of labeled text and need the last few points of accuracy (logistic regression, linear SVMs or fine-tuned language models), or when you need trustworthy probabilities without a calibration step.

Key takeaways

  • Naive Bayes applies Bayes’ theorem with one simplification: within a class, features are treated as independent, so the likelihood is a product of single-feature probabilities estimated by counting.
  • Pick the variant by feature type: multinomial or complement for word counts, Bernoulli for presence flags, Gaussian for continuous values, categorical for coded columns.
  • Smoothing prevents unseen words from zeroing out a class. Tune alpha when data is small, and lower it when you use a wide hashed feature space.
  • It learns quickly from few examples: on the ticket data it led logistic regression by ten points at 30 examples and drew level at 3,000.
  • Correlated features are counted repeatedly, which makes probabilities overconfident. Rankings and top-class predictions usually survive; calibrate before you rely on the probabilities.
  • Training is a single counting pass, so partial_fit supports streaming and continuous retraining, and Spark’s NaiveBayes scales the same model to data-lake volumes.

Part 7 of 7 · Machine Learning Algorithms

  1. 01Linear Regression: Predicting Outcomes
  2. 02Logistic Regression: Understanding Probabilities
  3. 03Decision Trees: Making Decisions Using Data
  4. 04Random Forest: An Ensemble Learning Method
  5. 05Support Vector Machines: Classifying Data
  6. 06K-Nearest Neighbors: Finding Similar Data Points
  7. 07Naive Bayes: Applying Probability in ML
Ready to check your understanding?Take the Artificial Unintelligence quiz
esc

Type to search all articles · ↑↓ to move · ↵ to open