Support Vector Machines: Classifying Data

Many lines can separate two groups of points. Logistic regression picks the one that makes the training labels most probable. A decision tree carves the space into boxes. A support vector machine (SVM) asks a different question: which boundary leaves the most room on both sides? It then builds that boundary from only a handful of training points, the ones closest to it, and ignores the rest.

Combined with the kernel trick, that idea made SVMs a leading general-purpose classifier for much of the 1990s and 2000s. Gradient boosting and neural networks have since taken over most large problems, but SVMs remain a good choice for small and medium datasets with many features, such as text.

This article follows random forests in the series and reuses the same synthetic alert data, so you can compare the models directly. It assumes you know train and test splits and ROC AUC; if not, start with model training and evaluation.

The widest street between two classes

If two clusters of points don’t overlap, infinitely many straight lines separate them. Some pass close to one cluster, so a slightly unusual new point lands on the wrong side. The line you want stays as far as possible from both.

An SVM formalizes this as a street. The decision boundary runs down the middle, and the street’s edges touch the nearest points of each class. The width of the street is the margin, and the SVM chooses the boundary that makes it as wide as possible.

A linear boundary is the set of points where w · x + b = 0, with a weight vector w and an offset b. The SVM scales w so that the closest points on each side satisfy w · x + b = +1 or −1. The distance between those two edges is then 2 / ‖w‖, so maximizing the margin is the same as minimizing ‖w‖, subject to every training point being on the correct side of its edge.

Support vectors

The points that sit exactly on the edges of the street are the support vectors. They alone determine the boundary. You can move or delete any other training point, as long as it doesn’t cross into the street, and the solution doesn’t change.

Here is a tiny example with ten alerts and two features, fitted with a linear SVM. A very large C (explained below) forces a hard margin, with no points allowed inside the street:

import numpy as np
from sklearn.svm import SVC

# Two small, linearly separable groups of alerts:
# x0 = error rate (percent), x1 = p95 latency (hundreds of ms)
X = np.array([
    [0.5, 1.5], [1.0, 2.0], [1.5, 1.0], [0.8, 3.0], [2.0, 2.2],   # noise
    [3.0, 5.0], [3.5, 4.0], [4.0, 6.0], [2.8, 6.5], [4.5, 5.0],   # incidents
])
y = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])

svm = SVC(kernel="linear", C=1000).fit(X, y)
w, b = svm.coef_[0], svm.intercept_[0]
print("support vectors:", svm.support_vectors_.tolist())
print(f"w = {w.round(3).tolist()}, b = {b:.3f}")
print(f"margin width = {2 / np.linalg.norm(w):.3f}")
print("decision values:", svm.decision_function(X).round(2).tolist())

support vectors: [[2.0, 2.2], [3.5, 4.0]]
w = [0.546, 0.656], b = -3.536
margin width = 2.343
decision values: [-2.28, -1.68, -2.06, -1.13, -1.0, 1.38, 1.0, 2.58, 2.26, 2.2]

Two of the ten points are support vectors, one from each class, and their decision values are exactly −1.0 and 1.0: they sit on the edges of the street. Every other point has an absolute decision value above 1, safely outside. The sign of the decision value is the predicted class, and its size tells you how far from the boundary a point is, in units of half the margin.

Soft margins and the C parameter

Real data overlaps. Some noisy alerts look exactly like incidents, and no straight line separates them. A hard margin has no solution at all in that case, and even when one exists, a single mislabeled point can drag the boundary far off course.

Corinna Cortes and Vladimir Vapnik’s 1995 soft-margin formulation fixes this by letting points violate the margin at a cost. The SVM minimizes:

½ ‖w‖²  +  C · Σ max(0, 1 − yᵢ (w · xᵢ + b))

with labels yᵢ written as +1 and −1. The first term rewards a wide street. The second is the hinge loss: zero for a point on the correct side of its edge, and growing linearly the further a point intrudes into the street or past the boundary. C sets the exchange rate between the two.

• Small C: violations are cheap, so the SVM prefers a wide street and tolerates many points inside it. More points become support vectors, and the boundary is smoother. This is stronger regularization.
• Large C: violations are expensive, so the SVM narrows the street to classify as many training points correctly as it can. Fewer support vectors, a more complicated boundary, and a higher risk of overfitting.

Compare the hinge loss with the log loss from logistic regression. Both penalize points on the wrong side. The difference is what happens to points that are already well classified: log loss keeps pushing them a little further out forever, while hinge loss stops caring entirely once a point is beyond the margin. That is why the SVM depends only on the support vectors, and why a linear SVM and logistic regression usually draw similar lines.

The kernel trick

A straight line can only do so much. If incidents occur when latency is high shortly after a deploy, or when replicas are low and the queue is deep, the true boundary is curved. You could add squared and product features and fit a linear SVM in that bigger space, but the feature count explodes.

The SVM’s optimization problem has a useful property. When you solve it in its dual form, the training data appears only through dot products between pairs of points, xᵢ · xⱼ, and the trained model predicts with:

f(x) = Σ αᵢ yᵢ K(xᵢ, x) + b        (sum over support vectors only)

where K(xᵢ, x) is a dot product and αᵢ is a weight learned for each support vector. Bernhard Boser, Isabelle Guyon and Vapnik pointed out in 1992 that you can replace that dot product with any kernel function K that equals a dot product in some other feature space, without ever computing the features themselves. The SVM then finds a maximum-margin linear boundary in that space, which maps back to a curved boundary in the original one.

The common kernels in scikit-learn:

The RBF kernel is the default in SVC and the one to try first. It is a similarity score: 1 for identical points, falling toward 0 as they move apart. The prediction for a new point becomes a weighted vote of the support vectors, where nearby support vectors count most. In that sense an RBF SVM is a smarter relative of a nearest-neighbor method that keeps only the training points that matter.

flowchart LR
    X["New point x"] --> S["Scale features"]
    S --> K["Kernel similarity to each support vector"]
    K --> W["Weight by alpha and label"]
    W --> F["Sum plus b = decision value"]
    F --> Q{"Positive?"}
    Q -->|"Yes"| P["Class 1"]
    Q -->|"No"| N["Class 0"]

Here are three kernels on the classic two-moons dataset, two interleaving half circles that no straight line can separate:

from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_moons(n_samples=1000, noise=0.3, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)

kernels = {
    "linear": SVC(kernel="linear"),
    "poly, degree 3": SVC(kernel="poly", degree=3, coef0=1),
    "rbf": SVC(kernel="rbf"),
}
for name, svc in kernels.items():
    model = make_pipeline(StandardScaler(), svc).fit(X_train, y_train)
    print(f"{name:<15} train={model.score(X_train, y_train):.3f}  "
          f"test={model.score(X_test, y_test):.3f}  "
          f"support vectors={model[-1].n_support_.sum()}")

linear          train=0.850  test=0.824  support vectors=164
poly, degree 3  train=0.908  test=0.884  support vectors=132
rbf             train=0.912  test=0.894  support vectors=142

The linear kernel can’t follow the curves and misclassifies about one test point in six. The two nonlinear kernels bend around the moons and reach close to 90 percent, which is about as good as this noise level allows.

gamma: how far each support vector reaches

The RBF kernel’s gamma controls how quickly similarity falls off with distance. With a small gamma, each support vector influences a wide area and the boundary is smooth. With a large gamma, each support vector influences only its immediate neighborhood, and the boundary can wrap tightly around individual points.

gamma and C interact, so look at them together. On the same moons data (both features already have a similar range here, so scaling changes little):

from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

X, y = make_moons(n_samples=1000, noise=0.3, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)

for C in [0.01, 1, 100]:
    for gamma in [0.1, 1, 300]:
        svm = SVC(kernel="rbf", C=C, gamma=gamma).fit(X_train, y_train)
        print(f"C={C:<5} gamma={gamma:<4} train={svm.score(X_train, y_train):.3f}  "
              f"test={svm.score(X_test, y_test):.3f}  support vectors={svm.n_support_.sum()}")

C=0.01  gamma=0.1  train=0.770  test=0.746  support vectors=492
C=0.01  gamma=1    train=0.862  test=0.820  support vectors=492
C=0.01  gamma=300  train=0.508  test=0.492  support vectors=498
C=1     gamma=0.1  train=0.856  test=0.826  support vectors=162
C=1     gamma=1    train=0.930  test=0.900  support vectors=132
C=1     gamma=300  train=0.986  test=0.814  support vectors=470
C=100   gamma=0.1  train=0.910  test=0.888  support vectors=136
C=100   gamma=1    train=0.932  test=0.904  support vectors=106
C=100   gamma=300  train=1.000  test=0.808  support vectors=462

The table shows three regimes:

• Underfitting (C=0.01): violations are so cheap that almost all 500 training points become support vectors. The model barely commits to a boundary.
• Overfitting (gamma=300): each support vector only covers a tiny circle, so the model memorizes the training points. With C=100 it gets every training point right and loses ten points of test accuracy. With C=0.01 it collapses to guessing.
• A good fit (gamma=1, C of 1 or 100): about 90 percent on the test set and the fewest support vectors.

A high support vector count is a warning: the model is either heavily regularized or memorizing, and prediction is slow, because every new point is compared with every support vector.

An SVM on the alert data

The moons are two clean features on the same scale. The alert dataset from the random forest article is closer to real life: 5,000 alerts with eight features on very different scales, from an error rate of 0 to 5 percent to latency in hundreds of milliseconds, with curved effects and interactions. Save the data-generating code from that article as alerts.py to run the examples below.

Always scale the features

An SVM measures distance, so a feature that ranges up to 900 swamps a feature that ranges up to 5. Fitting the same default SVC with and without scaling:

from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from alerts import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

raw = SVC().fit(X_train, y_train)
scaled = make_pipeline(StandardScaler(), SVC()).fit(X_train, y_train)

print(f"without scaling  test={raw.score(X_test, y_test):.3f}  "
      f"support vectors={raw.n_support_.sum()}")
print(f"with scaling     test={scaled.score(X_test, y_test):.3f}  "
      f"support vectors={scaled[-1].n_support_.sum()}")

without scaling  test=0.723  support vectors=2110
with scaling     test=0.834  support vectors=1516

Without scaling, distances are dominated by latency and minutes since deploy, and the model ends up predicting “not an incident” for every test alert. Its 72 percent accuracy is simply the share of non-incidents in the test set. Put the scaler inside a pipeline, as here, so it is fitted only on training data, including inside each cross-validation fold. This is the opposite of tree-based models, which ignore feature scale completely.

The default gamma="scale" uses 1 / (n_features × variance of X), which adapts to the data’s overall spread but not to differences between features. It is a reasonable start once the features are standardized, not a substitute for standardizing.

Tuning C and gamma together

Search C and gamma on a logarithmic grid, because their effects are multiplicative:

from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from alerts import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

pipe = Pipeline([("scale", StandardScaler()), ("svm", SVC(kernel="rbf"))])
grid = GridSearchCV(
    pipe,
    {
        "svm__C": [0.1, 1, 10, 100, 1000],
        "svm__gamma": [0.001, 0.003, 0.01, 0.03, 0.1, 0.3],
    },
    cv=5,
    scoring="roc_auc",
    n_jobs=-1,
)
grid.fit(X_train, y_train)
print("best params:", grid.best_params_)
print(f"cross-validated AUC: {grid.best_score_:.3f}")
print(f"test AUC:            {grid.score(X_test, y_test):.3f}")

best params: {'svm__C': 1000, 'svm__gamma': 0.001}
cross-validated AUC: 0.896
test AUC:            0.890

The best pair sits at a corner of the grid, which normally means you should extend the grid. Printing the full set of cross-validated AUC scores from grid.cv_results_ shows why that isn’t necessary here:

The good settings form a diagonal ridge. Raising C while lowering gamma gives about the same model: a smoother kernel compensated by a stricter penalty. Any point on the ridge is fine, so prefer the one with the smaller C, which trains faster. C=10 with gamma=0.01 is the choice used below. The bottom-right corner, with large values of both, is the overfitting regime from the moons example.

How it compares

On the same split as the earlier articles in the series:

logistic regression  accuracy=0.821  ROC AUC=0.873
linear SVM           accuracy=0.820  ROC AUC=0.873
RBF SVM, defaults    accuracy=0.834  ROC AUC=0.881
RBF SVM, tuned       accuracy=0.829  ROC AUC=0.890
random forest        accuracy=0.854  ROC AUC=0.902

(LinearSVC and the RBF models were scored with decision_function, the others with predict_proba; ROC AUC only needs a ranking, so both work.)

The linear SVM and logistic regression are effectively the same model here, as the hinge versus log loss comparison suggested. The RBF kernel picks up some of the curved structure and beats both. The random forest still wins: this data has threshold effects, such as latency above 600 ms, and an interaction with replicas below 5, which trees capture directly and a smooth kernel only approximates. That pattern is common. On tabular data with mixed features, tree ensembles are usually ahead. SVMs do relatively better when the features are numeric and comparable, and the boundary is smooth.

Probabilities from an SVM

An SVM outputs a decision value, a signed distance from the boundary, not a probability. If you need probabilities, for example to set an alert threshold by expected cost, you fit a calibration step on top. The standard approach, Platt scaling, fits a logistic curve that maps decision values to probabilities, using cross-validation so the curve isn’t fitted on the same predictions the SVM was trained on.

Older code does this with SVC(probability=True). In scikit-learn 1.9 that parameter is deprecated and scheduled for removal in 1.11; the warning recommends CalibratedClassifierCV with ensemble=False instead:

import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import brier_score_loss
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from alerts import X, y

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)
svm = make_pipeline(StandardScaler(), SVC(C=10, gamma=0.01)).fit(X_train, y_train)
calibrated = CalibratedClassifierCV(
    make_pipeline(StandardScaler(), SVC(C=10, gamma=0.01)),
    method="sigmoid", cv=5, ensemble=False,
).fit(X_train, y_train)

# error_rate, latency_ms, since_deploy, cpu_util, queue_depth, hour, retries, replicas
alert = np.array([[1.5, 780, 12, 55, 18, 14, 2, 4]])
print(f"decision value: {svm.decision_function(alert)[0]:+.2f}")
print(f"predicted:      {svm.predict(alert)[0]}")
print(f"probability:    {calibrated.predict_proba(alert)[0, 1]:.2f}")

proba = calibrated.predict_proba(X_test)[:, 1]
print(f"test Brier score: {brier_score_loss(y_test, proba):.3f}")

decision value: -0.51
predicted:      0
probability:    0.32
test Brier score: 0.117

This is the same alert the random forest article examined: moderate errors, high latency 12 minutes after a deploy, four replicas. The generating formula gives it a 0.63 chance of being an incident. The forest said 0.57; the SVM says 0.32 and calls it noise. The smooth RBF boundary averages over the sharp combination of conditions that makes this alert risky. The test Brier score of 0.117 is also a little worse than the forest’s 0.108. One alert proves little, but it matches the AUC table.

With ensemble=False, the calibrator uses cross-validation only to produce out-of-fold decision values for fitting the logistic curve, then fits one final SVM on all the training data. That keeps prediction as fast as a single SVM.

Scaling to large datasets

The kernel SVM’s weakness is training time. SVC uses the libsvm solver, and the scikit-learn documentation notes that its fit time scales at least quadratically with the number of samples, making it impractical beyond tens of thousands of rows. Prediction cost grows with the number of support vectors, which also tends to grow with the data.

There are two standard ways out:

• LinearSVC, built on liblinear, solves the linear problem directly and scales roughly linearly with the data. Note that its default loss is squared_hinge, not hinge, and it handles multiple classes with one-vs-rest rather than SVC’s one-vs-one.
• Kernel approximation: Nystroem builds an explicit feature map that approximates the RBF kernel from a sample of the data, and a linear SVM is trained on top. You trade a little accuracy for training time that grows linearly.

Here are the three on a synthetic problem with 20 features, timed on a single machine (your numbers will differ, but the trend won’t):

import time
from sklearn.datasets import make_classification
from sklearn.kernel_approximation import Nystroem
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC

X_all, y_all = make_classification(
    n_samples=80_000, n_features=20, n_informative=10, flip_y=0.05, random_state=0
)
X_test, y_test = X_all[-20_000:], y_all[-20_000:]

models = {
    "SVC (RBF)": lambda: make_pipeline(StandardScaler(), SVC()),
    "LinearSVC": lambda: make_pipeline(StandardScaler(), LinearSVC()),
    "Nystroem + LinearSVC": lambda: make_pipeline(
        StandardScaler(), Nystroem(n_components=500, random_state=0), LinearSVC()
    ),
}
for n in [5_000, 10_000, 20_000, 40_000, 60_000]:
    X, y = X_all[:n], y_all[:n]
    for name, make in models.items():
        start = time.perf_counter()
        model = make().fit(X, y)
        seconds = time.perf_counter() - start
        print(f"n={n:<6} {name:<21} {seconds:5.1f}s  test={model.score(X_test, y_test):.3f}")

Each doubling of the data roughly quadruples the kernel SVM’s training time: 6.4 seconds at 20,000 rows, 54 seconds at 60,000. Extrapolate to a million rows and it becomes hours, before any tuning. LinearSVC is nearly instant but stuck at 70 percent, because this problem isn’t linear. The Nystroem pipeline grows linearly and recovers most of the kernel’s accuracy. For very large or streaming data, SGDClassifier(loss="hinge") trains a linear SVM with stochastic gradient descent, and can be combined with the same kernel approximation.

flowchart TD
    A["Classification problem"] --> B{"More than about 50,000 rows?"}
    B -->|"No"| C{"Boundary likely linear, or many sparse features like text?"}
    C -->|"Yes"| D["LinearSVC"]
    C -->|"No"| E["SVC with RBF kernel, tune C and gamma"]
    B -->|"Yes"| F{"Linear model good enough?"}
    F -->|"Yes"| G["LinearSVC or SGDClassifier with hinge loss"]
    F -->|"No"| H["Nystroem plus linear SVM, or try a tree ensemble"]

SVMs compared with other classifiers

The same machinery also does regression. SVR fits a tube of width epsilon around the data and ignores errors inside it, applying the hinge idea to residuals; C and the kernel work as in classification. For multiple classes, SVC trains one classifier for every pair of classes and takes a vote, so k classes means k(k−1)/2 models.

Support vector machines in the cloud

Most SVM work happens in scikit-learn on a single machine, as set up in the simple ML project article, because the datasets where kernel SVMs shine are small enough. The managed platforms cover them in narrower ways:

• Amazon SageMaker AI has no built-in kernel SVM. Its built-in Linear Learner algorithm trains a linear SVM when you set predictor_type to binary_classifier and loss to hinge_loss; the default loss of auto uses logistic loss instead. For a kernel SVM, run scikit-learn in SageMaker’s prebuilt scikit-learn framework container.
• Azure Machine Learning automated ML includes a linear support vector machine (LinearSVM) among the algorithms it tries for classification.

For further reading, see the scikit-learn guide to support vector machines, its guide to kernel approximation, the probability calibration guide, and the SageMaker Linear Learner hyperparameters.

When to use an SVM

An SVM is a strong candidate when:

• the dataset has thousands to tens of thousands of rows, not millions
• the features are numeric, comparable after scaling, and possibly numerous, as in text classification with TF-IDF features (use a linear kernel there)
• the boundary is smooth rather than made of sharp thresholds
• you want a model whose complexity is controlled by two well-understood settings

Look elsewhere for millions of rows, for mixed tabular data with thresholds (a random forest or gradient boosting), for readable rules (decision trees) or for built-in probabilities (logistic regression). For images, audio and free text at scale, neural networks have replaced SVMs.

Key takeaways

• An SVM chooses the boundary with the widest margin between classes, and that boundary depends only on the support vectors, the training points on or inside the margin.
• The soft-margin SVM minimizes ½‖w‖² plus C times the hinge loss. Small C means a wider, more tolerant margin; large C fits the training data more closely.
• Kernels let the SVM draw curved boundaries by replacing dot products with a similarity function. The RBF kernel is the default, and its gamma sets how far each support vector’s influence reaches.
• Tune C and gamma together on a logarithmic grid. Good settings often lie on a diagonal ridge, so pick the one with the smaller C.
• Always standardize features in a pipeline. Without scaling, the default SVM on the alert data predicted the majority class for every alert.
• SVMs output decision values, not probabilities. Use CalibratedClassifierCV; SVC(probability=True) is deprecated as of scikit-learn 1.9.
• Kernel SVM training time grows at least quadratically with the number of rows. Beyond tens of thousands of rows, use LinearSVC, SGDClassifier, or a Nystroem kernel approximation.