Key Concepts and Terminologies in ML

Machine learning has a vocabulary problem. Read one tutorial and you meet “features”, “attributes”, “predictors” and “columns”, all meaning the same thing. Read another and “validation” means something slightly different from the first. Most of the confusion people have when starting out is not about the maths; it is about the words.

This article is a working glossary, ordered the way the terms show up in a real project: data first, then the model, then training, then evaluation. Each term comes with what it means, why it matters and, where it helps, a few lines of Python you can run. If you are new to the field, read What is Machine Learning? first; this piece picks up where that one leaves off.

The big picture

Before the individual terms, here is how they connect. Almost every supervised ML project follows the same loop:

flowchart LR
    D["Dataset"] --> S["Split: train / validation / test"]
    S --> T["Training: fit parameters"]
    T --> V["Validation: tune hyperparameters"]
    V -->|"try other settings"| T
    V --> E["Final test evaluation"]
    E --> I["Inference in production"]

Every word in that diagram has a precise meaning, and mixing them up leads to real mistakes, such as reporting a score that looks great and then collapses in production. The rest of the article walks through the loop from left to right.

Data terms

Dataset, example and instance

A dataset is the collection of records you learn from. Each record is an example (also called an instance, sample, observation or row). If you are predicting house prices, one house is one example.

Features

A feature is one measurable input about an example: square footage, number of bedrooms, postcode. Features are usually written as X, a matrix where each row is an example and each column is a feature. Other names you will see: attributes, predictors, independent variables, input variables.

Feature engineering is the work of turning raw data into useful features: extracting the day of week from a timestamp, converting text categories into numbers, or combining two columns into a ratio. On tabular data it often improves results more than switching algorithms does.

Labels and targets

A label (or target, or dependent variable) is the answer you want the model to predict, written as y. For house prices, the label is the sale price. A dataset where every example has a label is labelled data, and it is what supervised learning needs.

The type of label decides the type of problem:

The supervised vs. unsupervised learning article covers the difference between the first three rows in more depth.

Data quality terms

A few more data words come up constantly:

• Missing values: blank cells. Most algorithms cannot handle them directly, so you drop the rows or fill them in (imputation).
• Outliers: values far from the rest. Sometimes errors, sometimes the most interesting data you have.
• Class imbalance: when one class is much rarer than another, like fraud in card transactions. It breaks naive metrics, as you will see below.
• Noise: random variation in the data that no model can explain.

The role of data in ML article goes further into collecting and cleaning data.

Model terms

Model and algorithm

These two are often used interchangeably, but they are different things:

• An algorithm is the procedure for learning, such as linear regression or a decision tree. It is a recipe.
• A model is what you get after running the algorithm on your data: the recipe plus the specific numbers it learned. It is the cooked dish.

When someone says “we deployed the model”, they mean the trained artefact, usually a file containing learned values, not the algorithm.

Parameters versus hyperparameters

This is the distinction beginners most often get wrong, and it matters because the two are set in completely different ways.

Parameters are learned from the data during training. In linear regression they are the weights and the intercept. In a neural network they are the millions or billions of connection weights. You never set them by hand.

Hyperparameters are settings you choose before training that control how learning happens: the maximum depth of a decision tree, the number of trees in a forest, the learning rate of a neural network, the regularisation strength C in logistic regression. The algorithm does not learn them; you pick them, usually by trying several values and comparing results on validation data.

In scikit-learn the split is visible in the code. Hyperparameters go into the constructor; parameters appear after fit() as attributes ending in an underscore:

from sklearn.linear_model import LinearRegression
import numpy as np

X = np.array([[50], [80], [120], [150]])     # feature: size in square metres
y = np.array([150_000, 230_000, 340_000, 420_000])  # label: price

model = LinearRegression(fit_intercept=True)  # hyperparameter, chosen by you
model.fit(X, y)

print(model.coef_, model.intercept_)          # parameters, learned from data

Loss function

A loss function (or cost function) measures how wrong the model’s predictions are on the training data. Training is the process of adjusting parameters to make the loss smaller. Common choices are mean squared error for regression and log loss (cross-entropy) for classification.

Training, inference and epochs

Training (or fitting) is running the algorithm on data to learn parameters. Inference (or prediction) is using the trained model on new data. Training is expensive and happens occasionally; inference is cheap per request and happens constantly. That difference shapes how cloud ML platforms price things, separately for training jobs and for endpoints.

For models trained iteratively, such as neural networks, an epoch is one complete pass over the training data, and a batch is the slice of examples used for one parameter update. The layers and learning rates article shows how batch size and learning rate interact.

flowchart LR
    subgraph Training["Training (offline, expensive)"]
        X1["Features + labels"] --> A["Algorithm"]
        H["Hyperparameters"] --> A
        A --> M["Trained model (parameters)"]
    end
    subgraph Inference["Inference (online, per request)"]
        N["New features"] --> M2["Trained model"]
        M2 --> P["Prediction"]
    end
    M -.->|"deploy"| M2

Splitting the data

A model that scores well on the data it was trained on has proved very little. The whole point is to perform well on data it has never seen, which is called generalisation. To measure it, you hold data back.

Training, validation and test sets

• The training set is what the model learns parameters from.
• The validation set (sometimes “dev set”) is what you use to compare hyperparameter settings and models.
• The test set is touched once, at the very end, to estimate real-world performance.

The reason for three sets instead of two: every time you look at a score and change something in response, a little information about that data leaks into your choices. If you tune on the test set, the test score stops being an honest estimate. Common splits are 70/15/15 or 80/10/10, but with large datasets the validation and test sets can be a much smaller percentage.

Cross-validation

With a small dataset, a single validation split is noisy: a lucky or unlucky split can swing the score. K-fold cross-validation splits the training data into k parts, trains k times with a different part held out each time, and averages the scores.

flowchart TB
    subgraph R1["Round 1"]
        A1["Fold 1: validate"] --- B1["Folds 2-5: train"]
    end
    subgraph R2["Round 2"]
        A2["Fold 2: validate"] --- B2["Folds 1, 3-5: train"]
    end
    subgraph R5["Round 5"]
        A5["Fold 5: validate"] --- B5["Folds 1-4: train"]
    end
    R1 --> AVG["Average the 5 scores"]
    R2 --> AVG
    R5 --> AVG

In scikit-learn this is a single call:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y  # make 1 = malignant, the class we care about catching

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

scores = cross_val_score(model, X, y, cv=5, scoring="f1")
print(scores.round(3), "mean:", scores.mean().round(3))

[0.977 0.976 0.963 0.964 0.988] mean: 0.974

The spread across folds is useful information in itself: if scores ranged from 0.70 to 0.99, you would know the estimate is unstable.

Data leakage

Data leakage happens when information that would not be available at prediction time sneaks into training. The model looks excellent in testing and then fails in production. Two common forms:

• Preprocessing leakage: computing a scaler’s mean, or filling missing values, using the whole dataset before splitting. The training data then “knows” something about the test data. Putting preprocessing inside a Pipeline, as in the code above, avoids this because each step is fitted on training folds only.
• Target leakage: a feature that is only known after the outcome. Predicting whether a loan defaults using a “sent to collections” column is the classic example.

Scikit-learn’s common pitfalls guide has worked examples of both.

Fit: underfitting and overfitting

Once you can measure performance on unseen data, the two failure modes become visible.

• Underfitting: the model is too simple to capture the pattern. It scores poorly on both training and validation data.
• Overfitting: the model has memorised the training data, including its noise. It scores very well on training data and noticeably worse on validation data.

These map onto the bias-variance trade-off. Bias is error from overly simple assumptions (underfitting). Variance is error from being too sensitive to the particular training sample (overfitting). Making a model more flexible usually lowers bias and raises variance; the goal is the point where validation error is lowest.

You can see this directly by varying a decision tree’s depth:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

for depth in [1, 3, 5, None]:   # max_depth is a hyperparameter
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0)
    tree.fit(X_train, y_train)
    print(f"max_depth={str(depth):>4}  "
          f"train={tree.score(X_train, y_train):.3f}  "
          f"val={tree.score(X_val, y_val):.3f}")

max_depth=   1  train=0.930  val=0.888
max_depth=   3  train=0.977  val=0.916
max_depth=   5  train=1.000  val=0.902
max_depth=None  train=1.000  val=0.902

A depth of 1 underfits. From depth 5 onwards the tree gets every training example right but does worse on validation than depth 3: that gap is overfitting. Depth 3 is the best of these settings, and you found it using the validation set, not the test set.

Regularisation

Regularisation is any technique that discourages a model from becoming too complex. Limiting tree depth is one form. In linear models, L1 (lasso) and L2 (ridge) penalties shrink weights towards zero. In neural networks, dropout and early stopping play the same role. In scikit-learn’s LogisticRegression, a smaller C means stronger regularisation, which catches people out.

Evaluation metrics

A metric is the number you use to judge a model. Unlike the loss function, which the training algorithm optimises, metrics are chosen to reflect what matters to the business. Picking the wrong one is one of the most common ways a project goes off course.

The confusion matrix

For classification, almost every metric comes from four counts. Using “malignant tumour” as the positive class:

• True positive (TP): predicted malignant, actually malignant.
• False positive (FP): predicted malignant, actually benign. A false alarm.
• False negative (FN): predicted benign, actually malignant. A miss.
• True negative (TN): predicted benign, actually benign.

Accuracy, precision, recall and F1

• Accuracy = (TP + TN) / all predictions. The share of predictions that were right.
• Precision = TP / (TP + FP). Of the cases flagged positive, how many really were.
• Recall (or sensitivity) = TP / (TP + FN). Of the real positives, how many were caught.
• F1 score = the harmonic mean of precision and recall, useful when you need one number that balances both.

Accuracy is misleading under class imbalance. If 1 in 1,000 card transactions is fraud, a model that always says “not fraud” is 99.9% accurate and completely useless. Its recall is zero.

Here is a full train-and-evaluate run that prints all of them:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
                             f1_score, confusion_matrix)

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y  # make 1 = malignant, the class we care about catching

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

model = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000))
model.fit(X_train, y_train)      # training: learn the parameters
y_pred = model.predict(X_test)   # inference: predict on unseen data

print("accuracy :", round(accuracy_score(y_test, y_pred), 3))
print("precision:", round(precision_score(y_test, y_pred), 3))
print("recall   :", round(recall_score(y_test, y_pred), 3))
print("f1       :", round(f1_score(y_test, y_pred), 3))
print(confusion_matrix(y_test, y_pred))

accuracy : 0.965
precision: 0.975
recall   : 0.929
f1       : 0.951
[[71  1]
 [ 3 39]]

Scikit-learn’s confusion matrix puts true labels in rows and predictions in columns, ordered by label value. So the bottom row says 3 malignant tumours were predicted benign (false negatives) and 39 were caught. Precision is 39 / 40 and recall is 39 / 42. Accuracy of 96.5% sounds great, but for a screening tool those 3 misses are the number a clinician would care about, and you would likely trade some precision for higher recall by lowering the decision threshold.

Thresholds and ROC AUC

Most classifiers output a probability, and a decision threshold (0.5 by default) turns it into a class. Lowering the threshold catches more positives (higher recall) at the cost of more false alarms (lower precision). The ROC curve plots this trade-off across all thresholds, and ROC AUC summarises it as one number between 0.5 (random guessing) and 1.0 (perfect ranking).

Regression metrics

For regression, the common metrics are:

• MAE (mean absolute error): the average size of the error, in the label’s units. Easy to explain.
• RMSE (root mean squared error): like MAE but punishes large errors more heavily.
• R squared: the share of the label’s variance the model explains. 1.0 is perfect; 0 is no better than always predicting the mean.

Terms you will meet next

A few more words that come up as soon as you move past the basics:

• Baseline: the simplest reasonable model, such as always predicting the most common class or the average value. If your model cannot beat it, something is wrong.
• Embedding: a learned vector of numbers representing something like a word, product or user, positioned so similar items sit close together. Central to modern NLP and generative AI; see Key Definitions and Concepts in Generative AI.
• Gradient descent: the optimisation method most models use to reduce the loss, by repeatedly nudging parameters in the direction that lowers it. The step size is the learning rate.
• Data drift: when the data seen in production gradually differs from the training data, so a model’s accuracy decays over time. It is why models need monitoring and retraining.
• Pipeline: a chain of preprocessing and modelling steps treated as one object, so the same transformations are applied consistently in training and inference.

For a longer list, Google’s Machine Learning Glossary is a reliable reference, and scikit-learn’s guides on cross-validation and model evaluation cover the evaluation terms in depth. When you are ready to put these terms to work, Introduction to Model Training and Evaluation and Setting Up a Simple ML Project are the natural next steps in this series.

Key takeaways

• Features (X) are the inputs; the label or target (y) is what you predict. The type of label decides whether the problem is regression or classification.
• An algorithm is the learning procedure; a model is the trained result. Parameters are learned during training; hyperparameters are chosen by you before it.
• Split data into training, validation and test sets. Tune on validation, touch the test set once, and use cross-validation when data is small.
• Keep preprocessing inside a pipeline to avoid data leakage, and watch for features that would not exist at prediction time.
• A large gap between training and validation scores means overfitting; low scores on both mean underfitting. Regularisation trades a little training accuracy for better generalisation.
• Choose metrics that reflect the real cost of errors. Accuracy hides failure on imbalanced data; precision, recall, F1 and ROC AUC tell you more.