Classification in Machine Learning

January 19, 2026

1. Introduction

Many problems in data analysis require assigning discrete labels to observations. An email is classified as spam or not spam. An image patch is classified as background or object. A medical measurement is classified as healthy or pathological. In each case, the output is a category drawn from a finite set.

Problems of this type are instances of supervised classification: labeled examples are provided during training, and the goal is to infer a rule that assigns labels to new inputs in a way that generalizes beyond the training set.

The technical difficulty is not in defining a mapping from inputs to labels—infinitely many mappings can fit any finite dataset—but in choosing a mapping that reflects stable structure in the data-generating process. Doing so requires a modeling language (geometry or probability), an objective (a loss), and an algorithm (an optimizer).

2. Problem Formulation

We are given a dataset of input–output pairs

\{(x_i, y_i)\}_{i=1}^n,

where $x_i \in \mathbb{R}^d$ and $y_i \in \mathcal{Y}$ , a finite label set. For binary classification, $\mathcal{Y}=\{0,1\}$ .

A classifier is a function

f : \mathbb{R}^d \rightarrow \mathcal{Y}.

The objective is not merely to match labels on the training set but to achieve low error on unseen data drawn from the same distribution.

3. Classification as Space Partitioning

Any classifier partitions input space into decision regions. For binary classification, the boundary separating the regions is the decision boundary.

A linear classifier takes the form

f(x) = \mathbb{1}[w^\top x + b \ge 0],

with decision boundary $w^\top x + b = 0$ . This geometric view clarifies what can and cannot be represented by a model class: linear boundaries cannot represent disjoint regions without an explicit feature map.

Geometry alone, however, does not account for overlap, noise, and uncertainty. For that, probability is the more appropriate language.

4. A Necessary Contrast: Clustering Is Not Classification

It is common to confuse class structure with cluster structure. Clustering methods can produce visually plausible partitions, but they do not use labels and do not optimize predictive risk.

The next example demonstrates the difference. The same type of point cloud can be partitioned by an unsupervised algorithm without any notion of “correctness” relative to labels.

K-means clustering (unsupervised contrast)

python

import numpy as np
import matplotlib.pyplot as plt

# -----------------------------
# Toy dataset (no labels used for training)
# -----------------------------
rng = np.random.default_rng(0)
n_per = 80

C0 = rng.normal(loc=[-2.0, -1.0], scale=0.7, size=(n_per, 2))
C1 = rng.normal(loc=[+2.0, +1.2], scale=0.7, size=(n_per, 2))
C2 = rng.normal(loc=[-0.2, +2.4], scale=0.7, size=(n_per, 2))
X = np.vstack([C0, C1, C2])

# -----------------------------
# K-means (NumPy only)
# -----------------------------
def kmeans(X, k=3, iters=50, seed=0):
  rng = np.random.default_rng(seed)
  centers = X[rng.choice(len(X), size=k, replace=False)].copy()

for _ in range(iters):
      d2 = ((X[:, None, :] - centers[None, :, :]) ** 2).sum(axis=2)  # (n, k)
      labels = d2.argmin(axis=1)

new_centers = centers.copy()
      for j in range(k):
          mask = labels == j
          if np.any(mask):
              new_centers[j] = X[mask].mean(axis=0)
          else:
              new_centers[j] = X[rng.integers(0, len(X))]

if np.allclose(new_centers, centers):
          centers = new_centers
          break
      centers = new_centers

inertia = ((X - centers[labels]) ** 2).sum()
  return labels, centers, inertia

labels, centers, inertia = kmeans(X, k=3, iters=50, seed=0)

# -----------------------------
# Plot
# -----------------------------
fig, ax = plt.subplots(figsize=(8.5, 5.2), dpi=130)

for j in range(3):
  pts = X[labels == j]
  ax.scatter(pts[:, 0], pts[:, 1], s=28, alpha=0.85, edgecolors="none", label=f"Cluster {j}")

ax.scatter(centers[:, 0], centers[:, 1], s=180, marker="X", linewidths=1.6,
         label="Centers", color="gold", edgecolor="#333")

ax.set_title("K-means partitions points without labels", pad=10)
ax.set_xlabel("x1")
ax.set_ylabel("x2")
ax.grid(True, alpha=0.25, linewidth=1)
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
ax.legend(frameon=True, loc="lower left", bbox_to_anchor=(0.82, 0.02),
        edgecolor="black", facecolor="white")
ax.text(0.82, 0.95,
      f"$k=3$\n" + f"inertia = {inertia:.1f}",
      transform=ax.transAxes, va="top",
      bbox=dict(boxstyle="round,pad=0.3", facecolor="white",
                edgecolor="black", linewidth=0.8))

plt.tight_layout()
plt.show()

Idle.

This output is not “wrong,” but it is not a supervised classifier. The algorithm has no access to labels and therefore cannot minimize classification error. Any mapping from clusters to classes (if classes exist at all) is imposed after training.

5. Probabilistic Formulation

In supervised classification one typically models conditional probabilities

p(y \mid x),

and predicts by the decision rule

\hat{y}(x) = \arg\max_{y \in \mathcal{Y}} p(y \mid x).

This framework separates estimation (learning a probability model) from decision-making (choosing a label). It also provides a direct route to objective functions derived from likelihood principles.

6. Support Vector Machines

Logistic regression and related probabilistic models define classification through a conditional probability $p(y \mid x)$ and derive an objective from likelihood. Support Vector Machines (SVMs) take a different route: they define classification through separation with margin and derive an objective from geometric robustness.

Consider binary labels $y \in \{-1, +1\}$ . A linear decision function has the form

g(x) = w^\top x + b,

and predictions are made by

\hat{y}(x) = \mathrm{sign}(g(x)).

If the data are linearly separable, there are infinitely many separating hyperplanes. SVMs select the separator that maximizes the geometric margin. In the canonical normalization, the margin constraints are written as

y_i (w^\top x_i + b) \ge 1 \quad \text{for all } i,

and the (hard-margin) optimization problem is

\min_{w,b}\ \frac{1}{2}\|w\|^2 \quad \text{subject to } y_i (w^\top x_i + b) \ge 1.

The quantity $\|w\|^{-1}$ controls the margin width; minimizing $\|w\|^2$ maximizes the margin.

Real data are rarely separable. The soft-margin SVM introduces hinge penalties through the objective

\min_{w,b}\ \frac{1}{2}\|w\|^2 + C \sum_{i=1}^n \max(0,\ 1 - y_i(w^\top x_i + b)),

where $C>0$ controls the trade-off between large margin (small $\|w\|$ ) and training violations of the margin constraints. The term

\max(0,\ 1 - y\,g(x))

is the hinge loss. It is zero once a point is correctly classified with margin at least 1, and grows linearly for margin violations.

In contrast to logistic loss, hinge loss does not attempt to model probabilities. It enforces a margin-based notion of correctness that is particularly aligned with generalization arguments based on margins.

Linear SVM (soft margin) trained with subgradient descent

python

import numpy as np
import matplotlib.pyplot as plt

# -----------------------------
# Toy supervised dataset (2 classes, 2 features)
# -----------------------------
rng = np.random.default_rng(0)
n = 180

X0 = rng.normal(loc=[-1.6, -1.0], scale=0.8, size=(n // 2, 2))
X1 = rng.normal(loc=[+1.6, +1.0], scale=0.8, size=(n // 2, 2))
X = np.vstack([X0, X1])

# Use labels in {-1, +1} for SVM
y = np.hstack([-np.ones(n // 2), +np.ones(n // 2)])

# Standardize features (helps optimization, especially for margin methods)
mu = X.mean(axis=0, keepdims=True)
sig = X.std(axis=0, keepdims=True) + 1e-12
Xs = (X - mu) / sig

# -----------------------------
# Linear SVM primal: minimize 0.5||w||^2 + C * sum hinge
# Hinge: max(0, 1 - y*(w^T x + b))
# Subgradient descent
# -----------------------------
C = 2.0
lr0 = 0.15
steps = 2500

w = np.zeros(2)
b = 0.0

def hinge_and_grads(w, b, X, y, C):
  # margins: y * (w^T x + b)
  m = y * (X @ w + b)
  active = m < 1.0  # points violating margin
  # Objective
  hinge = np.maximum(0.0, 1.0 - m)
  obj = 0.5 * (w @ w) + C * hinge.sum()

# Subgradients
  # d/dw: w - C * sum_{active} y_i x_i
  # d/db:     - C * sum_{active} y_i
  grad_w = w.copy()
  if np.any(active):
      grad_w -= C * (X[active].T @ y[active])
      grad_b = -C * y[active].sum()
  else:
      grad_b = 0.0
  return obj, grad_w, grad_b, active.mean()

for t in range(steps):
  lr = lr0 / (1.0 + 0.002 * t)  # mild decay
  obj, gw, gb, frac_active = hinge_and_grads(w, b, Xs, y, C)
  w -= lr * gw / len(Xs)
  b -= lr * gb / len(Xs)

# -----------------------------
# Evaluate + plot in original coordinates
# Decision boundary in standardized space: w^T x + b = 0
# Convert line back to original feature space for plotting.
# -----------------------------
def predict(Xraw):
  Xz = (Xraw - mu) / sig
  return np.sign(Xz @ w + b)

yhat = predict(X)
acc = (yhat == y).mean()

# Prepare plotting grid
x1_min, x1_max = X[:, 0].min() - 0.8, X[:, 0].max() + 0.8
x2_min, x2_max = X[:, 1].min() - 0.8, X[:, 1].max() + 0.8
x1_line = np.linspace(x1_min, x1_max, 300)

# In standardized coordinates: w0*((x1-mu1)/sig1) + w1*((x2-mu2)/sig2) + b = k
# Solve for x2 given x1 and k in {0, +1, -1}
def line_x2(x1, k):
  # w0*(x1-mu1)/sig1 + w1*(x2-mu2)/sig2 + b = k
  # => (w1/sig2) * x2 = k - b - (w0/sig1)*x1 + (w0*mu1/sig1) + (w1*mu2/sig2)
  if abs(w[1]) < 1e-12:
      return None
  a = (w[0] / sig[0,0])
  c = (w[1] / sig[0,1])
  rhs = k - b - a * x1 + a * mu[0,0] + c * mu[0,1]
  return rhs / c

x2_dec = line_x2(x1_line, 0.0)
x2_m1  = line_x2(x1_line, -1.0)
x2_p1  = line_x2(x1_line, +1.0)

fig, ax = plt.subplots(figsize=(8.5, 5.2), dpi=130)

ax.scatter(X0[:, 0], X0[:, 1], s=28, alpha=0.85, edgecolors="none", label="Class -1")
ax.scatter(X1[:, 0], X1[:, 1], s=28, alpha=0.85, edgecolors="none", 
          label="Class +1", color="#380")

if x2_dec is not None:
  ax.plot(x1_line, x2_dec, linewidth=3, label="Decision boundary", color="gold")
  ax.plot(x1_line, x2_p1, linewidth=2, linestyle="--", label="Margin")
  ax.plot(x1_line, x2_m1, linewidth=2, linestyle="--")
else:
  # vertical boundary case (rare here)
  ax.axvline(mu[0,0], linewidth=3, label="Decision boundary")

ax.set_title("Linear SVM: maximum-margin classification", pad=10)
ax.set_xlabel("x1")
ax.set_ylabel("x2")
ax.grid(True, alpha=0.25, linewidth=1)
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
ax.legend(frameon=False, loc="lower left", bbox_to_anchor=(0.02, 0.02))

ax.text(0.58, 0.95,
      f"$g(x)=w^T x + b$ (trained in standardized space)\n"
      f"$C={C:.2f},\ \mathrm{{acc}}={acc:.3f}$",
      transform=ax.transAxes, va="top")

plt.tight_layout()
plt.show()

Idle.

The key distinction from logistic regression is conceptual, not cosmetic. Logistic regression optimizes a probabilistic loss that depends on log-likelihood; the SVM optimizes a margin objective in which only points on or inside the margin directly influence the solution through hinge penalties.

Support Vector Machines and logistic regression both learn linear decision functions of the form $g(x)=w^\top x+b$ , but they differ in what they optimize and what their outputs mean. Logistic regression is a conditional probabilistic model: it fits $p(y=1\mid x)=\sigma(g(x))$ by maximizing likelihood (equivalently minimizing log loss), so its scores are interpretable as calibrated probabilities under the model and training continues to be influenced by essentially all points, with increasingly large penalties for confident mistakes. A (soft-margin) SVM instead minimizes $\tfrac12|w|^2 + C\sum_i \max(0,1-y_i g(x_i))$ , which is a margin-based objective that does not define probabilities; once a point is correctly classified with margin at least 1, it exerts no further influence on the hinge term, so the solution is determined primarily by points near the boundary (the “support vectors”). Consequently, logistic regression is typically preferred when probabilistic interpretation or downstream decision-making under uncertainty matters, whereas SVMs are often preferred when margin maximization and robustness to small perturbations near the boundary are the primary goals.