Regression in Machine Learning

January 19, 2026

Linear and Polynomial Regression

1. Problem setting

We consider supervised regression with scalar input $x \in \mathbb{R}$ and scalar output $y \in \mathbb{R}$ . Given a dataset $\{(x_i, y_i)\}_{i=1}^n$ , we fit a function $\hat f(x)$ from a specified model class and evaluate its agreement with the observed targets.

A central point: a model can be nonlinear in the input $x$ while remaining linear in its parameters. Polynomial regression is the canonical example.

2. Data generation model

We generate inputs on an interval and synthesize targets from a function that is mostly linear but contains systematic curvature:

y = 2.4x + 5 + 2\sin(0.9x) + 0.12(x - 5)^2 + \varepsilon, \quad where

\varepsilon \sim \mathcal{N}(0, \sigma^2).

Interpretation:

$2.4x + 5$ provides a dominant linear trend.
$2\sin(0.9x)$ adds oscillatory structure that cannot be captured by a polynomial of very low degree without approximation error.
$0.12(x-5)^2$ adds global curvature.
$\varepsilon$ injects irreducible noise; no deterministic regressor can interpolate the data without overfitting.

All models below are trained on the same dataset (same random seed and generation procedure) so differences in fit are attributable to the model class, not to resampling.

3. Linear regression (degree 1)

3.1 Model

The linear model is

\hat y = \beta_1 x + \beta_0.

3.2 Least-squares solution (closed form)

Minimizing $\sum_i (y_i - (\beta_1 x_i + \beta_0))^2$ yields the standard closed-form solution:

\beta_1 = \frac{\sum_{i=1}^n (x_i - \bar x)(y_i - \bar y)}{\sum_{i=1}^n (x_i - \bar x)^2}, \qquad \beta_0 = \bar y - \beta_1 \bar x.

Expected behavior on this dataset

The fit captures the global trend.
Residuals retain structure (systematic nonlinearity), indicating model misspecification rather than random noise.
Training $R^2$ is limited by the inability of a line to represent curvature.

Linear regression

python

import warnings
warnings.filterwarnings("ignore")

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Data (same in every section)
rng = np.random.default_rng(0)
n = 220
x = np.sort(rng.uniform(0, 10, size=n))

y_true = 2.4 * x + 5.0 + 2.0 * np.sin(0.9 * x) + 0.12 * (x - 5.0) ** 2
y = y_true + rng.normal(0, 1.8, size=n)

def r2_score(y_true, y_pred):
  y_bar = np.mean(y_true)
  ss_tot = np.sum((y_true - y_bar) ** 2)
  ss_res = np.sum((y_true - y_pred) ** 2)
  return 1.0 - ss_res / ss_tot

# Closed-form LEast Square Solution:
x_mean = x.mean()
y_mean = y.mean()
b1 = np.sum((x - x_mean) * (y - y_mean)) / np.sum((x - x_mean) ** 2)
b0 = y_mean - b1 * x_mean

# Predictions
x_line = np.linspace(x.min(), x.max(), 600)
y_line = b0 + b1 * x_line
r2 = r2_score(y, b0 + b1 * x)

plt.rcParams.update({
  "figure.dpi": 140,
  "axes.spines.top": False,
  "axes.spines.right": False,
  "font.size": 11,
})

fig, ax = plt.subplots(1, 1, figsize=(8.4, 4.6), layout="constrained")
ax.scatter(x, y, s=34, alpha=0.78, edgecolors="white", linewidths=0.8, label="data")
ax.plot(x_line, y_line, linewidth=2.6, color="goldenrod", label="fit")
ax.set_title("Linear Regression")
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.grid(True, alpha=0.25)
ax.text(
  0.02, 0.98,
  f"$\\hat y = {b1:.3f}x + {b0:.3f}$\n$R^2 = {r2:.3f}$",
  transform=ax.transAxes, va="top", ha="left",
  bbox=dict(boxstyle="round,pad=0.35", facecolor="white", alpha=0.92, linewidth=0.0),
)
ax.legend(frameon=False, loc="lower right")
plt.show()

Idle.

4. Quadratic regression (degree 2)

4.1 Model

A quadratic regressor is

\hat y = a_2 x^2 + a_1 x + a_0.

This is nonlinear in $x$ but linear in parameters $(a_2,a_1,a_0)$ .

4.2 Design matrix formulation

Define the design matrix

X = \begin{bmatrix} x_1^2 & x_1 & 1 \\ x_2^2 & x_2 & 1 \\ \vdots & \vdots & \vdots \\ x_n^2 & x_n & 1 \end{bmatrix}, \qquad a = \begin{bmatrix} a_2 \\ a_1 \\ a_0 \end{bmatrix}, \qquad y = \begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix}.

Then $\hat y = Xa$ and the least-squares problem is

\min_a \|Xa - y\|_2^2.

Expected behavior on this dataset

Adds curvature and typically improves training fit relative to the linear model.
Still cannot represent oscillations well, but approximates the global quadratic component.
Training $R^2$ increases because the model class is strictly richer than degree 1.

Quadratic regression (least squares)

python

import warnings
warnings.filterwarnings("ignore")

import numpy as np
import matplotlib.pyplot as plt

# Data (same in every section)
rng = np.random.default_rng(0)
n = 220
x = np.sort(rng.uniform(0, 10, size=n))

y_true = 2.4 * x + 5.0 + 2.0 * np.sin(0.9 * x) + 0.12 * (x - 5.0) ** 2
y = y_true + rng.normal(0, 1.8, size=n)

def r2_score(y_true, y_pred):
  y_bar = np.mean(y_true)
  ss_tot = np.sum((y_true - y_bar) ** 2)
  ss_res = np.sum((y_true - y_pred) ** 2)
  return 1.0 - ss_res / ss_tot

# Least squares quadratic
X2 = np.column_stack([x**2, x, np.ones_like(x)])
a2, a1, a0 = np.linalg.lstsq(X2, y, rcond=None)[0]

x_line = np.linspace(x.min(), x.max(), 600)
y_line = a2 * x_line**2 + a1 * x_line + a0
r2 = r2_score(y, a2 * x**2 + a1 * x + a0)

plt.rcParams.update({
  "figure.dpi": 140,
  "axes.spines.top": False,
  "axes.spines.right": False,
  "font.size": 11,
})

fig, ax = plt.subplots(1, 1, figsize=(8.4, 4.6), layout="constrained")
ax.scatter(x, y, s=34, alpha=0.78, edgecolors="white", linewidths=0.8, label="data")
ax.plot(x_line, y_line, linewidth=2.6, color="goldenrod", label="fit")
ax.set_title("Quadratic Regression (Least Squares)")
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.grid(True, alpha=0.25)
ax.text(
  0.02, 0.98,
  f"$\\hat y = {a2:.3f}x^2 + {a1:.3f}x + {a0:.3f}$\n$R^2 = {r2:.3f}$",
  transform=ax.transAxes, va="top", ha="left",
  bbox=dict(boxstyle="round,pad=0.35", facecolor="white", alpha=0.92, linewidth=0.0),
)
ax.legend(frameon=False, loc="lower right")
plt.show()

Idle.

5. Cubic regression (degree 3)

5.1 Model

The cubic model is

\hat y = c_3 x^3 + c_2 x^2 + c_1 x + c_0.

5.2 Least squares (linear in parameters)

As with degree 2, we construct a polynomial design matrix with columns $(x^3, x^2, x, 1)$ and solve a linear least-squares problem.

Expected behavior on this dataset

Greater flexibility enables closer tracking of systematic structure.
Training error typically decreases again.
Increased flexibility also increases sensitivity to noise, especially near domain boundaries.

Cubic regression (least squares)

python

import warnings
warnings.filterwarnings("ignore")

import numpy as np
import matplotlib.pyplot as plt

# Data (same in every section)
rng = np.random.default_rng(0)
n = 220
x = np.sort(rng.uniform(0, 10, size=n))

y_true = 2.4 * x + 5.0 + 2.0 * np.sin(0.9 * x) + 0.12 * (x - 5.0) ** 2
y = y_true + rng.normal(0, 1.8, size=n)

def r2_score(y_true, y_pred):
  y_bar = np.mean(y_true)
  ss_tot = np.sum((y_true - y_bar) ** 2)
  ss_res = np.sum((y_true - y_pred) ** 2)
  return 1.0 - ss_res / ss_tot

# Least squares cubic
X3 = np.column_stack([x**3, x**2, x, np.ones_like(x)])
c3, c2, c1, c0 = np.linalg.lstsq(X3, y, rcond=None)[0]

x_line = np.linspace(x.min(), x.max(), 600)
y_line = c3 * x_line**3 + c2 * x_line**2 + c1 * x_line + c0
r2 = r2_score(y, c3 * x**3 + c2 * x**2 + c1 * x + c0)

plt.rcParams.update({
  "figure.dpi": 140,
  "axes.spines.top": False,
  "axes.spines.right": False,
  "font.size": 11,
})

fig, ax = plt.subplots(1, 1, figsize=(8.4, 4.6), layout="constrained")
ax.scatter(x, y, s=34, alpha=0.78, edgecolors="white", linewidths=0.8, label="data")
ax.plot(x_line, y_line, linewidth=2.6, color="goldenrod", label="fit")
ax.set_title("Cubic Regression (Least Squares)")
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.grid(True, alpha=0.25)
ax.text(
  0.02, 0.98,
  f"$\\hat y = {c3:.3f}x^3 + {c2:.3f}x^2 + {c1:.3f}x + {c0:.3f}$\n$R^2 = {r2:.3f}$",
  transform=ax.transAxes, va="top", ha="left",
  bbox=dict(boxstyle="round,pad=0.35", facecolor="white", alpha=0.92, linewidth=0.0),
)
ax.legend(frameon=False, loc="lower right")
plt.show()

Idle.

6. Degree-5 polynomial regression (degree 5)

6.1 Model

A degree-5 polynomial is

\hat y = d_5 x^5 + d_4 x^4 + d_3 x^3 + d_2 x^2 + d_1 x + d_0.

Expected behavior on this dataset

Training fit typically improves further (lower residual norms, higher training $R^2$ ).
However, improved training metrics do not imply improved test performance.
High-degree polynomials can show undesirable oscillations and boundary artifacts, especially without regularization.

This section exists to make the bias–variance point concrete: capacity increases training fit monotonically, but generalization is not monotone.

Degree-5 polynomial regression (least squares)

python

import warnings
warnings.filterwarnings("ignore")

import numpy as np
import matplotlib.pyplot as plt

# Data (same in every section)
rng = np.random.default_rng(0)
n = 220
x = np.sort(rng.uniform(0, 10, size=n))

y_true = 2.4 * x + 5.0 + 2.0 * np.sin(0.9 * x) + 0.12 * (x - 5.0) ** 2
y = y_true + rng.normal(0, 1.8, size=n)

def r2_score(y_true, y_pred):
  y_bar = np.mean(y_true)
  ss_tot = np.sum((y_true - y_bar) ** 2)
  ss_res = np.sum((y_true - y_pred) ** 2)
  return 1.0 - ss_res / ss_tot

# Least squares degree-5
X5 = np.column_stack([x**5, x**4, x**3, x**2, x, np.ones_like(x)])
d5, d4, d3, d2, d1, d0 = np.linalg.lstsq(X5, y, rcond=None)[0]

x_line = np.linspace(x.min(), x.max(), 600)
y_line = d5 * x_line**5 + d4 * x_line**4 + d3 * x_line**3 + d2 * x_line**2 + d1 * x_line + d0
r2 = r2_score(y, d5 * x**5 + d4 * x**4 + d3 * x**3 + d2 * x**2 + d1 * x + d0)

plt.rcParams.update({
  "figure.dpi": 140,
  "axes.spines.top": False,
  "axes.spines.right": False,
  "font.size": 11,
})

fig, ax = plt.subplots(1, 1, figsize=(8.4, 4.6), layout="constrained")
ax.scatter(x, y, s=34, alpha=0.78, edgecolors="white", linewidths=0.8, label="data")
ax.plot(x_line, y_line, linewidth=2.6, color="goldenrod", label="fit")
ax.set_title("Degree-5 Polynomial Regression (Least Squares)")
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.grid(True, alpha=0.25)
eq = (
  rf"$hat{{y}} = {d5:.3f}x^5 + {d4:.3f}x^4 + {d3:.3f}x^3 + "
  rf"{d2:.3f}x^2 + {d1:.3f}x + {d0:.3f}$"
  rf"$R^2 = {r2:.3f}$"
)

ax.text(
  0.02, 0.98, eq,
  transform=ax.transAxes, va="top", ha="left",
  bbox=dict(boxstyle="round,pad=0.35", facecolor="white", alpha=0.92, linewidth=0.0),
)
ax.legend(frameon=False, loc="lower right")
plt.show()

Idle.

7. Training metrics and interpretation

7.1 Coefficient of determination $R^2$

For observed targets $y$ and predictions $\hat y$ , the (training) coefficient of determination is

R^2 = 1 - \frac{\sum_{i=1}^n (y_i - \hat y_i)^2}{\sum_{i=1}^n (y_i - \bar y)^2}.

Key facts:

$R^2$ measures the fraction of variance (around $\bar y$ ) explained by the fitted model on the data evaluated.
When computed on the training set, $R^2$ usually increases with model flexibility.
A higher training $R^2$ does not certify better out-of-sample performance.

7.2 L2 error and RMSE (training set)

Two complementary training diagnostics are:

$\ell_2$ residual norm: $\|y-\hat y\|_2$
RMSE: $\sqrt{\frac{1}{n}\sum_i (y_i-\hat y_i)^2}$

Both decrease as the model class becomes more expressive (on training data), but neither prevents overfitting.

Compare L2 error across models (same data)

python

import warnings
warnings.filterwarnings("ignore")

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Data (must match the earlier sections exactly)
rng = np.random.default_rng(0)
n = 220
x = np.sort(rng.uniform(0, 10, size=n))

y_true = 2.4 * x + 5.0 + 2.0 * np.sin(0.9 * x) + 0.12 * (x - 5.0) ** 2
y = y_true + rng.normal(0, 1.8, size=n)

def l2_error(y_true, y_pred):
  return float(np.linalg.norm(y_true - y_pred, ord=2))

def rmse(y_true, y_pred):
  return float(np.sqrt(np.mean((y_true - y_pred) ** 2)))

# ---- Fit models ----

# Linear (closed-form Least Square Solution)
x_mean = x.mean()
y_mean = y.mean()
b1 = np.sum((x - x_mean) * (y - y_mean)) / np.sum((x - x_mean) ** 2)
b0 = y_mean - b1 * x_mean
yhat_lin = b0 + b1 * x

# Quadratic (least squares)
X2 = np.column_stack([x**2, x, np.ones_like(x)])
a2, a1, a0 = np.linalg.lstsq(X2, y, rcond=None)[0]
yhat_quad = a2 * x**2 + a1 * x + a0

# Cubic (least squares)
X3 = np.column_stack([x**3, x**2, x, np.ones_like(x)])
c3, c2, c1, c0 = np.linalg.lstsq(X3, y, rcond=None)[0]
yhat_cub = c3 * x**3 + c2 * x**2 + c1 * x + c0

# Degree-5 (least squares)
X5 = np.column_stack([x**5, x**4, x**3, x**2, x, np.ones_like(x)])
d5, d4, d3, d2, d1, d0 = np.linalg.lstsq(X5, y, rcond=None)[0]
yhat_deg5 = d5 * x**5 + d4 * x**4 + d3 * x**3 + d2 * x**2 + d1 * x + d0

models = [
("Linear", yhat_lin),
("Quadratic", yhat_quad),
("Cubic", yhat_cub),
("Degree-5", yhat_deg5),
]

# ---- Compute errors ----
# ---- Compute errors ----
labels = []
l2s = []
rmses = []
for name, yhat in models:
  labels.append(name)
  l2s.append(l2_error(y, yhat))
  rmses.append(rmse(y, yhat))

print("Training-set errors (same data used to fit):")
for name, l2v, rv in zip(labels, l2s, rmses):
  print(f"{name:9s}  ||y - yhat||_2 = {l2v:8.3f}   RMSE = {rv:6.3f}")

df = pd.DataFrame({
  "Model": labels,
  "L2 ||y-ŷ||2": l2s,
  "RMSE": rmses,
}).round({"L2 ||y-ŷ||2": 3, "RMSE": 3})

plt.rcParams.update({
  "figure.dpi": 170,
  "font.size": 11
})

fig, ax = plt.subplots(figsize=(8.2, 3.4), layout="constrained")
xpos = np.arange(len(labels))

ax.plot(xpos, rmses, linewidth=2.2, marker="o", markersize=6.5)

ax.set_title("Training RMSE across model classes", pad=10)
ax.set_ylabel("RMSE")
ax.set_xticks(xpos, labels)

ax.grid(True, which="major", axis="both", alpha=0.22, linewidth=1.0, color="#aaa")

for spine in ["top", "right"]:
  ax.spines[spine].set_visible(False)

for spine in ["left", "bottom"]:
  ax.spines[spine].set_color("#C9C9C9")
  ax.spines[spine].set_linewidth(1.2)

ax.tick_params(axis="both", colors="#6F6F6F")
ax.tick_params(axis="y", length=0)

ymin, ymax = float(min(rmses)), float(max(rmses))
pad = 0.18 * (ymax - ymin + 1e-12)
ax.set_ylim(max(0, ymin - 0.35 * pad), ymax + 1.25 * pad)

for x_i, v in zip(xpos, rmses):
  ax.text(x_i, v + 0.22 * pad, f"{v:.2f}",
          ha="center", va="bottom", fontsize=10, color="#444444")

ax.legend(["RMSE"], loc="upper center", bbox_to_anchor=(0.5, -0.18),
        frameon=False, ncol=1, handlelength=2.5)

plt.show()

Idle.

8. k-Nearest Neighbors (k-NN): a nonparametric local model

Polynomial regression is a parametric approach: the hypothesis class is determined by a fixed set of parameters (the coefficients). In contrast, k-Nearest Neighbors (k-NN) is a nonparametric method: it stores the training set and predicts using local neighborhoods at query time.

8.1 Classification vs. regression

k-NN can be used for both tasks:

k-NN classification: predict a class label by majority vote among the k nearest training points.
k-NN regression: predict a real value by averaging the neighbors’ targets (often with distance weights).

In both cases, the inductive bias is the same: nearby points in input space should have similar outputs.

8.2 Decision boundary geometry

For classification, the k-NN decision boundary is determined by the local arrangement of labeled samples:

Small k yields a high-variance boundary that can track noise.
Large k yields a smoother, higher-bias boundary that can wash out fine structure.

A common refinement is distance-weighted voting, which reduces sensitivity to the arbitrary choice of k by giving more influence to closer neighbors. One typical weighting is

w(d) = \exp(-\lambda d^2),

where $d$ is Euclidean distance and $\lambda > 0$ controls how quickly influence decays with distance.

k-NN classification (distance-weighted) and the effect of k

python

import warnings
warnings.filterwarnings("ignore")

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap

# -----------------------------
# 1) Toy dataset (2 classes, 2D)
# -----------------------------
rng = np.random.default_rng(0)

n = 80
X0 = rng.normal(loc=[-1.2, -0.8], scale=1.0, size=(n // 2, 2))
X1 = rng.normal(loc=[+1.2, +0.8], scale=1.0, size=(n // 2, 2))

# optional noisy points near the boundary
m = 10
X_noise = rng.normal(loc=[0.0, 0.0], scale=0.45, size=(m, 2))
y_noise = rng.integers(0, 2, size=m)

X = np.vstack([X0, X1, X_noise])
y = np.hstack([
  np.zeros(len(X0), dtype=int),
  np.ones(len(X1), dtype=int),
  y_noise
])

# -----------------------------
# 2) Distance-weighted k-NN probability for class 1
# -----------------------------
lam = 3.0  # fixed across plots

def knn_proba_weighted(X_train, y_train, X_query, k=5, lam=3.0):
  """Return p(y=1 | x) using distance-weighted voting.

Weight: w(d) = exp(-lam * d^2). For binary labels {0,1}, the weighted
  average of labels equals the estimated probability of class 1.
  """
  # Pairwise squared distances: ||x - x_i||^2
  # X_query: (Q,2), X_train: (N,2)
  diff = X_query[:, None, :] - X_train[None, :, :]
  D2 = np.sum(diff * diff, axis=2)  # (Q,N)

# Indices of k smallest distances (partial sort)
  nn_idx = np.argpartition(D2, kth=k-1, axis=1)[:, :k]  # (Q,k)
  D2k = np.take_along_axis(D2, nn_idx, axis=1)

W = np.exp(-lam * D2k)
  votes = y_train[nn_idx].astype(float)
  p1 = (W * votes).sum(axis=1) / (W.sum(axis=1) + 1e-12)
  return p1

# -----------------------------
# 3) Grid for plotting
# -----------------------------
pad = 0.8
x1_min, x1_max = X[:, 0].min() - pad, X[:, 0].max() + pad
x2_min, x2_max = X[:, 1].min() - pad, X[:, 1].max() + pad

gx1 = np.linspace(x1_min, x1_max, 320)
gx2 = np.linspace(x2_min, x2_max, 260)
G1, G2 = np.meshgrid(gx1, gx2)
G = np.c_[G1.ravel(), G2.ravel()]

region_cmap = ListedColormap(["#cfe3f5", "#fde0c2"])

# -----------------------------
# 4) Compare boundaries for different k
# -----------------------------
Ks = [1, 6, 16]  # low -> high k

plt.rcParams.update({
  "figure.dpi": 140,
  "axes.spines.top": False,
  "axes.spines.right": False,
  "font.size": 11,
})

fig, axes = plt.subplots(3, 1, figsize=(8.5, 12.5), layout="constrained")

for ax, k in zip(axes, Ks):
  P = knn_proba_weighted(X, y, G, k=k, lam=lam).reshape(G1.shape)

# hard regions from probability threshold
  ax.contourf(G1, G2, (P >= 0.5).astype(int),
              levels=[-0.5, 0.5, 1.5], cmap=region_cmap, alpha=0.35)

# decision boundary p=0.5
  ax.contour(G1, G2, P, levels=[0.5], colors="black", linewidths=1.0)

ax.scatter(X[y == 0, 0], X[y == 0, 1],
             s=34, alpha=0.95, edgecolors="none", color="#1f77b4", label="Class 0")
  ax.scatter(X[y == 1, 0], X[y == 1, 1],
             s=34, alpha=0.95, edgecolors="none", color="#ff7f0e", label="Class 1")

ax.set_title(f"k-NN (Distance-weighted): k = {k}", pad=10)
  ax.set_xlabel("x1")
  ax.set_ylabel("x2")
  ax.grid(True, alpha=0.25)

ax.legend(frameon=True, loc="lower right",
            bbox_to_anchor=(0.98, 0.02), edgecolor="black", facecolor="white")

ax.text(0.98, 0.95,
          rf"$k={k}, lambda={lam}$",
          transform=ax.transAxes, ha="right", va="top",
          bbox=dict(boxstyle="round,pad=0.3", facecolor="white", 
                  edgecolor="black", linewidth=1.0))

plt.show()

Idle.

9. Logistic Regression

Logistic regression models

p(y=1 \mid x) = \sigma(w^\top x + b),

with

\sigma(z) = \frac{1}{1+e^{-z}}.

Equivalently,

\log\frac{p(y=1 \mid x)}{p(y=0 \mid x)} = w^\top x + b.

The model is linear in the log-odds, but it produces calibrated probabilities rather than hard decisions.

10. The Logistic Loss

Let $z=w^\top x + b$ . Maximizing the conditional likelihood of labels is equivalent to minimizing the negative log-likelihood, which yields the logistic loss. For data $(x_i,y_i)$ :

\min_{w,b}\ \sum_{i=1}^n \left[\log(1+e^{z_i}) - y_i z_i\right].

This objective is convex in $(w,b)$ .

9. Demonstration: Logistic Regression Trained by Gradient Descent

The next cell trains logistic regression using NumPy gradient descent and visualizes the decision boundary.

Supervised classification with logistic regression (NumPy training)

python

import numpy as np
import matplotlib.pyplot as plt

# -----------------------------
# Toy supervised dataset (2 classes, 2 features)
# -----------------------------
rng = np.random.default_rng(0)
n = 160

X0 = rng.normal(loc=[-1.6, -1.0], scale=0.7, size=(n // 2, 2))
X1 = rng.normal(loc=[+1.6, +1.0], scale=0.7, size=(n // 2, 2))
X = np.vstack([X0, X1])
y = np.hstack([np.zeros(n // 2), np.ones(n // 2)]).astype(float)

# Add bias
Xb = np.c_[np.ones(len(X)), X]  # (n, 3)

def sigmoid(z):
  # numerically stable sigmoid
  z = np.clip(z, -40.0, 40.0)
  return 1.0 / (1.0 + np.exp(-z))

def nll_and_grad(w):
  z = Xb @ w
  # Stable negative log-likelihood: sum(log(1+exp(z)) - y*z)
  nll = np.sum(np.logaddexp(0.0, z) - y * z)
  p = sigmoid(z)
  grad = Xb.T @ (p - y)
  return nll, grad

# -----------------------------
# Gradient descent
# -----------------------------
w = np.zeros(Xb.shape[1])
lr = 0.05
steps = 800

for t in range(steps):
  loss, grad = nll_and_grad(w)
  w -= lr * grad / len(X)

# Predictions
proba = sigmoid(Xb @ w)
yhat = (proba >= 0.5).astype(int)
acc = (yhat == y.astype(int)).mean()

# -----------------------------
# Plot points + decision boundary
# -----------------------------
fig, ax = plt.subplots(figsize=(8.5, 5.2), dpi=130)

ax.scatter(X0[:, 0], X0[:, 1], s=28, alpha=0.85, edgecolors="none", label="Class 0")
ax.scatter(X1[:, 0], X1[:, 1], s=28, alpha=0.85, edgecolors="none", 
          label="Class 1", color="#380")

# Decision boundary: w0 + w1*x1 + w2*x2 = 0
x1_line = np.linspace(X[:, 0].min() - 0.8, X[:, 0].max() + 0.8, 250)
if abs(w[2]) > 1e-12:
  x2_line = -(w[0] + w[1] * x1_line) / w[2]
  ax.plot(x1_line, x2_line, linewidth=3, label="Decision boundary", color="goldenrod")
else:
  ax.axvline(-w[0] / w[1], linewidth=3, label="Decision boundary")

ax.set_title("Logistic regression: probabilities with a linear boundary", pad=10)
ax.set_xlabel("x1")
ax.set_ylabel("x2")
ax.grid(True, alpha=0.25, linewidth=1)
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
ax.legend(frameon=False, loc="lower left", bbox_to_anchor=(0.02, 0.02))

ax.text(0.60, 0.95,
      f"$p(y=1|x)=\sigma(w_0+w_1x_1+w_2x_2)$\n"
      f"$w=[{w[0]:.2f},{w[1]:.2f},{w[2]:.2f}],\ \mathrm{{acc}}={acc:.3f}$",
      transform=ax.transAxes, va="top")

plt.tight_layout()
plt.show()

Idle.

Regression in Machine Learning

Linear and Polynomial Regression

1. Problem setting

2. Data generation model

3. Linear regression (degree 1)

3.1 Model

3.2 Least-squares solution (closed form)

Linear regression

4. Quadratic regression (degree 2)

4.1 Model

4.2 Design matrix formulation

Quadratic regression (least squares)

5. Cubic regression (degree 3)

5.1 Model

5.2 Least squares (linear in parameters)

Cubic regression (least squares)

6. Degree-5 polynomial regression (degree 5)

6.1 Model

Degree-5 polynomial regression (least squares)

7. Training metrics and interpretation

7.1 Coefficient of determination R2R^2R2

7.2 L2 error and RMSE (training set)

Compare L2 error across models (same data)

8. k-Nearest Neighbors (k-NN): a nonparametric local model

8.1 Classification vs. regression

8.2 Decision boundary geometry

k-NN classification (distance-weighted) and the effect of k

9. Logistic Regression

10. The Logistic Loss

9. Demonstration: Logistic Regression Trained by Gradient Descent

Supervised classification with logistic regression (NumPy training)

7.1 Coefficient of determination $R^2$