Regression Algorithms

ML interview notes — the "why," not the derivation

Linear Regression

Fits a straight line (or hyperplane) y = β₀ + β₁x₁ + ... + βₙxₙ that best predicts a continuous target.

Cost function & OLS

OLS (Ordinary Least Squares) finds the coefficients that minimize the sum of squared errors between predicted and actual values — squaring penalizes big misses much more than small ones and keeps the math differentiable.

Key assumptions

Linearity: relationship between X and y is linear.
Independence: residuals aren't correlated with each other (no autocorrelation — matters a lot for time series).
Homoscedasticity: residual variance is constant across all predicted values (not fanning out).
Normality of residuals: mainly needed for valid confidence intervals / hypothesis tests, less so for pure prediction.
No severe multicollinearity: predictors shouldn't be highly correlated with each other.

Why does it fail? Non-linear relationships (a curved trend forced through a straight line), multicollinearity (unstable, flip-flopping coefficients), outliers (squared error lets a few extreme points dominate the fit), or heteroscedasticity (model over/under-estimates error in different ranges).

Explain to a stakeholder: "Each coefficient tells us how much the outcome changes for a one-unit increase in that factor, holding everything else constant" — e.g. "each extra bedroom adds about $18k to predicted price."

Ridge vs Lasso vs ElasticNet

Regularization adds a penalty for large coefficients to the cost function — it trades a little bias for a lot less variance, which fights overfitting.

Ridge (L2)

Penalty ∝ sum of squared coefficients. Shrinks all coefficients toward zero but rarely to exactly zero. Good when most features are somewhat useful and are correlated with each other.

Lasso (L1)

Penalty ∝ sum of absolute coefficients. Can shrink some coefficients to exactly zero — performs automatic feature selection. Good when you suspect many features are irrelevant.

ElasticNet

Mix of L1 + L2. Gets Lasso's feature selection while handling groups of correlated features more gracefully than Lasso alone (which tends to arbitrarily pick one and zero out the rest).

When regularization matters: many features relative to rows, high multicollinearity, or a model that fits training data suspiciously well but generalizes poorly — regularization is the standard fix before reaching for a different algorithm entirely.

Explain to a stakeholder: "We're deliberately making the model a bit more conservative on the training data so it doesn't overreact to noise and performs more reliably on new data."

Polynomial Regression

Adds powers of existing features (x², x³, ...) as new inputs to a still-linear model — it's "linear in the coefficients" even though the curve it draws is not a straight line.

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)   # adds x^2 terms

Why does it fail? Higher degrees fit training noise, not signal — the curve wiggles wildly outside the training range (poor extrapolation) even while training error keeps dropping. Classic bias-variance tradeoff: low degree underfits, high degree overfits.

Explain to a stakeholder: "The relationship isn't a straight line — for example, price might rise fast with size at first, then level off. This lets the model bend to match that pattern instead of forcing a straight-line assumption."

Gradient Descent Intuition

An iterative way to find the coefficients that minimize the cost function — take small steps downhill on the cost "surface" in the direction that reduces error fastest.

Batch GD

Uses the entire dataset to compute each gradient step. Stable, accurate direction, but slow and memory-heavy on large data.

Stochastic GD (SGD)

Updates using one random row at a time. Fast, can escape shallow local minima, but noisy — the path zig-zags toward the minimum.

Mini-batch GD

Updates using a small batch (e.g. 32–256 rows). The practical default — balances speed and stability, and is what most deep learning training actually uses.

Role of the learning rate

Too small: converges, but painfully slowly — many iterations needed.
Too large: can overshoot the minimum and diverge, or oscillate without ever settling.
Just right / decayed over time: fast, stable convergence — many optimizers (Adam, etc.) adapt it automatically.

Why does it fail? Poorly scaled features distort the cost surface into a narrow, elongated valley — gradient descent zig-zags slowly instead of heading straight to the minimum. This is exactly why features are standardized before gradient-based models.

Evaluation Metrics

Metric	What it measures	When to use
MAE	Average absolute error, same units as target	Easiest to explain to non-technical stakeholders; robust to outliers.
MSE	Average squared error	Penalizes large errors heavily — use when big misses are especially costly.
RMSE	Square root of MSE — back in original units	Like MSE's outlier sensitivity, but easier to interpret on the target's own scale.
R²	Proportion of variance in y explained by the model	Quick "how good is the fit" summary; 0–1 scale (can go negative for a bad fit).
Adjusted R²	R² penalized for adding features that don't help	Comparing models with different numbers of predictors — plain R² always rises (or stays flat) as you add features, even useless ones.

Explain MAE to a stakeholder: "On average, our price predictions are off by about $8,200 in either direction."

Explain R² to a stakeholder: "Our model explains about 72% of the variation in sales — the rest comes from factors we haven't captured."

Trap: a high R² doesn't mean a good model — it can be inflated by adding irrelevant features (use Adjusted R²) or hide poor performance on the minority of high-value cases that matter most to the business.

sklearn: Fit + Metrics

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
import numpy as np

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

mae  = mean_absolute_error(y_test, y_pred)
mse  = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
r2   = r2_score(y_test, y_pred)

print(f"MAE: {mae:.2f} | RMSE: {rmse:.2f} | R2: {r2:.3f}")