Model Evaluation & Tuning

bias-variance · cross-validation · hyperparameter search

Bias-Variance Tradeoff

Bias is error from overly simplistic assumptions (model too rigid to capture the real pattern). Variance is error from being too sensitive to the specific training data (model captures noise, not just signal). Total error ≈ bias² + variance + irreducible noise.

As model complexity grows, bias drops but variance climbs — total error is minimized somewhere in the middle, not at either extreme.

Interview line: "A high-bias model underfits both train and test data similarly. A high-variance model does great on train but much worse on test — that gap is the tell."

Overfitting vs Underfitting

Underfitting (high bias)

Model is too simple to capture the pattern — poor performance on both train and test/validation data.

Overfitting (high variance)

Model memorizes training data, including noise — great train performance, much worse test performance.

How do you know your model isn't overfitting? (the practical checklist)

Detect it:

Compare train vs validation metric — a large gap (e.g. 98% train accuracy, 74% validation) is the classic red flag.
Plot a learning curve: training error stays very low while validation error plateaus much higher and doesn't close the gap as you add data.
Use k-fold cross-validation and check the variance across folds — wildly different scores per fold hints at instability/overfitting.
Check performance degradation over time in production (a model that overfit to a stale training distribution decays faster).

Fix overfitting:

More training data, or data augmentation.
Regularization (L1/L2, dropout for neural nets).
Simplify the model — fewer features, shallower tree, lower polynomial degree.
Early stopping (stop training once validation error starts rising again).
Ensemble/bagging methods (Random Forest averages out individual trees' overfitting).

Fix underfitting:

Add more/better features, or reduce regularization strength.
Increase model complexity — deeper tree, higher polynomial degree, more layers/neurons.
Train longer / reduce excessive early stopping.

Interview trap: "My model gets 99% accuracy" is not automatically good news — always ask about the train/test gap and class balance before celebrating a number.

Cross-Validation Types

A single train/test split can be lucky or unlucky — cross-validation averages performance across multiple splits for a more reliable estimate.

K-Fold

Split data into k equal parts; train on k−1, validate on the remaining one, rotate k times, average the results. Default choice for i.i.d. tabular data.

Stratified K-Fold

Like k-fold, but preserves the class proportion in every fold. Required for classification, especially with imbalanced classes — plain k-fold can accidentally create folds with very few (or zero) minority-class examples.

Time-Series Split

Folds respect chronological order — always train on the past, validate on the future, expanding the training window each time. Required whenever data has a time component; regular k-fold would leak future information into training.

Interview line: "For time-series, I'd never randomly shuffle folds — that would let the model 'see the future' during training, giving an unrealistically optimistic validation score."

Hyperparameter Tuning Methods

Method	How it searches	Pros	Cons
GridSearchCV	Tries every combination in a specified parameter grid	Exhaustive, guaranteed to try the best combo in the grid	Expensive — combinations explode as parameters/values grow
RandomizedSearchCV	Samples a fixed number of random combinations from the parameter space	Much cheaper than grid search; often finds near-optimal settings with far fewer trials	Might miss the true best combo; still somewhat inefficient (doesn't learn from past trials)
Bayesian Optimization (e.g. Optuna)	Builds a probabilistic model of "which regions look promising," and picks the next trial based on past results	Converges to good hyperparameters in fewer trials — smart, adaptive search	More setup/complexity; less parallelizable than grid/random search (trials depend on prior outcomes)

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

param_grid = {
    'n_estimators': [100, 200, 300],
    'max_depth': [None, 5, 10],
    'min_samples_split': [2, 5, 10]
}

grid = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring='f1',
    n_jobs=-1
)
grid.fit(X_train, y_train)

print(grid.best_params_)
print(grid.best_score_)

Optuna basics: define an objective(trial) function that suggests hyperparameter values (trial.suggest_float, trial.suggest_int), trains/evaluates the model, and returns a score. Optuna's sampler (TPE by default) uses past trial results to intelligently choose the next set of values to try — pruning unpromising trials early to save compute.

Interview line: "For a small grid I'd just use GridSearchCV. For a big search space or expensive models, I'd reach for RandomizedSearchCV first, and Optuna/Bayesian optimization if I have compute budget and want to squeeze out more performance efficiently."