Deep Learning Fundamentals

the "why" behind every building block

Perceptron → MLP

A single perceptron computes a weighted sum of inputs plus a bias, passes it through an activation, and outputs a value — essentially logistic regression with a step/sigmoid activation. It can only draw a straight-line decision boundary.

A Multi-Layer Perceptron (MLP) stacks several such layers with non-linear activations in between — this is what lets it approximate curved, complex decision boundaries a single perceptron can't.

Why stack layers at all? Stacking linear layers without non-linear activations between them collapses back into one big linear function — the non-linearity is what actually gives depth its extra representational power.

Activation Functions

Sigmoid

Squashes to (0,1). Saturates for large |x| → gradients near zero → slows/stalls learning in deep nets ("vanishing gradient"). Mostly reserved for binary output layers now.

Tanh

Squashes to (−1,1), zero-centered (a bit better than sigmoid for hidden layers) — but still saturates at the extremes, same vanishing-gradient issue.

ReLU

max(0, x). No saturation on the positive side, cheap to compute, gradient is either 0 or 1 — the default choice for hidden layers in most modern networks.

Leaky ReLU

Allows a small non-zero slope for x<0 (e.g. 0.01x) instead of a hard zero — fixes the "dying ReLU" problem where a neuron gets stuck always outputting 0.

Softmax

Converts a vector of raw scores into probabilities that sum to 1 — used in the output layer for multi-class classification, not hidden layers.

Why ReLU is preferred: it doesn't saturate for positive inputs, so gradients keep flowing during backprop (much less vanishing-gradient trouble than sigmoid/tanh); it's computationally trivial (just a max); and empirically it trains faster and deeper networks more reliably. Downside (dying ReLU) is usually manageable and fixed by Leaky ReLU or careful initialization/learning rate.

Forward & Backward Propagation (Intuition)

Forward pass: input flows layer by layer — each layer computes weighted sum + activation — until it produces a prediction, which is compared to the true label via the loss function.

Backward pass: the error is propagated backward through the network using the chain rule — each layer asks "how much did I contribute to the final error, and in which direction should my weights move to reduce it?" Gradients flow from the output layer back to the input layer, layer by layer, each one reusing the gradient computed by the layer after it.

Chain rule intuition, no derivation: think of it as passing blame backward — the loss tells the last layer "you're off by this much," that layer tells the one before it "because of you, I was off by this much," and so on. Each layer only needs the error signal handed to it by the next layer plus its own local derivative — it never needs to re-derive things from scratch.

Why this matters in interviews: the "why" isn't the math — it's that backprop is just efficient bookkeeping (dynamic programming, essentially) that reuses intermediate gradients instead of recomputing them for every weight independently.

Loss Functions

Loss	Use when	Why
MSE (Mean Squared Error)	Regression — continuous target	Penalizes larger errors more heavily; smooth gradient everywhere.
Cross-Entropy	Classification (binary or multi-class)	Directly measures the distance between predicted and true probability distributions; pairs naturally with sigmoid/softmax outputs and produces well-behaved gradients for classification.

Trap: using MSE for a classification problem gives much weaker gradients when predictions are confidently wrong (since the sigmoid output is squashed) — cross-entropy's gradient stays informative in exactly that case, which is why it's the standard for classification.

Optimizers

Optimizer	Idea	Practical note
SGD	Update weights using the gradient from one (or a mini-batch of) example(s)	Simple, but noisy path and can be slow to converge, especially in narrow/steep valleys.
Momentum	Adds a "velocity" term that accumulates past gradients	Smooths out the noisy zig-zag path and speeds convergence, like a ball rolling downhill gaining momentum.
RMSProp	Divides the learning rate by a running average of recent squared gradients per-parameter	Adapts step size per parameter — bigger steps for rarely-updated weights, smaller for frequently-large-gradient ones.
Adam	Combines Momentum (first moment) + RMSProp (second moment)	The default go-to optimizer in practice — fast convergence, works well with little tuning across many problem types.

Why Adam over plain SGD? Adam adapts the learning rate per parameter and smooths the gradient direction using momentum, so it converges faster and is far less sensitive to a poorly chosen learning rate — it "just works" in more situations. Plain SGD (especially with a well-tuned learning rate schedule) can sometimes generalize slightly better at the very end of training, which is why some practitioners switch to SGD for final fine-tuning after Adam gets them most of the way there.

Vanishing / Exploding Gradients

In deep networks, gradients are products of many layers' local derivatives (chain rule). If those local derivatives are consistently <1 (e.g. sigmoid/tanh saturation), the product shrinks toward zero as it propagates back — early layers barely learn (vanishing). If they're consistently >1, the product explodes (exploding), causing unstable, huge weight updates.

Problem	Common fixes
Vanishing gradient	ReLU-family activations, careful weight initialization (He/Xavier), batch normalization, residual/skip connections (as in ResNets), LSTM/GRU gating for sequence models.
Exploding gradient	Gradient clipping (cap the gradient norm), smaller learning rate, careful weight initialization.

Batch Normalization

Normalizes each layer's inputs (per mini-batch) to have roughly zero mean and unit variance, then applies a learned scale/shift — keeps activations in a stable, well-behaved range as they flow through the network.

Why it works: it reduces "internal covariate shift" — the constant reshuffling of each layer's input distribution as earlier layers' weights update during training — letting deeper networks train faster and more stably, tolerate higher learning rates, and act as a mild regularizer (since batch statistics add a bit of noise).

Dropout

During training, randomly "switches off" a fraction of neurons (e.g. 20–50%) on each forward pass, forcing the network not to rely too heavily on any single neuron or co-adapted group of neurons.

Why it works: it's like training a large ensemble of thinner sub-networks that share weights, and averaging their combined effect at test time (when dropout is turned off and outputs are scaled accordingly) — this discourages overfitting to specific neuron combinations and improves generalization, similar in spirit to how Random Forest de-correlates individual trees.

Interview line: "Dropout is regularization for neural nets the same way bagging is for trees — both work by preventing over-reliance on any one part of the model."

Basic MLP — Keras & PyTorch

# Keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, BatchNormalization

model = Sequential([
    Dense(64, activation='relu', input_shape=(n_features,)),
    BatchNormalization(),
    Dropout(0.3),
    Dense(32, activation='relu'),
    Dense(1, activation='sigmoid')   # binary classification
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=20, batch_size=32, validation_split=0.2)

# PyTorch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(n_features, 64),
    nn.ReLU(),
    nn.BatchNorm1d(64),
    nn.Dropout(0.3),
    nn.Linear(64, 32),
    nn.ReLU(),
    nn.Linear(32, 1),
    nn.Sigmoid()
)

criterion = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)