Convolutional Neural Networks

1

The Convolution Operation

A small filter/kernel slides over the input, and at each position computes an element-wise multiply + sum — producing one output value per position. This detects local patterns (edges, textures) regardless of where they appear in the image.

Input (4×4)

1

2

3

0

1

2

1

0

3

1

0

2

1

4

×

Filter (2×2)

1

0

1

=

Output (3×3), stride 1

2

3

1

2

4

2

1

2

5

Top-left output cell (highlighted): (1×1 + 2×0 + 0×0 + 1×1) = 2 — the filter's top-left 2×2 window (blue) dotted with the filter (yellow), summed.

Why not just flatten and use an MLP? A convolution filter shares the same weights across every position in the image — far fewer parameters than a fully-connected layer, and it naturally detects the same pattern (e.g. an edge) no matter where it appears (translation invariance).

2

Padding & Stride

Padding adds a border of zeros around the input, so the filter can be applied to edge pixels properly and/or to keep output size equal to input size ("same" padding). Without it ("valid" padding), the output shrinks and edge information gets used less often than center pixels.
Stride is how many pixels the filter moves per step. Stride 1 = dense overlapping windows; stride 2 = filter skips every other position, roughly halving output size — a cheap way to downsample instead of (or alongside) pooling.

Output size formula:
output = ⌊ (input_size − filter_size + 2×padding) / stride ⌋ + 1

# check: input=4, filter=2, padding=0, stride=1
output = (4 - 2 + 2*0) / 1 + 1 = 3   # matches the 3x3 output above

3

Pooling

Max Pooling

Takes the max value in each window. Keeps the strongest activation (the clearest sign a feature is present) — the more common choice, especially for detecting "is this pattern present anywhere in this region."

Average Pooling

Takes the mean of each window. Smoother, less prone to picking up a single noisy spike, but can dilute a strong but localized signal.

Why pool at all? Reduces spatial dimensions (fewer parameters, less compute downstream), adds a bit of translation invariance (small shifts in the input barely change the pooled output), and helps control overfitting by summarizing rather than preserving every raw activation.

4

Why CNNs Beat Plain MLPs for Images

Parameter sharing: the same small filter is reused across the whole image instead of a separate weight per pixel per neuron — dramatically fewer parameters than a fully-connected layer on raw pixels.
Local connectivity: each neuron only looks at a small patch, matching the fact that nearby pixels are far more related than distant ones — an MLP treats every pixel pair as equally (un)related from the start.
Translation invariance: a feature (edge, texture) is detected the same way no matter where it appears in the image; an MLP would need to separately learn that feature at every possible location.
Hierarchical features: early layers learn edges/textures, later layers combine those into shapes and object parts — a natural hierarchy that plain MLPs don't structurally encourage.

5

Architecture Overview — Brief

Architecture	Key idea	Why it mattered
LeNet	Simple conv + pool + fully-connected stack	Proved CNNs work at all (digit recognition); the template everything else builds on.
VGG	Very deep, uniform small (3×3) filters stacked repeatedly	Showed depth + simplicity can beat hand-tuned filter sizes — but very parameter-heavy and slow.
ResNet	Skip/residual connections — add the input of a block back to its output	Solved the "deeper networks train worse" problem by letting gradients flow directly through the skip path, bypassing vanishing gradients in very deep nets.
Inception	Multiple filter sizes (1×1, 3×3, 5×5) applied in parallel in the same block, concatenated	Captures features at multiple scales simultaneously without having to guess one "right" filter size per layer.

Why ResNet works (the interview answer): instead of forcing each block to learn a full transformation from scratch, the skip connection lets it learn just the residual (the difference from the input) — if the ideal function for a block is close to "do nothing," that's trivial to learn (push weights toward zero). This also gives gradients a direct shortcut path back to early layers, largely sidestepping vanishing gradients even in networks hundreds of layers deep.

6

Transfer Learning

Start from a model pretrained on a large dataset (e.g. ImageNet) and adapt it to your task, instead of training from scratch.

Why: early/mid layers learn generic features (edges, textures, shapes) that transfer well across almost any vision task — reusing them saves enormous amounts of data and compute.
When: your dataset is small, your task is visually similar to the pretrained domain, or you have limited time/compute to train from scratch.
How, typically: freeze the early layers, replace/retrain the final classification layer(s) ("feature extraction"), or unfreeze and retrain the whole network at a low learning rate ("fine-tuning") once the new head has stabilized.

Trap: fine-tuning the entire pretrained network with a small dataset and a normal-sized learning rate can quickly overfit or destroy the useful pretrained features ("catastrophic forgetting") — use a small learning rate and/or freeze more layers when data is limited.

7

Data Augmentation

Artificially expands the training set by applying label-preserving transformations — helps the model generalize and reduces overfitting, especially with limited data.

Geometric: random flips, rotations, crops, zooms, translations.
Color/intensity: brightness/contrast jitter, hue shifts, adding noise.
Occlusion-style: random erasing/cutout — blanking out a patch forces the model not to over-rely on any single region.
Mixing methods: Mixup / CutMix — blend two images (and their labels) to create harder, more informative training examples.

Why it works: it's a cheap way to simulate more data / more real-world variation than you actually have, teaching the model to be invariant to changes that shouldn't affect the label (e.g. a cat is still a cat if flipped horizontally) — directly fighting overfitting without collecting new data.

8

CNN Layer Stack — Keras & PyTorch

# Keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout

model = Sequential([
    Conv2D(32, (3,3), activation='relu', padding='same', input_shape=(64,64,3)),
    MaxPooling2D((2,2)),
    Conv2D(64, (3,3), activation='relu', padding='same'),
    MaxPooling2D((2,2)),
    Flatten(),
    Dense(128, activation='relu'),
    Dropout(0.5),
    Dense(10, activation='softmax')
])

# PyTorch
import torch.nn as nn

model = nn.Sequential(
    nn.Conv2d(3, 32, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(32, 64, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Flatten(),
    nn.Linear(64*16*16, 128), nn.ReLU(), nn.Dropout(0.5),
    nn.Linear(128, 10)
)