RNN → LSTM → GRU

* interview cheat-sheet — gate intuition, not BPTT math

1 why sequences need RNNs

A plain feedforward net treats every input as fixed-size and independent — it has no idea that word 5 came after word 4.

Sequential data (text, speech, sensor streams, stock prices) has order and context baked into it — meaning depends on what came before.

RNN keeps a hidden state h_t — a running summary updated every step:
h_t = tanh( W_xh · x_t + W_hh · h_(t-1) + b )
Same weights W_xh, W_hh reused at every timestep → handles variable-length sequences with a fixed param count

the error has to travel back across every one of these arrows during training ↴

2 vanishing gradient (why vanilla RNN forgets)

Backprop-through-time multiplies the same weight matrix (and the tanh derivative, which is ≤ 1) over and over — once per timestep, going backward.

Multiply a bunch of numbers < 1 together → the product shrinks toward zero exponentially with sequence length.

Interview one-liner: "gradients from far-back timesteps shrink to ~0 before they reach early layers, so the network can't learn long-range dependencies — it effectively forgets anything more than a few steps back."

Values > 1 repeatedly multiplied → exploding gradient instead (fixed with gradient clipping)
This is the whole reason LSTM/GRU exist — give the network a path where gradients don't get crushed

3 LSTM — gated memory

Big idea: add a cell state C_t — a conveyor belt that runs straight through the sequence, barely touched, so info (and gradient) can survive many steps.

Three gates decide what happens to that belt. Each gate is a sigmoid layer → outputs numbers 0–1, i.e. "how much to let through."

forget → decide what to erase · input → decide what to write · output → decide what to reveal

remember it as: forget gate = eraser, input gate = pencil, output gate = the window you're allowed to read C through.

4 GRU — LSTM, simplified

GRU asks: do we really need a separate cell state and 3 gates? It merges things down:

no separate cell state — only hidden state h_t carries memory (C_t and h_t become one)
forget + input gates merged into a single update gate z_t — one knob that trades off "keep old h" vs "write new content", instead of two independent gates
adds a reset gate r_t — controls how much of the past hidden state feeds into the new candidate

Net effect: fewer parameters, fewer matrix multiplies per step → trains faster, and on many tasks performs about as well as LSTM.

Interview one-liner: "GRU = LSTM with the cell state and hidden state fused, and the forget/input gates fused into one update gate — same core idea, leaner."

	LSTM	GRU
states carried	C_t and h_t	h_t only
gates	forget, input, output	update, reset
params	more	~25% fewer

5 sequence-to-sequence (encoder–decoder)

Used when input length ≠ output length — e.g. translating a 5-word sentence into a 7-word one.

decoder generates output tokens one at a time, conditioned on that context vector

gotcha to mention: a single fixed-size context vector is a bottleneck for long inputs — that limitation is exactly what motivated attention (lets the decoder look back at all encoder states, not just the last one).

6 where these actually get used

time series forecasting stock prices, weather, sensor / IoT readings — model runs over past values, predicts the next step (or several steps ahead). Order and recency matter a lot here.

text generation char- or word-level language models — feed in tokens so far, predict a distribution over the next token, sample, repeat. Same trick under early chatbots and autocomplete.

also: speech recognition, sentiment/sequence classification, video captioning, anomaly detection in logs — anywhere "what happened before" changes the meaning of "what's happening now."

7 code — LSTM layer

PyTorch

import torch.nn as nn

lstm = nn.LSTM(
    input_size=64,     # features per timestep
    hidden_size=128,   # size of h_t / C_t
    num_layers=2,
    batch_first=True   # x shape: (batch, seq_len, input_size)
)

output, (h_n, c_n) = lstm(x)
# output: h_t for every timestep -> (batch, seq_len, hidden_size)
# h_n, c_n: final hidden & cell state -> (num_layers, batch, hidden_size)

Keras

from tensorflow.keras.layers import LSTM

lstm = LSTM(
    units=128,             # hidden size
    return_sequences=True  # False -> only last h_t
)
output = lstm(x)  # x shape: (batch, seq_len, input_size)

30-second recap: RNN carries a hidden state forward, but repeated multiplication during BPTT kills gradients over long sequences → LSTM fixes this with a cell-state highway plus forget/input/output gates → GRU strips that down to one update gate and one reset gate, no separate cell state → seq2seq chains an encoder and decoder RNN through a context vector for input-length ≠ output-length tasks like translation. — good luck in there.