NLP fundamentals

* interview cheat-sheet — preprocessing → representation → embeddings → pipeline

1 text preprocessing

Goal: strip noise, shrink vocabulary, normalize text into a form a model can actually learn from — every step below trades signal loss for less noise / smaller vocab, so skip a step whenever the thing it removes is actually meaningful for your task.

tokenization — split raw text into words/subwords/sentences. Foundation step — everything downstream operates on tokens, not raw strings.
lowercasing — "Cat" and "cat" collapse to one vocab entry, shrinks vocab size. Skip when case carries meaning (NER — "Apple" the company vs "apple" the fruit — or sentiment, where CAPS ≈ shouting).
stopword removal — drop high-frequency, low-information words (the, is, at). Cuts dimensionality and noise for search/classification. Skip for tasks where function words matter — phrase search, translation, or sentiment/negation ("not good" loses its meaning if "not" is removed).

stemming vs lemmatization stemming = crude rule-based chopping ("studies" → "studi") — fast, can produce non-words. lemmatization = dictionary + POS-aware, returns a real root ("studies" → "study", "better" → "good") — slower, more accurate.

When to use which: stemming for speed at scale — search engines, large-scale IR, quick baselines where crude matching is fine. Lemmatization when the actual word meaning matters downstream — feeding into embeddings, chatbots, anything sensitive to semantic accuracy.

2 Bag of Words vs TF-IDF

Bag of Words: represent each document as a vector of raw word counts (or just presence/absence), order thrown away entirely. Simple, but treats every word as equally important — "the" ends up with a huge count and drowns out words that actually distinguish documents.

TF-IDF fixes that by weighting each word by how discriminative it is:

TF(t, d) = count of t in d / total terms in d IDF(t) = log( N / (1 + df(t)) ) TF-IDF(t, d) = TF(t, d) × IDF(t)

N = total docs · df(t) = number of docs containing t

Why it's better than BoW: a word appearing in almost every document (low IDF) gets downweighted automatically — no need to hand-pick a stopword list. A word that's rare across the corpus but frequent in one document (high TF, high IDF) gets boosted — it's genuinely informative for that document.

Still a bag — no word order, no semantics, and vectors are high-dimensional and sparse (mostly zeros). That gap is exactly what embeddings solve next.

3 Word2Vec & GloVe — dense embeddings

Why embeddings capture meaning: built on the distributional hypothesis — "you shall know a word by the company it keeps." Words that show up in similar contexts get pushed to similar points in vector space during training. No one tells the model "king" and "queen" are related — it falls out of the fact that both words appear near similar neighboring words across millions of sentences.

Result: dense, low-dimensional vectors (e.g. 100–300 numbers per word, vs tens of thousands for BoW) where distance and direction mean something — the classic example being vector("king") − vector("man") + vector("woman") ≈ vector("queen").

CBOW averages context to guess one word · skip-gram uses one word to guess its neighbors

CBOW vs skip-gram, in one line CBOW is faster and smooths over frequent words well; skip-gram is slower but represents rare words better since it generates more training pairs per sentence.

GloVe takes a different route to the same kind of vectors: instead of predicting words from context (a neural, local-window approach like Word2Vec), it builds a global word-word co-occurrence matrix over the whole corpus and factorizes it directly. Both land on dense vectors that capture similar relationships — Word2Vec learns them predictively and locally, GloVe learns them from global counts.

4 end-to-end classification pipeline

split into train/test before vectorizing — fit the vectorizer only on train to avoid leakage

classic path: TF-IDF features → Logistic Regression / Naive Bayes / linear SVM — strong, fast baselines, easy to explain
modern path: pretrained embeddings or transformer encoders feed a classifier head — better on nuance/context, costs more compute
always hold out a test set and check a real metric (F1, not just accuracy) — especially with imbalanced classes

5 code — TF-IDF + classifier

scikit-learn

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score

X_train, X_test, y_train, y_test = train_test_split(
    texts, labels, test_size=0.2, random_state=42
)

vectorizer = TfidfVectorizer(stop_words="english", max_features=5000)
X_train_tfidf = vectorizer.fit_transform(X_train)   # fit only on train
X_test_tfidf  = vectorizer.transform(X_test)         # reuse the same vocab

clf = LogisticRegression(max_iter=1000)
clf.fit(X_train_tfidf, y_train)

preds = clf.predict(X_test_tfidf)
print(f1_score(y_test, preds, average="weighted"))

30-second recap: preprocess to shrink noise (tokenize → lowercase → drop stopwords → stem or lemmatize) → turn text into numbers (BoW counts, or TF-IDF to downweight common words) → or go dense with Word2Vec/GloVe, where similar contexts pull words to similar vectors → feed vectors into a classifier, evaluate on held-out data with F1, not just accuracy. — good luck in there.