Unsupervised Learning

clustering & dimensionality reduction — interview notes

K-Means Clustering

Partitions data into k clusters by iterating: (1) assign each point to its nearest centroid, (2) recompute centroids as the mean of their assigned points — repeat until stable.

Elbow method: plot inertia vs k. Pick k at the "bend," where adding more clusters stops giving much extra improvement.

from sklearn.cluster import KMeans

inertias = []
for k in range(1, 10):
    km = KMeans(n_clusters=k, random_state=42, n_init='auto')
    km.fit(X)
    inertias.append(km.inertia_)

Limitations: must pick k in advance; assumes roughly spherical, similar-sized clusters (struggles with elongated or nested shapes); sensitive to centroid initialization (mitigated by k-means++, the sklearn default); sensitive to outliers, since means get pulled by extreme points; needs scaled features (distance-based).

Hierarchical Clustering

Builds a tree of nested clusters. Agglomerative (bottom-up, far more common): start with every point as its own cluster, repeatedly merge the two closest clusters until one remains.

Dendrogram: the height of each merge shows how (dis)similar the merged clusters were. Cutting horizontally at a chosen height reveals that many clusters — no need to pick k upfront.

Why it's useful: no need to fix k beforehand — inspect the dendrogram and pick a cut height afterward. Also captures nested cluster structure that flat clustering (like K-Means) can't represent. Downside: O(n²) or worse in time/memory, so it scales poorly to very large datasets.

DBSCAN (Density-Based)

Groups points that are closely packed together (dense regions), marking points in low-density regions as noise instead of forcing them into a cluster. Driven by two parameters: eps (neighborhood radius) and min_samples (minimum points to form a dense region).

Two dense clusters (blue, green) found automatically; the isolated point (pink) is labeled noise, not forced into either cluster.

	K-Means	DBSCAN
Cluster shape	Assumes roughly spherical/convex	Finds arbitrary shapes (crescents, rings)
Number of clusters	Must specify k	Discovered automatically from density
Outliers	Forced into nearest cluster, skewing centroids	Explicitly labeled as noise, doesn't distort clusters
Parameter sensitivity	Sensitive to k, initialization	Sensitive to eps/min_samples; struggles with varying-density clusters

Interview line: "I'd reach for DBSCAN over K-Means when I expect noise/outliers in the data or the clusters aren't roughly round — e.g. geographic/GPS clustering or anomaly-adjacent tasks."

PCA (Principal Component Analysis)

Finds new axes (principal components) — combinations of original features — that capture the most variance in the data, ordered from most to least. Projecting onto the top few components reduces dimensions while keeping most of the information.

When / why to reduce dimensions: speeds up downstream models, fights the curse of dimensionality, removes multicollinearity (components are orthogonal/uncorrelated by construction), helps visualization (2D/3D plots), and can reduce noise if later components are mostly noise.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)   # scale first — PCA is variance-based

pca = PCA(n_components=5)
X_reduced = pca.fit_transform(X_scaled)

print(pca.explained_variance_ratio_)          # variance captured per component
print(pca.explained_variance_ratio_.cumsum())  # running total — pick components covering ~90-95%

Trap: always scale features before PCA — otherwise a feature with a naturally larger numeric range (e.g. income vs age) dominates the variance and hijacks the first component for the wrong reason. Also, components lose direct interpretability — "PC1" is a mix of original features, not a single business meaning.

t-SNE

A non-linear technique that maps high-dimensional data into 2D/3D specifically to preserve local neighborhood structure for visualization — points close together in high-D stay close in the plot.

t-SNE vs PCA: PCA is linear, fast, deterministic, and preserves global variance/structure — good for actual dimensionality reduction feeding into a model. t-SNE is non-linear, slower, non-deterministic (depends on random init/perplexity), and distorts global distances/densities — it's a visualization tool, not a general preprocessing step for modeling.

Great for eyeballing whether clusters/classes visually separate in a complex dataset.
Cluster sizes and distances between clusters in a t-SNE plot are not meaningful — only relative local groupings are.
Sensitive to the perplexity parameter — different values can tell different visual "stories" from the same data.

Common mistake: feeding t-SNE output into a downstream ML model, or over-interpreting distances between t-SNE clusters — it was never designed to preserve those, only to make visual inspection easier.

★

Comparison: Clustering & Dimensionality Reduction

Clustering: K-Means vs Hierarchical vs DBSCAN

	K-Means	Hierarchical	DBSCAN
Needs k upfront?	Yes	No — cut dendrogram later	No — density-driven
Cluster shape	Spherical / convex	Flexible, depends on linkage	Arbitrary shape
Handles outliers?	Poorly — skews centroids	Poorly — still gets merged in	Well — labels as noise
Scalability	Good — scales to large n	Poor — O(n²)+ typically	Moderate — depends on implementation
Best for	Large, roughly round clusters	Small/medium data, nested structure, no known k	Noisy data, irregular cluster shapes

Dimensionality Reduction: PCA vs t-SNE

	PCA	t-SNE
Type	Linear	Non-linear
Goal	Preserve global variance	Preserve local neighborhoods
Use case	Preprocessing / feature reduction before modeling	Visual exploration of clusters/classes
Deterministic?	Yes	No — depends on init & perplexity
Speed	Fast, scales well	Slow on large datasets