SVD and PCA

Eigen-decompositions love square matrices with enough eigenvectors. Data matrices are usually rectangular: rows as samples, columns as features. You still want a stretch story - which directions in feature space carry energy, which combinations of samples matter - without pretending the matrix is square.

The SVD was discovered in the nineteenth century independently by Eugenio Beltrami and Camille Jordan, with later contributions from Sylvester, Schmidt, Weyl, and others (standard surveys such as Stewart's SIAM Review piece map the lineage). Low-rank approximation theorems often associated with Eckart-Young have earlier infinite-dimensional roots in Schmidt's work. Separately, Karl Pearson (1857-1936) and later Hotelling developed principal-component ideas for multivariate data; MacTutor sketches Pearson's biometric and correlation program. Steal Pearson's ethic for the data-facing half: when variables tangle, seek clearer coordinates backed by measurement. Keep the SVD invention story layered, not monolothic.

For a modern rereader, SVD is the factorization; PCA is the statistical mission that often uses SVD as its engine.

Any real $m\times n$ matrix $A$ admits a singular value decomposition

(with orthogonal $U$, $V$ and rectangular diagonal $\Sigma$ of nonnegative singular values $\sigma_1\ge\sigma_2\ge\cdots$). Geometry slogan: rotation (or reflection), axis-aligned scaling, rotation. $V$'s columns are right singular vectors (input directions); $U$'s columns are left singular vectors (output directions); $\sigma_i$ are stretch factors.

Truncating to the top $k$ singular values gives the best rank-$k$ approximation to $A$ in the usual Frobenius/spectral senses (Eckart-Young-Mirsky theorem family). That is compression with a theorem.

Principal component analysis (PCA) asks a statistical question: given centered data, which orthonormal directions capture the most variance? For a centered data matrix $X$, PCA directions are eigenvectors of the covariance (proportional to $X^{\mathsf{T}}X$), equivalently the right singular vectors of $X$. Projecting onto the top components yields a variance-maximizing low-dimensional basis.

Centering matters: without subtracting means, the first "component" often just points at the mean offset rather than meaningful spread.

Imagine $A$ acting on a sphere of input vectors. The image is an ellipsoid. Singular vectors align with the ellipsoid's axes; singular values are the axis lengths. That picture works even when $A$ is rectangular: it maps between spaces of different dimension.

PCA on a cloud of points: find the line that maximizes projected variance, then the next orthogonal line, and so on. SVD of the centered cloud's matrix computes those axes efficiently.

Least squares connects: SVD yields stable solutions even when $A$ is rank-deficient, via the pseudoinverse built from reciprocal singular values (with care for tiny $\sigma$).

Algebrica has strong eigen and variance pages; use them as spectral and statistical companions while 3Blue1Brown supplies the SVD visual.

SVD is "just" eigenvalues of $A$. Singular values of $A$ are square roots of eigenvalues of $A^{\mathsf{T}}A$ (and of $AA^{\mathsf{T}}$). Nouns differ; kinship is real.

PCA without centering. Often a silent bug. Mean structure masquerades as a principal direction.

More components always better. For compression and denoising, trailing components can be noise. For prediction, PCA features are unsupervised: high variance need not mean high label relevance.

Neural embeddings are PCA. Sometimes PCA is a strong baseline. Deep features are nonlinear and task-tuned. Honest bridge: PCA is linear variance geometry; modern representation learning is a broader game.

Singular values equal explained variance automatically. In PCA they relate after centering and appropriate scaling. Track your normalization.

Orthogonal components mean independent causes. Orthogonality is geometric. Independence is probabilistic. Do not smuggle causation.

Beltrami and Jordan opened the SVD; later mathematicians extended and applied it; mid-twentieth-century numerical linear algebra (Golub and others) made computing SVD practical. Pearson and Hotelling shaped PCA as statistical practice. The meeting point is modern data matrices: one factorization, many missions (compression, denoising, regression, visualization).

JPEG-like low-rank thinking, eigenfaces-style image bases, climate modes from spatiotemporal grids, and "how many sensors do we actually need?" reviews all lean on SVD/PCA ideas.

A spreadsheet with twenty correlated columns often has only a few dominant singular directions. The rest may be noise or rare structure - decide with domain knowledge, not only scree plots.