ACADEMIC PAPER · LINEAR ALGEBRA / DATA ANALYSIS
A mathematical and practical examination of Principal Component Analysis: how variance maximisation turns covariance into dimensionality reduction, why that works so well on some datasets, and why it can confidently preserve the wrong information on others.
01 · THE CORE IDEA
High-dimensional datasets often contain features that tell nearly the same story. PCA replaces those correlated coordinates with a smaller set of orthogonal directions — the principal components — while retaining as much variance as possible.
After centring the data matrix X, its covariance matrix records how the features vary together. The first component is the unit direction v that maximises the variance of the projection:
Applying Lagrange multipliers turns that optimisation into the eigenvalue equation Σv = λv. The eigenvectors of the covariance matrix are therefore the principal directions, ordered by their eigenvalues: the amount of variance each direction captures. This is the bridge from linear algebra to data analysis at the centre of the paper.
02 · THE ROUTE
Subtract each feature mean so that PCA describes the structure around the origin rather than the absolute position of the cloud.
Σ = XTX / n summarises the variance of each feature and the linear relationships between feature pairs.
Successive eigenvectors maximise the remaining projected variance while staying orthogonal to the components already chosen.
Explained variance ratios quantify retained variance; a scree plot or elbow offers a visual alternative. In practice, SVD computes the same decomposition more stably than explicitly forming XTX.
03 · WHERE IT FAILS
PCA always returns components. That does not mean those components are meaningful. The paper isolates two different ways its central assumption can fail.
VARIANCE ≠ VALUE
In a synthetic tumour-classification example, cell size accounts for roughly 99.5% of the variance but does not separate benign from malignant cases. A low-variance biomarker does. PCA optimises variance without seeing the labels, so an excellent explained-variance score can still preserve the wrong information for classification.
NON-LINEAR STRUCTURE
The Swiss Roll is locally two-dimensional but curved through three-dimensional space. PCA measures straight-line Euclidean structure and searches only linear subspaces. Its projection therefore mixes points that are far apart along the surface and cannot preserve the roll's geodesic geometry.
04 · WHERE IT WORKS
The counterexample is the Wisconsin Breast Cancer Dataset: 569 observations with 30 numerical cell-nucleus features. Many measurements are strongly correlated — 70 of the 435 feature pairs have |ρ| > 0.7, and 21 exceed |ρ| > 0.9 — so much of the dataset is genuinely redundant.
Here the high-variance directions also carry biologically useful separation. The first two components retain about 63.2% of the variance and already separate the benign and malignant groups reasonably well; ten components pass a 95% cumulative explained-variance threshold. PCA turns a correlated 30-dimensional space into a compact, workable representation without erasing the structure of interest.
05 · TAKE-AWAY
PCA is not weak; it is specific. It is powerful when high variance tracks useful information, the relevant structure is approximately linear, and correlated features make compression possible. It becomes misleading when variance and task relevance diverge or when the data lives on a curved manifold.
The practical question is therefore not simply “How much variance did PCA preserve?” but “Was variance the information I needed to preserve?” Recognising when that assumption holds is what turns PCA from an automatic preprocessing step into a well-judged analytical tool.