Dimensionality Reduction with PCA

Why Reduce Dimensions?

If you have 100 features, visualizing them is impossible. Worse, models train slower and can overfit on irrelevant features.

Principal Component Analysis (PCA) finds new axes for your data that capture the maximum amount of variance (spread).

• PC1: The axis where data is most spread out.
• PC2: The axis perpendicular to PC1 with the next most spread.

The Math (It's just Linear Algebra!)

1. Center the data (subtract the mean of every column).
2. Calculate the Covariance Matrix.
3. Calculate the Eigenvectors and Eigenvalues of that matrix.
4. The Eigenvectors are your new axes (Principal Components).

Note: PCA does NOT delete columns. It rotates the coordinate system so the first few columns contain 90% of the information.