Kernel PCA

What is Kernel PCA?

Kernel Principal Component Analysis (Kernel PCA) is an extension of the standard Principal Component Analysis (PCA) that allows for nonlinear dimensionality reduction. It employs the kernel trick to map data into a higher-dimensional space, making linear separation possible.

What Problem Does Kernel PCA Solve?

Traditional Principal Component Analysis (PCA) is a linear dimensionality reduction technique, meaning it assumes that the data can be separated using straight-line principal components. However, many real-world datasets have complex nonlinear structures that cannot be effectively captured using standard PCA.

Kernel Principal Component Analysis (Kernel PCA) extends standard Principal Component Analysis (PCA) by enabling nonlinear dimensionality reduction.

How Does Kernel PCA Work?

Let’s consider a dataset with two input features and one output variable, containing 400 data points arranged in concentric circles. Our goal is to reduce this dataset to one dimension while preserving its structure.

Step 1: Centering the Data

Before applying Kernel PCA, we mean-center the data by subtracting the mean of each feature. This ensures that computations are performed relative to the data's centroid.

Step 2: Applying the Kernel Transformation

Instead of working directly in the original feature space, we apply a kernel function to transform the data into a higher-dimensional space. One commonly used kernel is the Gaussian (RBF) Kernel :

This kernel transformation produces a 400 × 400 kernel matrix, where each entry represents the similarity between two data points :

• Smaller Euclidean distances → Higher similarity (larger kernel values)

• Larger Euclidean distances → Lower similarity (smaller kernel values)

Thus, Kernel PCA implicitly computes distances in a higher-dimensional space without explicitly performing the transformation, which is computationally efficient .

Step 3: Constructing the Kernel Matrix

• The kernel function is applied pairwise to every point in the dataset, forming a symmetric 400 × 400 matrix.

• Each row represents how a single data point relates to every other point in the dataset.

Step 4: Eigen Decomposition

• Since the kernel matrix is symmetric, it has 400 eigenvalues and 400 corresponding eigenvectors, which form an orthogonal basis.

• These eigenvectors correspond to the principal components in the transformed space.

• Selecting the top k eigenvectors helps retain the most significant principal components, thereby reducing dimensionality .

Step 5: Dimensionality Reduction

• If we want to reduce our dataset to one dimension, we select the first principal component and project the data onto it .

• This transformed representation preserves the essential structure of the data while minimizing irrelevant variations .