Yet Another Algorithm for Supervised Principal Component Analysis:
  Supervised Linear Centroid-Encoder
Tomojit Ghosh Michael Kirby
We propose a new supervised dimensionality reduction technique called
Supervised Linear Centroid-Encoder (SLCE), a linear counterpart of the
nonlinear Centroid-Encoder (CE) \citep{ghosh2022supervised}. SLCE works by
mapping the samples of a class to its class centroid using a linear
transformation. The transformation is a projection that reconstructs a point
such that its distance from the corresponding class centroid, i.e.,
centroid-reconstruction loss, is minimized in the ambient space. We derive a
closed-form solution using an eigendecomposition of a symmetric matrix. We did
a detailed analysis and presented some crucial mathematical properties of the
proposed approach. %We also provide an iterative solution approach based
solving the optimization problem using a descent method. We establish a
connection between the eigenvalues and the centroid-reconstruction loss. In
contrast to Principal Component Analysis (PCA) which reconstructs a sample in
the ambient space, the transformation of SLCE uses the instances of a class to
rebuild the corresponding class centroid. Therefore the proposed method can be
considered a form of supervised PCA. Experimental results show the performance
advantage of SLCE over other supervised methods.