How to use the reduced data - the output of principal component analysis
<p>I am  finding it hard to link the theory with the implementation. I would appreciate help in knowing where my understanding is wrong.</p>

<p>Notations - matrix in bold capital and vectors in bold font small letter</p>

<p><sub><a href="https://i.stack.imgur.com/MEkHe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MEkHe.png" alt="**X**"></a></sub> is a dataset on <a href="https://i.stack.imgur.com/IAyup.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IAyup.png" alt="&lt;code&gt;n&lt;/code&gt;"></a> observations, each of <a href="https://i.stack.imgur.com/to8Zc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/to8Zc.png" alt="&lt;code&gt;d&lt;/code&gt;"></a> variables. So, given these observed <a href="https://i.stack.imgur.com/ionSC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ionSC.png" alt="&lt;code&gt;d&lt;/code&gt;"></a>-dimensional data vectors, the <a href="https://i.stack.imgur.com/Aeqge.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Aeqge.png" alt="&lt;code&gt;m&lt;/code&gt;"></a>-dimensional principal axes are <sub><sub><sub><a href="https://i.stack.imgur.com/GWp8m.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GWp8m.png" alt="**w**&lt;sub&gt;j&lt;/sub&gt;"></a></sub></sub></sub>, for <sub><sub><a href="https://i.stack.imgur.com/NJln1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/NJln1.png" alt="j"></a></sub></sub> in <sub><sub><sub><a href="https://i.stack.imgur.com/Uvo6K.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Uvo6K.png" alt="{1,...,m}"></a></sub></sub></sub> where <a href="https://i.stack.imgur.com/4Vr0E.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4Vr0E.png" alt="&lt;code&gt;m&lt;/code&gt;"></a> is the target dimension.</p>

<p>The <a href="https://i.stack.imgur.com/BsGgR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BsGgR.png" alt="&lt;code&gt;m&lt;/code&gt;"></a> principal components of the observed data matrix would be <sub><a href="https://i.stack.imgur.com/GkE4r.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GkE4r.png" alt="**Y = X W**"></a></sub> where matrix <sub><a href="https://i.stack.imgur.com/v58SP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/v58SP.png" alt="**Y** in ℝ&lt;sup&gt; n x m&lt;/sup&gt;"></a></sub>, matrix <sub><a href="https://i.stack.imgur.com/OCemy.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OCemy.png" alt="**X** in ℝ&lt;sup&gt; n x d&lt;/sup&gt;"></a></sub>, and matrix <sub><a href="https://i.stack.imgur.com/n4aDJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/n4aDJ.png" alt="**W** in ℝ&lt;sup&gt; d x m&lt;/sup&gt;"></a></sub>. </p>

<p>Columns of <a href="https://i.stack.imgur.com/8uUOJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8uUOJ.png" alt="**W**"></a> form an orthogonal basis for the <a href="https://i.stack.imgur.com/2g0Y3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/2g0Y3.png" alt="&lt;code&gt;m&lt;/code&gt;"></a> features and the output <sub><sub><a href="https://i.stack.imgur.com/XlMKP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XlMKP.png" alt="**y**&lt;sub&gt;n&lt;/sub&gt;"></a></sub></sub> is the principal component projection that minimizes the squared reconstruction error:</p>

<p><a href="https://i.stack.imgur.com/pekNZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pekNZ.png" alt="∑&lt;sub&gt;n&lt;/sub&gt; ||**x**&lt;sub&gt;n&lt;/sub&gt; - **x&#770;**&lt;sub&gt;n&lt;/sub&gt;||&lt;sup&gt;2&lt;/sup&gt;"></a><br>
and the optimal reconstruction of <sub><a href="https://i.stack.imgur.com/ZP9LS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZP9LS.png" alt="**x**&lt;sub&gt;n&lt;/sub&gt;"></a></sub> is given by <sub><a href="https://i.stack.imgur.com/O8HqU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/O8HqU.png" alt="**x&#770;**&lt;sub&gt;n&lt;/sub&gt; = **W*y**&lt;sub&gt;n&lt;/sub&gt;"></a></sub>.</p>

<p>The data model is </p>

<pre><code>X(i,j) = A(i,:)*S(:,j) + noise
</code></pre>

<p>where PCA should be done on X to get the output S. S must be equal to Y.</p>

<p>Problem 1: The reduced data Y is not equal to S that is used in the model. Where is my understanding wrong?</p>

<p>Problem 2: How to reconstruct such that the error is minimum?</p>

<p>Please help. Thank you.</p>

<pre><code>   clear all
clc
n1 = 5; %d dimension
n2 = 500; % number of examples

ncomp = 2; % target reduced dimension
%Generating data according to the model
% X(i,j) = A(i,:)*S(:,j) + noise
Ar = orth(randn(n1,ncomp))*diag(ncomp:-1:1);
T = 1:n2;
%generating synthetic data from a dynamical model
S = [ exp(-T/150).*cos( 2*pi*T/50 )
       exp(-T/150).*sin( 2*pi*T/50 ) ];
% Normalizing to zero mean and unit variance
S = ( S - repmat( mean(S,2), 1, n2 ) );
S = S ./ repmat( sqrt( mean( Sr.^2, 2 ) ), 1, n2 );
Xr = Ar * S;
Xrnoise = Xr + 0.2 * randn(n1,n2);

h1 = tsplot(S);


    X = Xrnoise;

XX = X';
[pc, ~] = eigs(cov(XX), ncomp);
Y = XX*pc;
</code></pre>

<p>UPDATE [10 Aug]</p>

<p>Based on the Answer, here is the full code that </p>

<pre><code> clear all
clc
n1 = 5; %d dimension
n2 = 500; % number of examples

ncomp = 2; % target reduced dimension
%Generating data according to the model
% X(i,j) = A(i,:)*S(:,j) + noise
Ar = orth(randn(n1,ncomp))*diag(ncomp:-1:1);
T = 1:n2;
%generating synthetic data from a dynamical model
S = [ exp(-T/150).*cos( 2*pi*T/50 )
       exp(-T/150).*sin( 2*pi*T/50 ) ];
% Normalizing to zero mean and unit variance
S = ( S - repmat( mean(S,2), 1, n2 ) );
S = S ./ repmat( sqrt( mean( S.^2, 2 ) ), 1, n2 );
Xr = Ar * S;
Xrnoise = Xr + 0.2 * randn(n1,n2);

    X = Xrnoise;

XX = X';
[pc, ~] = eigs(cov(XX), ncomp);
Y = XX*pc; %Y are the principal components of X' 
%what you call pc is misleading, these are not the principal components
%These Y columns are orthogonal, and should span the same space 
%as S approximatively indeed (not exactly, since you introduced noise).

%If you want to reconstruct 
%the original data can be retrieved by projecting 
%the principal components back on the original space like this:
Xrnoise_reconstructed = Y*pc';

%Then, you still need to project it through 
%to the S space, if you want to reconstruct S
S_reconstruct = Ar'*Xrnoise_reconstructed';


plot(1:length(S_reconstruct),S_reconstruct,'r')
hold on
 plot(1:length(S),S)
</code></pre>

<p>The plot is  <a href="https://i.stack.imgur.com/LAxS0.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LAxS0.jpg" alt="plot"></a> which is very different from the one that is shown in the Answer. Only one component of S exactly matches with that of S_reconstructed. Shouldn't the entire original 2 dimensional space of the source input S be reconstructed?
Even if I cut off the noise, then also onely one component of S is exactly reconstructed.</p>