How do I correctly use PCA followed by Logistic Regression?
<p>In the program, I am scanning a number of brain samples taken in a time series of 40 x 64 x 64 images every 2.5 seconds. The number of 'voxels' (3D pixels) in each image is thus ~ 168,000 ish (40 * 64 * 64), each of which is a 'feature' for an image sample. </p>

<p>I thought of using Principle Component Analysis (PCA) because of the rediculously high <code>n</code> to perform dimensionality reduction. I am aware PCA does not actually reduce the number of features. Correct me if I'm wrong, but PCA will produce a new set of features  from the original ones. The new features however must follow certain conditions.</p>

<p>I definined a method to get the number of components :</p>

<pre><code>def get_optimal_number_of_components():
    cov = np.dot(X,X.transpose())/float(X.shape[0])
    U,s,v = svd(cov)

    S_nn = sum(s)

    for num_components in range(0,s.shape[0]):
        temp_s = s[0:num_components]
        S_ii = sum(temp_s)
        if (1 - S_ii/float(S_nn)) &lt;= 0.01:
            return num_components

    return s.shape[0]
</code></pre>

<p>This function will return the number of components such that 99% of the variance from the original data is retained. Now, we can create these components :</p>

<pre><code>#Scaling the values
X = scale(X)

n_comp = get_optimal_number_of_components()
print 'optimal number of components = ', n_comp

pca = PCA(n_components = n_comp)
X_new = pca.fit_transform(X)
</code></pre>

<p>I get the optimal number of components = 1001 on running the program on this dataset. This number agrees with the plot I obtained on executing :</p>

<pre><code>#Cumulative Variance explains
var1 = np.cumsum(np.round(pca.explained_variance_ratio_, decimals=4)*100)

plt.plot(var1)
plt.title('Principle Component Analysis for Feature Selection')
plt.ylabel('Percentage of variance')
plt.xlabel('Number of voxels considered')

plt.show()
</code></pre>

<p>After this PCA stage is complete, I used the newly created 'X_new' in place of X for the next stage : Logistic Regression</p>

<pre><code>#After PCA
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_new, y, test_size=0.10, random_state=42)

classifier = LogisticRegression()
classifier.fit(X_train,y_train)
</code></pre>

<p>When I test for the accuracy, I get around <strong>77.57%</strong></p>

<p>But this is less than when I just analysed the middle voxel samples (like 9K samples in the middle of brain image). I was wondering if I pipelined PCA and logistic regression correctly.</p>

<p>I even tried this in another method using <code>sklearn.pipeline</code> :</p>

<pre><code>pipe = Pipeline(steps=[('pca', pca), ('logistic', classifier)])

pipe.set_params(pca__n_components = n_comp).fit(X_train,y_train)

print 'score = ', pipe.score(X_test,y_test)
</code></pre>

<p>But I got the exact same accuracy of 77.57%. Am I implementing PCA + Logistic Regression Correctly? There must be something wrong, I just can't figure out what it is. </p>