Pearson correlation for all rows in Data Frames Pandas
<p>I have a dataframe in Pandas and its shape is (136, 1445). I try to create correlation(Pearson) matrix for my 136 rows. So in the result, i need  a matrix with size 136x136.</p>

<p>I tried two different ways but i cannot get a results from them or when i  create a 136x136 correlation matrix, i lost the columns name of dataframe. </p>

<p>First, </p>

<pre><code>gene_expression = pd.read_csv('padel_all_drug_results_original.csv',dtype='unicode')
gene_expression = gene_expression.convert_objects(convert_numeric=True)
gene_expression.corr()
</code></pre>

<p>This gives the column based pearson correlation matrix(1445*1445), and when I try to transpose my dataframe and then try to find correlation, the structure of dataframe is broken (like the columns name is lost or i dont even sure that the correlations are correct or not).</p>

<p>Secondly, </p>

<pre><code>distance = lambda column1, column2: pearsonr(column1,column2)[0]
result = gene_expression.apply(lambda col1: gene_expression.apply(lambda col2: distance(col1, col2)))
</code></pre>

<p>What should i do to calculate 136x136 pearson correlation matrix to do not change the original dataframe ?</p>

<p>Also, I have a 1445 features and some of columns nearly full of zeros. So I dropped that columns because they are noisy columns but have you got another idea to feature redeuction ? </p>

<p>Thanks in advance</p>