Pearson correlation between two dataframes in python and output all combinations between columns
<p>Hello All I have two data frames with 3 columns and each of the rows from col1 has 82 different values from column variable and so as I have df2 with three same columns with different values for these three columns </p>

<p>df1.head()</p>

<pre><code>    ID  variable value
0   ENSG3   AE02_ID 0.000001
1   ENSG1   AE02_ID 0.329180
2   ENSG8   AE02_ID 0.000000
3   ENSG10  AE02_ID 29.157761
4   ENSG20  AE02_ID 0.633884
</code></pre>

<p>and <code>df2.head()</code></p>

<pre><code>    Geneid  variable    value
0   ENSG60  AE02_ID 7.408430
1   ENSG53  AE02_ID 0.000000
2   ENSG94  AE02_ID 2.556464
3   ENSG49  AE02_ID 0.032384
4   ENSG9   AE02_ID 0.000000
</code></pre>

<p>I wanted to do a Pearson correlation on these two data frames, the output data frame should be with correlation coefficient from all possible combinations from both data frames. for instance something like this</p>

<pre><code>ID1    ID2   coefficient
ENSG60 ENSG3 0.2
ENSG1 ENSG53  0.14
.
.
</code></pre>

<p>I tried with this one liner      <code>df1.value.corrwith(df2.value)</code></p>

<p>But it's giving out only one coefficient with the corresponding </p>