Find Similarity between two DataFrames of different lengths
<p>I have two Pandas Dataframes, both of varying length.
DF1 has about 1.2 millions row (and just 1 column), DF2 has about 300,000 rows (and a single column), and I am trying to find similar items from both lists.</p>

<p>DF1 has about 75% Company Names, and 25% People, and the reverse is true for DF2, but they are both alphanumeric. What I would like is to write a function that will highlight the most similar items from the two lists, ranked by a score (or percentage).
For example,</p>

<pre><code>Apple -&gt; Apple Inc. (0.95) 
Apple -&gt; Applebees (0.68)
Banana Boat -&gt; Banana Bread (0.25)
</code></pre>

<p>So far, I have tried two approaches, both of which have failed.</p>

<p><strong>Method 1</strong>: Find Jaccard Coefficients for the two lists.</p>

<pre><code>import numpy as np
from sklearn.metrics import jaccard_similarity_score
jaccard_similarity_score(df_1, df_2)
</code></pre>

<p>This does not work, probably due to the varying lengths of the two data frames and I get this error:</p>

<blockquote>
  <p>ValueError: Found arrays with inconsistent numbers of samples</p>
</blockquote>

<p><strong>Method 2:</strong>: Using Sequence Matcher</p>

<pre><code>from difflib import SequenceMatcher
def similar(a, b):
    return SequenceMatcher(None, a, b).ratio()
</code></pre>

<p>And then calling the Dataframes:</p>

<pre><code>similar(df_1, df_2)
</code></pre>

<p>This results in an error:</p>

<blockquote>
  <p>pandas/index.pyx in pandas.index.IndexEngine.get_loc
  (pandas/index.c:3979)()</p>
  
  <p>pandas/index.pyx in pandas.index.IndexEngine.get_loc
  (pandas/index.c:3843)()</p>
  
  <p>pandas/hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item
  (pandas/hashtable.c:12265)()</p>
  
  <p>pandas/hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item
  (pandas/hashtable.c:12216)()</p>
  
  <p>KeyError: 0</p>
</blockquote>

<p>How could I approach this problem?</p>