<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Testing string match functions
Note
Step1:  Cosine Similarity
Step2:  String comparison using cosine similarity https
Step3:  Cosine works fine with whole words and word transposition but will start to trip up on CO vs COMPANY and when too much extraneous text is introduced.
DIFFLIB (Python Module)
String comparison using difflib - https
Step4:  May have some issues where it comes to partial string matches http
Step5:  Also wants to use python-Levenshtein to improve speed, but install failed on gcc - will complain below
Step6:  Fuzzywuzzy has an interesting "process" function
Step7:  taken from https
Step8:  Note that soundex and nysiis both appear to just take the first word
<ASSISTANT_TASK:>
Python Code:
# Note - these lines added to make it work with shared Jupyter Hub instance, 
# modifying the system path so that locally installed modules installed with the shell commands below will be found - 
# they would need to be modified for your instance, or to install the modules normally remove the --user param
# import sys
# import os
# sys.path.append(os.path.abspath("/...path to your local module install dir..."))
# these are the values we want to test
text1 = 'General Electric Company'
text2 = 'General Electric Co Inc'
Explanation: Testing string match functions
Note:  These examples were built with an Anaconda distro, on a Python 3.x kernel.  External modules were installed locally using --user option (since these were being run on a shared Jupyter Hub instance)
End of explanation
import re, math
from collections import Counter
Explanation: Cosine Similarity
End of explanation
WORD = re.compile(r'\w+')
def get_cosine(vec1, vec2):
     intersection = set(vec1.keys()) & set(vec2.keys())
     numerator = sum([vec1[x] * vec2[x] for x in intersection])
     sum1 = sum([vec1[x]**2 for x in vec1.keys()])
     sum2 = sum([vec2[x]**2 for x in vec2.keys()])
     denominator = math.sqrt(sum1) * math.sqrt(sum2)
     if not denominator:
        return 0.0
     else:
        return float(numerator) / denominator
def text_to_vector(text):
     words = WORD.findall(text)
     return Counter(words)
vector1 = text_to_vector(text1)
vector2 = text_to_vector(text2)
cosine = get_cosine(vector1, vector2)
print ('Cosine:', cosine)
Explanation: String comparison using cosine similarity https://en.wikipedia.org/wiki/Cosine_similarity
Code sample copypasta from Stack Overflow: http://stackoverflow.com/questions/15173225/how-to-calculate-cosine-similarity-given-2-sentence-strings-python
End of explanation
import difflib
from difflib import SequenceMatcher
m = SequenceMatcher(None, text1, text2)
print (m.ratio())
Explanation: Cosine works fine with whole words and word transposition but will start to trip up on CO vs COMPANY and when too much extraneous text is introduced.
DIFFLIB (Python Module)
String comparison using difflib - https://docs.python.org/3/library/difflib.html
End of explanation
## %%sh
## pip install fuzzywuzzy --user
Explanation: May have some issues where it comes to partial string matches http://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/
FuzzyWuzzy (Python Module)
Background on FuzzyWuzzy - http://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/
Package install should only need to be done once, unless the cluster was reset - this will install locally (using --user parameter), so variables need to be set
End of explanation
## %%sh
## pip install python-Levenshtein --user
from fuzzywuzzy import fuzz
from fuzzywuzzy import process
print (fuzz.ratio(text1, text2))
print (fuzz.partial_ratio(text1, text2))
print (fuzz.token_sort_ratio(text1, text2))
print (fuzz.token_set_ratio(text1, text2))
Explanation: Also wants to use python-Levenshtein to improve speed, but install failed on gcc - will complain below
End of explanation
import numpy as np
# Jaccard Similarity J (A,B) = | Intersection (A,B) | / | Union (A,B) |
def compute_jaccard_similarity_score(x, y):
    intersection_cardinality = len(set(x).intersection(set(y)))
    union_cardinality = len(set(x).union(set(y)))
    return intersection_cardinality / float(union_cardinality)
score = compute_jaccard_similarity_score(text1, text2)
print ("Jaccard Similarity Score: ",score)
Explanation: Fuzzywuzzy has an interesting "process" function:
choices = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]
process.extract("new york jets", choices, limit=2)
    [('New York Jets', 100), ('New York Giants', 78)]
process.extractOne("cowboys", choices)
    ("Dallas Cowboys", 90)
Jaccard
End of explanation
## %%sh
## pip install jellyfish --user
import jellyfish
jellyfish.levenshtein_distance(text1,text2)
jellyfish.damerau_levenshtein_distance(text1,text2)
jellyfish.jaro_distance(text1,text2)
jellyfish.jaro_winkler(text1,text2)
jellyfish.match_rating_comparison(text1,text2)
jellyfish.hamming_distance(text1,text2)
jellyfish.soundex(text1)
jellyfish.soundex(text2)
soundexenc = ''
sentence=text1.split()
for word in sentence:
    soundexenc = soundexenc+' '+jellyfish.soundex(word)
print(soundexenc)
jellyfish.metaphone(text1)
jellyfish.metaphone(text2)
jellyfish.metaphone(text1) == jellyfish.metaphone(text2)
jellyfish.nysiis(text1)
jellyfish.nysiis(text2)
Explanation: taken from https://codegists.com/code/python%20jaccard/
Jellyfish
Testing Jellyfish library, with the following algorithms
String comparison:
Levenshtein Distance
Damerau-Levenshtein Distance
Jaro Distance
Jaro-Winkler Distance
Match Rating Approach Comparison
Hamming Distance
Phonetic encoding:
American Soundex
Metaphone
NYSIIS (New York State Identification and Intelligence System)
Match Rating Codex
https://github.com/jamesturk/jellyfish
End of explanation
jellyfish.nysiis(text1) == jellyfish.nysiis(text2)
nysiisenc = ''
sentence=text2.split()
for word in sentence:
    nysiisenc = nysiisenc+' '+jellyfish.nysiis(word)
print(nysiisenc)
jellyfish.match_rating_codex(text1)
jellyfish.match_rating_codex(text2)
Explanation: Note that soundex and nysiis both appear to just take the first word
End of explanation
<END_TASK>