<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Treat the genres of the movies
Attribute a new number to each new genre and replace in the dataframe
Step1:  1. Parsing example
Step2:  2. Determine different genres
Determine the number of different genres and what they are.
Step3:  3. Create vector of genres for each movie and a dataframe
Binary vector where the elements are 1 if the film has the genre corresponding to the index of the film. Otherwise the elements are zero.
Quick example
Step4:  Observe the result in the dataframe
Step5:  VIsual example of the genres
Step6:  3.1 Determine most frequent genres
Step7:  Display of the number of times a genre appears in the dataframe
Step8:  3.2 Determine genres that are most commonly associated with each other
We only consider two genres together at the moment
Step9:  Determining ranking of genre associations
1 indicates most frequently associated and 19 is least frequently associated. 
$\textbf{Reminder
Step10:  Display of the ranking of the other genres with which each genre is most often associated
Step11:  3.3 Determine how many films are successful or are non-successful depending on the genre
Step12:  Don't forget that the number of successful movies is not equal to the sum of the success rate of the genres since movies often have multiple genres.
Step13:  4. Create a similarity  graph between films depending on genre
In this case, the similarity is the number of genres that the movie has in common
Step14:  4.1 Normalization of the matrix
Step15:  Maximum normalization
Step16:  Plot the degree distribution
Step17:  Gaussian normalization
Step18:  4.3 Save the dataset
Step19:  5. Graph Laplacian and Embedding for maximum normalization
5.1 Compute the graph Laplacian
With pygsp
Step20:  Normally
Step21:  5.2 Compute the Fourier basis
With pygsp
Step22:  Normally
Step23:  5.3 Graph embedding
Step24:  6. Graph Laplacian and Embedding for gaussian normalization
6.1. Sparsification of the graph
Keep only a certain number of the weights
Step25:  6.1. Save the sparsed dataset
Step26:  6.2. Laplacian and graph embedding
With pygsp
Step27:  Other
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import configparser
import os
import requests
from tqdm import tqdm
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy import sparse, stats, spatial
import scipy.sparse.linalg
from sklearn import preprocessing, decomposition
import librosa
import IPython.display as ipd
import json
from imdb import IMDb
import tmdbsimple as tmdb
from pygsp import graphs, filters, plotting
plt.rcParams['figure.figsize'] = (17, 5)
plotting.BACKEND = 'matplotlib'
df = pd.read_csv('Saved_Datasets/NewFeaturesDataset.csv')
print('There are {} movies'.format(len(df)))
df['genres'][1]
df.head()
#df.iloc[100:150]
Explanation: Treat the genres of the movies
Attribute a new number to each new genre and replace in the dataframe
End of explanation
df['genres'] = df['genres'].str.replace('|', ',') 
i = 1
newgenres = df['genres'][i].split(",")
print(newgenres)
print(len(newgenres))
Explanation: 1. Parsing example
End of explanation
Diffgenres = [];
genres = {}
movies_dic = {}
for i in range(0, len(df)):
    
    movies_dic[i] = df['id'][i]
    
    if df['genres'][i] == 'NaN':
        newgenres = []
    else:
        newgenres = df['genres'][i].split(",")
        
        genres.setdefault(i, [])
        
        for j in range (0, len(newgenres)):
            Diffgenres.append(newgenres[j])
            genres[i].append(newgenres[j])
Diffgenres = set(Diffgenres)
Diffgenres = list(Diffgenres)
print('There are {} different genres'.format(len(Diffgenres)))
print(Diffgenres)
df.head()
Explanation: 2. Determine different genres
Determine the number of different genres and what they are.
End of explanation
print(genres[0][0])
len(genres[0][0])
vector = (genres[0][0] == np.array(Diffgenres)).astype(int)
print(vector)
genreArray = np.ndarray(shape=(len(df), len(Diffgenres)), dtype=int)
for i in range(0, len(df)):
    vector = np.zeros(len(Diffgenres))
    
    for j in range(0, len(genres[i])):
        vector += (genres[i][j] == np.array(Diffgenres)).astype(int)
    
    genreArray[i] = vector
print(genreArray[0])
print(genreArray.size)
Explanation: 3. Create vector of genres for each movie and a dataframe
Binary vector where the elements are 1 if the film has the genre corresponding to the index of the film. Otherwise the elements are zero.
Quick example:
End of explanation
Genres = pd.DataFrame(genreArray, columns=Diffgenres)
Genres.head(10)
Explanation: Observe the result in the dataframe
End of explanation
#Genres.iloc[120:150]
plt.spy(Genres[120:150])
Explanation: VIsual example of the genres
End of explanation
freqGenre = np.ndarray(shape=(1, len(Diffgenres)), dtype=int)
for i in range(0, len(Diffgenres)):
    freqGenre[0][i] = sum(Genres[Diffgenres[i]] == 1)
Explanation: 3.1 Determine most frequent genres
End of explanation
NbGenre = pd.DataFrame(freqGenre, columns=Diffgenres)
NbGenre
NbGenre.to_csv('Saved_Datasets/NbGenre.csv', index=False)
plt.bar(Diffgenres, freqGenre[0], align='center');
plt.setp(plt.gca().get_xticklabels(), rotation=45, horizontalalignment='right');
plt.xlabel('Genres');
plt.ylabel('Counts');
plt.savefig('images/GenreFreq.png', dpi =300, bbox_inches='tight')
Explanation: Display of the number of times a genre appears in the dataframe
End of explanation
assosGenre = np.ndarray(shape=(len(Diffgenres), len(Diffgenres)), dtype=int)
for i in range(0, len(Diffgenres)):
    for j in range(0, len(Diffgenres)):
        if i != j:
            assosGenre[i][j] = sum((Genres[Diffgenres[i]] == 1) & (Genres[Diffgenres[j]] == 1))
        else:
            assosGenre[i][j] = 0
#ensure the matrix is symmetric
assosGenreSym = assosGenre.transpose() > assosGenre
assosGenre = assosGenre - assosGenre*assosGenreSym + assosGenre.transpose()*assosGenreSym
plt.spy(assosGenre)
NbGenreAssos = pd.DataFrame(assosGenre, columns=Diffgenres, index = Diffgenres)
NbGenreAssos
NbGenreAssos.to_csv('Saved_Datasets/NbGenreAssos.csv', index=False)
Explanation: 3.2 Determine genres that are most commonly associated with each other
We only consider two genres together at the moment
End of explanation
assosRank = {} 
rank = np.argsort(-assosGenre, axis=1) #negative for ascending order
Diffgenres[rank[0][1]]
for i in range(0, len(Diffgenres)):
    for j in range(0, len(Diffgenres)):
        assosRank.setdefault(Diffgenres[i], [])
        
        #Only if not comparing with the same genre
        if Diffgenres[i] != Diffgenres[rank[i][j]]:
            assosRank[Diffgenres[i]].append(Diffgenres[rank[i][j]])
Explanation: Determining ranking of genre associations
1 indicates most frequently associated and 19 is least frequently associated. 
$\textbf{Reminder:}$ This is only the case of our dataset and may not represent reality
End of explanation
ranking = np.linspace(1, len(Diffgenres)-1, num=len(Diffgenres)-1, endpoint=True, retstep=False, dtype=int)
Rankdf = pd.DataFrame(assosRank, index=ranking)
Rankdf
Rankdf.to_csv('Saved_Datasets/GenreRanking.csv', index=False)
Explanation: Display of the ranking of the other genres with which each genre is most often associated
End of explanation
genreArray[0]
genreSuccess = np.zeros(shape=(1, len(Diffgenres)), dtype=float)
genreSuccessPc = np.zeros(shape=(1, len(Diffgenres)), dtype=float)
for i in range(0, len(Diffgenres)):
    for j in range(0, len(df)):
        if genreArray[j][i] == 1:
            if df['success'][j] == 1:
                genreSuccess[0][i] += 1
    
    genreSuccessPc[0][i] = (genreSuccess[0][i]/freqGenre[0][i])*100
plt.bar(Diffgenres, genreSuccessPc[0], align='center');
plt.setp(plt.gca().get_xticklabels(), rotation=45, horizontalalignment='right');
plt.xlabel('Genres');
plt.ylabel('Success rate [%]');
Explanation: 3.3 Determine how many films are successful or are non-successful depending on the genre
End of explanation
print(sum(sum(genreSuccess)))
print('The number of films that are succesful: {}'.format(len(df[df['success'] == 1])))
print('The number of films that are unsuccesful: {}'.format(len(df[df['success'] == 0])))
Explanation: Don't forget that the number of successful movies is not equal to the sum of the success rate of the genres since movies often have multiple genres.
End of explanation
weights = np.ndarray(shape=(len(df), len(df)), dtype=int)
weights = genreArray @ genreArray.T
#fill the diagonal values to zero, i.e. no self-connections
np.fill_diagonal(weights, 0)
plt.spy(weights)
plt.hist(weights[weights > 0].reshape(-1), bins=50);
print('There are {} weights equal to zero'.format(np.sum(weights == 0)))
print('There are {} weights equal to one'.format(np.sum(weights == 1)))
print('There are {} weights equal to seven'.format(np.sum(weights == 7)))
meanW = weights.mean()
maxW = weights.max()
minW = weights.min()
print('The mean value of the similarity in terms of genre is: {}'.format(meanW))
print('The max value of the similarity is: {}'.format(maxW))
print('The min value of the similarity is: {}'.format(minW))
Explanation: 4. Create a similarity  graph between films depending on genre
In this case, the similarity is the number of genres that the movie has in common:
$\mathbf{W}(u,v) = sum(u \cdot v) \ \in [0, 20]$
End of explanation
print(genreArray[1])
print(sum(genreArray[1]))
weightsNorm = np.ndarray(shape=(len(df), len(df)), dtype=float)
lengths =  np.ndarray(shape=(1, 2), dtype=int)
lenMax = 0;
for i in range(0, len(weights)):
    for j in range(0, len(weights)):
        if i!=j: 
            lengths = [sum(genreArray[i]), sum(genreArray[j])]
            weightsNorm[i][j] = (weights[i][j])/max(lengths) 
np.fill_diagonal(weightsNorm, 0)
sigma = np.std(weights)
print(sigma)
mu = np.mean(weights)
print(mu)
#1/(sigma*math.sqrt(2*math.pi))*
Wgauss = np.exp(-((weights-mu)**2)/(2*sigma**2))
#fill the diagonal values to zero, i.e. no self-connections
np.fill_diagonal(Wgauss, 0)
Explanation: 4.1 Normalization of the matrix
End of explanation
plt.spy(weightsNorm)
plt.hist(weightsNorm.reshape(-1), bins=50);
print('The mean value is: {}'.format(weightsNorm.mean()))
print('The max value is: {}'.format(weightsNorm.max()))
print('The min value is: {}'.format(weightsNorm.min()))
Explanation: Maximum normalization
End of explanation
degrees = np.zeros(len(weightsNorm)) 
#reminder: the degrees of a node for a weighted graph are the sum of its weights
for i in range(0, len(weightsNorm)):
    degrees[i] = sum(weightsNorm[i])
plt.hist(degrees, bins=50);
print('The mean value is: {}'.format(degrees.mean()))
print('The max value is: {}'.format(degrees.max()))
print('The min value is: {}'.format(degrees.min()))
Explanation: Plot the degree distribution
End of explanation
plt.spy(Wgauss)
Explanation: Gaussian normalization
End of explanation
NormW = pd.DataFrame(weightsNorm)
NormW.head()
NormW.to_csv('Saved_Datasets/NormalizedGenreW.csv', index=False)
Explanation: 4.3 Save the dataset
End of explanation
G = graphs.Graph(weightsNorm)
G.compute_laplacian('normalized')
Explanation: 5. Graph Laplacian and Embedding for maximum normalization
5.1 Compute the graph Laplacian
With pygsp
End of explanation
#reminder: L = D - W for weighted graphs
laplacian = np.diag(degrees) - weightsNorm
#computation of the normalized Laplacian
laplacian_norm = scipy.sparse.csgraph.laplacian(weightsNorm, normed = True)
plt.spy(laplacian_norm);
laplacian_norm = sparse.csr_matrix(laplacian_norm)
Explanation: Normally
End of explanation
G.compute_fourier_basis(recompute=True)
plt.plot(G.e[0:10]);
Explanation: 5.2 Compute the Fourier basis
With pygsp
End of explanation
eigenvalues, eigenvectors =  sparse.linalg.eigsh(laplacian_norm, k = 10, which = 'SM') 
plt.plot(eigenvalues, '.-', markersize=15);
plt.xlabel('')
plt.ylabel('Eigenvalues')
plt.show()
Explanation: Normally
End of explanation
genres = preprocessing.LabelEncoder().fit_transform(df['success'])
x = eigenvectors[:, 1] 
y = eigenvectors[:, 2] 
plt.scatter(x, y, c=genres, cmap='RdBu', alpha=0.5);
G.set_coordinates(G.U[:, 1:3])
G.plot()
G.plot_signal(genres, vertex_size=20)
Explanation: 5.3 Graph embedding
End of explanation
NEIGHBORS = 300
#sort the order of the weights
sort_order = np.argsort(Wgauss, axis = 1)
#declaration of a sorted weight matrix
sorted_weights = np.zeros((len(Wgauss), len(Wgauss)))
for i in range (0, len(Wgauss)):  
    for j in range(0, len(Wgauss)):
        if (j >= len(Wgauss) - NEIGHBORS):
            #copy the k strongest edges for each node
            sorted_weights[i, sort_order[i,j]] = Wgauss[i,sort_order[i,j]]
        else:
            #set the other edges to zero
            sorted_weights[i, sort_order[i,j]] = 0
#ensure the matrix is symmetric
bigger = sorted_weights.transpose() > sorted_weights
sorted_weights = sorted_weights - sorted_weights*bigger + sorted_weights.transpose()*bigger
plt.spy(sorted_weights)
plt.hist(sorted_weights.reshape(-1), bins=50);
Explanation: 6. Graph Laplacian and Embedding for gaussian normalization
6.1. Sparsification of the graph
Keep only a certain number of the weights
End of explanation
NormW = pd.DataFrame(sorted_weights)
NormW.head()
NormW.to_csv('Saved_Datasets/NormalizedGenreWSparse.csv', index=False)
Explanation: 6.1. Save the sparsed dataset
End of explanation
G = graphs.Graph(sorted_weights)
G.compute_laplacian('normalized')
Explanation: 6.2. Laplacian and graph embedding
With pygsp
End of explanation
#reminder: L = D - W for weighted graphs
laplacian = np.diag(degrees) - sorted_weights
#computation of the normalized Laplacian
laplacian_norm = scipy.sparse.csgraph.laplacian(sorted_weights, normed = True)
plt.spy(laplacian_norm);
G.compute_fourier_basis(recompute=True)
plt.plot(G.e[0:10]);
G.set_coordinates(G.U[:, 1:3])
G.plot()
G.plot_signal(genres, vertex_size=20)
Explanation: Other
End of explanation
<END_TASK>