<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Nonnegative Matrix Factorization In The Movielens Dataset
This example continues illustrating using pandas-munging capabilities in estimators building features that draw from several rows, this time using NMF (nonnegative matrix factorization). We will use a single table from the Movielens dataset (F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets
Step1:  Munging NMF With Pandas
In Simple Row-Aggregating Features In The Movielens Dataset we looked at direct attributes obtainable from the rankings
Step2:  We now use NMF for decomposition, and then find the user latent factors in U and item latent factors in I
Step3:  Note that the Ibex version of NMF sets the indexes and columns of the U and I appropriately.
Step4:  Pandas makes it easy to merge the user and item latent factors to the users and items, respectively.
Step5:  Let's merge to the results also the number of occurrences to of the users and items, respectively.
Step6:  We now have a dataframe of latent variables. Let's build a random forest regressor, and use it on this dataframe.
Step7:  Finally, let's check the feature importances.
Step8:  Building A Pandas-Munging Estimator
We'll now build a Scikit-Learn / Pandas step doing the above.
Step9:  We can now use cross validation to assess this scheme.
<ASSISTANT_TASK:>
Python Code:
import os
from sklearn import base
import pandas as pd
import scipy as sp
import seaborn as sns
sns.set_style('whitegrid')
sns.despine()
import ibex
from ibex.sklearn import model_selection as pd_model_selection
from ibex.sklearn import decomposition as pd_decomposition
from ibex.sklearn import decomposition as pd_decomposition
from ibex.sklearn import ensemble as pd_ensemble
%pylab inline
ratings = pd.read_csv(
    'movielens_data/ml-100k/u.data', 
    sep='\t', 
    header=None, 
    names=['user_id', 'item_id', 'rating', 'timestamp'])
features = ['user_id', 'item_id']
ratings[features + ['rating']].head()
Explanation: Nonnegative Matrix Factorization In The Movielens Dataset
This example continues illustrating using pandas-munging capabilities in estimators building features that draw from several rows, this time using NMF (nonnegative matrix factorization). We will use a single table from the Movielens dataset (F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS)).
Loading The Data
In this example too, we'll only use the dataset table describing the ratings themselves. I.e., each row is an instance of a single rating given by a specific user to a specific movie.
End of explanation
UI = pd.pivot_table(ratings, values='rating', index='user_id', columns ='item_id')
UI
Explanation: Munging NMF With Pandas
In Simple Row-Aggregating Features In The Movielens Dataset we looked at direct attributes obtainable from the rankings: the average user and item ranking. Here we'll use Pandas to bring the dataset to a form where we can find latent factors through NMF.
First we pivot the table so that we have a UI matrix of the users as rows, the items as columns, and the ratings as the values:
End of explanation
d = pd_decomposition.NMF(n_components=20)
U = d.fit_transform(UI.fillna(0))
I = d.components_
Explanation: We now use NMF for decomposition, and then find the user latent factors in U and item latent factors in I:
End of explanation
U.head()
I.head()
Explanation: Note that the Ibex version of NMF sets the indexes and columns of the U and I appropriately.
End of explanation
ratings.head()
rating_comps = pd.merge(
    ratings,
    U,
    left_on='user_id',
    right_index=True,
    how='left')
rating_comps = pd.merge(
    rating_comps,
    I.T,
    left_on='item_id',
    right_index=True,
    how='left')
rating_comps.head()
Explanation: Pandas makes it easy to merge the user and item latent factors to the users and items, respectively.
End of explanation
rating_comps = pd.merge(
    rating_comps,
    ratings.groupby(ratings.user_id).size().to_frame().rename(columns={0: 'user_id_count'}),
    left_on='user_id',
    right_index=True,
    how='left')
rating_comps = pd.merge(
    rating_comps,
    ratings.groupby(ratings.item_id).size().to_frame().rename(columns={0: 'item_id_count'}),
    left_on='user_id',
    right_index=True,
    how='left')
prd_features = [c for c in rating_comps if 'comp_' in c] + ['user_id_count', 'item_id_count']
rating_comps.head()
Explanation: Let's merge to the results also the number of occurrences to of the users and items, respectively.
End of explanation
prd = pd_ensemble.RandomForestRegressor().fit(rating_comps[prd_features], ratings.rating)
prd.score(rating_comps[prd_features], ratings.rating)
Explanation: We now have a dataframe of latent variables. Let's build a random forest regressor, and use it on this dataframe.
End of explanation
prd.feature_importances_.to_frame().plot(kind='barh');
Explanation: Finally, let's check the feature importances.
End of explanation
class RatingsFactorizer(base.BaseEstimator, base.TransformerMixin, ibex.FrameMixin):
    def fit(self, X, y):
        X = pd.concat([X[['user_id', 'item_id']], y])
        X.columns = ['user_id', 'item_id', 'rating']
        self._user_id_count = X.groupby(X.user_id).size().to_frame().rename(columns={0: 'user_id_count'})
        self._item_id_count = X.groupby(X.item_id).size().to_frame().rename(columns={0: 'item_id_count'})
        UI = pd.pivot_table(ratings, values='rating', index='user_id', columns ='item_id')
        d = pd_decomposition.NMF(n_components=10)
        self._U = d.fit_transform(UI.fillna(0))
        self._I = d.components_
        return self
    def transform(self, X):
        rating_comps = pd.merge(
            X[['user_id', 'item_id']],
            self._U,
            left_on='user_id',
            right_index=True,
            how='left')
        rating_comps = pd.merge(
            rating_comps,
            self._I.T,
            left_on='item_id',
            right_index=True,
            how='left')
        rating_comps = pd.merge(
            rating_comps,
            self._user_id_count,
            left_on='user_id',
            right_index=True,
            how='left')
        rating_comps = pd.merge(
            rating_comps,
            self._item_id_count,
            left_on='user_id',
            right_index=True,
            how='left')
        prd_features = [c for c in rating_comps if 'comp_' in c] + ['user_id_count', 'item_id_count']
        return rating_comps[prd_features].fillna(0)
Explanation: Building A Pandas-Munging Estimator
We'll now build a Scikit-Learn / Pandas step doing the above.
End of explanation
prd = RatingsFactorizer() | pd_ensemble.RandomForestRegressor()
hist(
    pd_model_selection.cross_val_score(
        prd, 
        ratings[features], 
        ratings.rating,
        cv=20,
        n_jobs=-1),
    color='grey');
xlabel('CV Score')
ylabel('Num Occurrences')
figtext(
    0, 
    -0.1, 
    'Histogram of cross-validated scores');
Explanation: We can now use cross validation to assess this scheme.
End of explanation
<END_TASK>