<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 01 - Recommender System
"Given the values I do have, predict my missing values"
This is a kind of imputation / recommender system
Straight-forward matrix decomposition methods can help here
Step1:  Hooray, we did it
Now we need to figure out how well it actually did
Step3:  Make a recommender class, a la sklearn
Should have fit, predict methods
Step4:  Tune the one hyperparameter we have
Step5:  Optimal performance at k=7
Step6:  Surprise!
Try out this well-supported recommender package
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
from scipy.sparse.linalg import svds
from sklearn.metrics import mean_squared_error
import matplotlib.pyplot as plt
from jlab import load_test_data
X_train = pd.read_csv('MLchallenge2_training.csv')
X_test = load_test_data('test_in.csv')
X = (pd.concat([X_test, X_train], axis=0)
     .reset_index(drop=True)
     .fillna(0.0))
X_true = pd.read_csv('test_prediction.csv', names=['x', 'y', 'px', 'py', 'pz'],
                     header=None)
X.head()
X_true.head()
U, sigma, Vt = svds(X, k=30)
sigma = np.diag(sigma)
X_pred = pd.DataFrame(np.dot(np.dot(U, sigma), Vt), columns=X.columns, index=X.index)
X_pred.head()
Explanation: 01 - Recommender System
"Given the values I do have, predict my missing values"
This is a kind of imputation / recommender system
Straight-forward matrix decomposition methods can help here
End of explanation
def get_test_detector_plane(row):
    # Find location of nans, get the first one
    # Then divide by 6 (6 values per detector plane)
    plane = np.where(np.isnan(row.values))[0][0]/6
    return int(plane)
def get_vals_at_plane(row, plane):
    cols = [i + str(int(plane)) for i in ['x','y','px','py','pz']]
    return row[cols].values
def get_vals_at_eval_plane(X_test, X_pred):
    X = X_pred.copy()
    X['eval_plane'] = X_test.apply(get_test_detector_plane, axis=1)
    retvals = X.loc[X_test.index.values].apply(lambda x: get_vals_at_plane(x, x['eval_plane']), axis=1)
    return retvals
eval_planes = X_test.apply(get_test_detector_plane, axis=1)
get_vals_at_plane(X_test.loc[15], 7)
Explanation: Hooray, we did it
Now we need to figure out how well it actually did
End of explanation
import logging
from jlab import COLS
from sklearn.preprocessing import StandardScaler
class DetectorRecommender(object):
    
    def __init__(self, k=20):
        
        self.logger = logging.getLogger(__name__)
        self.k = k
        self.planes = 27
        self.kinematics = ["x", "y", "px", "py", "pz"]
        self.cols = COLS
        self.X_train = pd.DataFrame(columns=self.cols)
        self.X_test = pd.DataFrame(columns=self.cols)
        self.scaler = StandardScaler()
        
    def fit(self, df):
        SVD isn't really 'trained', but... 
        
        self.X_train = df.copy(deep=True)
        
    def predict(self, df):
        
        # Make a copy, index it from 0 to N
        self.logger.debug("Making a copy")
        self.X_test = df.copy(deep=True).reset_index(drop=True)
        
        # For each track, figure out which detector plane we'll evaluate
        self.logger.debug("Determining evaluation planes")
        eval_planes = self.X_test.apply(self.get_eval_detector_plane, axis=1)
        
        # Combine with the training set, shuffle it, and fill missing values
        self.logger.debug("Combining train and test sets for SVD")
        X = (pd.concat([self.X_test, self.X_train], axis=0)
             .reset_index(drop=True)
             .sample(replace=False, frac=1.0))
        
        # Fill with the mean values of each column
        self.logger.debug("Filling with mean values")
        X = X.fillna(X.mean())
        
        # Normalize the values
        self.logger.debug("Applying standardscaler")
        X_norm_values = self.scaler.fit_transform(X)
        X_norm = pd.DataFrame(X_norm_values, columns=X.columns, index=X.index)
        
        # Single-value Decomposition
        self.logger.debug("Making predictions")
        X_pred_norm = self.fit_predict_svds(X_norm)
        
        # Extract our test tracks
        X_pred_norm = X_pred_norm.loc[self.X_test.index, :].sort_index()
        
        # Un-normalize them
        X_pred_values = self.scaler.inverse_transform(X_pred_norm)
        X_pred = pd.DataFrame(X_pred_values, columns=X_pred_norm.columns,
                              index=X_pred_norm.index)
        self.logger.debug("De-normalized. Extracting pred values.")
        
        # Extract just the non-z kinematic values for the eval planes
        det_eval_values = self.extract_values_at_eval_planes(X_pred, eval_planes)
        
        return det_eval_values
    
    def fit_predict_svds(self, X):
        U, sigma, Vt = svds(X, k=self.k)
        sigma = np.diag(sigma)
        X_pred = pd.DataFrame(np.dot(np.dot(U, sigma), Vt),
                              columns=X.columns, index=X.index)
        return X_pred
        
    def extract_values_at_eval_planes(self, pred, planes):
        X = pred.copy(deep=True)
        X['eval_plane'] = planes
        retvals = X.apply(lambda x: self.get_vals_at_plane(x, x['eval_plane']), axis=1)
        retvals_df = pd.DataFrame(retvals.values.tolist(), columns=self.kinematics)
        return retvals_df
    
    def get_vals_at_plane(self, row, plane):
        cols = [i + str(int(plane)) for i in self.kinematics]
        return row[cols].values
    
    def get_eval_detector_plane(self, row):
        # Find location of nans, get the first one
        # Then divide by 6 (6 values per detector plane)
        plane = np.where(np.isnan(row.values))[0][0]/6
        return int(plane)
logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s - %(name)-12s - %(levelname)-8s - %(message)s')
predictor = DetectorRecommender()
predictor.fit(X_train)
X_pred = predictor.predict(X_test)
X_pred.head()
mean_squared_error(X_true, X_pred)
Explanation: Make a recommender class, a la sklearn
Should have fit, predict methods
End of explanation
for k in range(5,15):
    predictor = DetectorRecommender(k=k)
    predictor.fit(X_train)
    X_pred = predictor.predict(X_test)
    print(k, mean_squared_error(X_true, X_pred))
Explanation: Tune the one hyperparameter we have
End of explanation
predictor = DetectorRecommender(k=7)
predictor.fit(X_train)
X_pred = predictor.predict(X_test)
print(mean_squared_error(X_true, X_pred))
Explanation: Optimal performance at k=7
End of explanation
!pip install scikit-surprise
import surprise
X.melt()
X.index.name = "track_id"
X.head().reset_index().melt(id_vars=['track_id'])
X.sample(replace=False, frac=1.0)
Explanation: Surprise!
Try out this well-supported recommender package
End of explanation
<END_TASK>