<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <a href="https
Step1:  These look good, but I want the format of my ratings matrix to be one row per user and one column per movie. I'll pivot ratings_df to get that and call the new variable R.
Step2:  The last thing I need to do is de-mean the data (normalize by each users mean) and convert it from a dataframe to a numpy array.
Step3:  Singular Value Decomposition
Scipy and Numpy both have functions to do the singular value decomposition. I'm going to use the Scipy function svds because it let's me choose how many latent factors I want to use to approximate the original ratings matrix (instead of having to truncate it after).
Step4:  Making Predictions from the Decomposed Matrices
I now have everything I need to make movie ratings predictions for every user. I can do it all at once by following the math and matrix multiply $U$, $\Sigma$, and $V^{T}$ back to get the rank $k=50$ approximation of $R$.
I also need to add the user means back to get the actual star ratings prediction.
Step5:  Making Movie Recommendations
Finally, it's time. With the predictions matrix for every user, I can build a function to recommend movies for any user. All I need to do is return the movies with the highest predicted rating that the specified user hasn't already rated. Though I didn't use actually use any explicit movie content features (such as genre or title), I'll merge in that information to get a more complete picture of the recommendations.
I'll also return the list of movies the user has already rated, for the sake of comparison.
Step6:  So, how'd I do?
Step7:  Pretty cool! These look like pretty good recommendations. It's also good to see that, though I didn't actually use the genre of the movie as a feature, the truncated matrix factorization features "picked up" on the underlying tastes and preferences of the user. I've recommended some film-noirs, crime, drama, and war movies - all of which were genres of some of this user's top rated movies.
Visualizing true and predicted ratings matrix
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import os
import matplotlib.pyplot as plt
!wget http://files.grouplens.org/datasets/movielens/ml-100k.zip
!ls
!unzip ml-100k
folder = "ml-100k"
!wget http://files.grouplens.org/datasets/movielens/ml-1m.zip
!unzip ml-1m
!ls
folder = "ml-1m"
ratings_list = [
    [int(x) for x in i.strip().split("::")] for i in open(os.path.join(folder, "ratings.dat"), "r").readlines()
]
users_list = [i.strip().split("::") for i in open(os.path.join(folder, "users.dat"), "r").readlines()]
movies_list = [
    i.strip().split("::") for i in open(os.path.join(folder, "movies.dat"), "r", encoding="latin-1").readlines()
]
ratings_df = pd.DataFrame(ratings_list, columns=["UserID", "MovieID", "Rating", "Timestamp"], dtype=int)
movies_df = pd.DataFrame(movies_list, columns=["MovieID", "Title", "Genres"])
movies_df["MovieID"] = movies_df["MovieID"].apply(pd.to_numeric)
movies_df.head()
def get_movie_name(movies_df, movie_id_str):
    ndx = movies_df["MovieID"] == int(movie_id_str)
    name = movies_df["Title"][ndx].to_numpy()[0]
    return name
print(get_movie_name(movies_df, 1))
print(get_movie_name(movies_df, "527"))
def get_movie_genres(movies_df, movie_id_str):
    ndx = movies_df["MovieID"] == int(movie_id_str)
    name = movies_df["Genres"][ndx].to_numpy()[0]
    return name
print(get_movie_genres(movies_df, 1))
print(get_movie_genres(movies_df, "527"))
ratings_df.head()
Explanation: <a href="https://colab.research.google.com/github/Nirzu97/pyprobml/blob/matrix-factorization/notebooks/matrix_factorization_recommender.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
Matrix Factorization for Movie Lens Recommendations
This notebook is based on  code from Nick Becker 
https://github.com/beckernick/matrix_factorization_recommenders/blob/master/matrix_factorization_recommender.ipynb
Setting Up the Ratings Data
We read the data directly from MovieLens website, since they don't allow redistribution. We want to include the metadata (movie titles, etc), not just the ratings matrix.
End of explanation
R_df = ratings_df.pivot(index="UserID", columns="MovieID", values="Rating").fillna(0)
R_df.head()
Explanation: These look good, but I want the format of my ratings matrix to be one row per user and one column per movie. I'll pivot ratings_df to get that and call the new variable R.
End of explanation
R = R_df.to_numpy()
user_ratings_mean = np.mean(R, axis=1)
R_demeaned = R - user_ratings_mean.reshape(-1, 1)
print(R.shape)
print(np.count_nonzero(R))
Explanation: The last thing I need to do is de-mean the data (normalize by each users mean) and convert it from a dataframe to a numpy array.
End of explanation
from scipy.sparse.linalg import svds
U, sigma, Vt = svds(R_demeaned, k=50)
sigma = np.diag(sigma)
latents = [10, 20, 50]
errors = []
for latent_dim in latents:
    U, sigma, Vt = svds(R_demeaned, k=latent_dim)
    sigma = np.diag(sigma)
    Rpred = np.dot(np.dot(U, sigma), Vt) + user_ratings_mean.reshape(-1, 1)
    Rpred[Rpred < 0] = 0
    Rpred[Rpred > 5] = 5
    err = np.sqrt(np.sum(np.power(R - Rpred, 2)))
    errors.append(err)
print(errors)
Explanation: Singular Value Decomposition
Scipy and Numpy both have functions to do the singular value decomposition. I'm going to use the Scipy function svds because it let's me choose how many latent factors I want to use to approximate the original ratings matrix (instead of having to truncate it after).
End of explanation
all_user_predicted_ratings = np.dot(np.dot(U, sigma), Vt) + user_ratings_mean.reshape(-1, 1)
Explanation: Making Predictions from the Decomposed Matrices
I now have everything I need to make movie ratings predictions for every user. I can do it all at once by following the math and matrix multiply $U$, $\Sigma$, and $V^{T}$ back to get the rank $k=50$ approximation of $R$.
I also need to add the user means back to get the actual star ratings prediction.
End of explanation
preds_df = pd.DataFrame(all_user_predicted_ratings, columns=R_df.columns)
preds_df.head()
def recommend_movies(preds_df, userID, movies_df, original_ratings_df, num_recommendations=5):
    # Get and sort the user's predictions
    user_row_number = userID - 1  # UserID starts at 1, not 0
    sorted_user_predictions = preds_df.iloc[user_row_number].sort_values(ascending=False)  # UserID starts at 1
    # Get the user's data and merge in the movie information.
    user_data = original_ratings_df[original_ratings_df.UserID == (userID)]
    user_full = user_data.merge(movies_df, how="left", left_on="MovieID", right_on="MovieID").sort_values(
        ["Rating"], ascending=False
    )
    print("User {0} has already rated {1} movies.".format(userID, user_full.shape[0]))
    print("Recommending highest {0} predicted ratings movies not already rated.".format(num_recommendations))
    # Recommend the highest predicted rating movies that the user hasn't seen yet.
    recommendations = (
        movies_df[~movies_df["MovieID"].isin(user_full["MovieID"])]
        .merge(pd.DataFrame(sorted_user_predictions).reset_index(), how="left", left_on="MovieID", right_on="MovieID")
        .rename(columns={user_row_number: "Predictions"})
        .sort_values("Predictions", ascending=False)
        .iloc[:num_recommendations, :-1]
    )
    return user_full, recommendations
already_rated, predictions = recommend_movies(preds_df, 837, movies_df, ratings_df, 10)
Explanation: Making Movie Recommendations
Finally, it's time. With the predictions matrix for every user, I can build a function to recommend movies for any user. All I need to do is return the movies with the highest predicted rating that the specified user hasn't already rated. Though I didn't use actually use any explicit movie content features (such as genre or title), I'll merge in that information to get a more complete picture of the recommendations.
I'll also return the list of movies the user has already rated, for the sake of comparison.
End of explanation
already_rated.head(10)
df = already_rated[["MovieID", "Title", "Genres"]].copy()
df.head(10)
predictions
Explanation: So, how'd I do?
End of explanation
Rpred = all_user_predicted_ratings
Rpred[Rpred < 0] = 0
Rpred[Rpred > 5] = 5
print(np.linalg.norm(R - Rpred, ord="fro"))
print(np.sqrt(np.sum(np.power(R - Rpred, 2))))
import matplotlib.pyplot as plt
nusers = 20
nitems = 20
plt.figure(figsize=(10, 10))
plt.imshow(R[:nusers, :nitems], cmap="jet")
plt.xlabel("item")
plt.ylabel("user")
plt.title("True ratings")
plt.colorbar()
plt.figure(figsize=(10, 10))
plt.imshow(Rpred[:nusers, :nitems], cmap="jet")
plt.xlabel("item")
plt.ylabel("user")
plt.title("Predcted ratings")
plt.colorbar()
Explanation: Pretty cool! These look like pretty good recommendations. It's also good to see that, though I didn't actually use the genre of the movie as a feature, the truncated matrix factorization features "picked up" on the underlying tastes and preferences of the user. I've recommended some film-noirs, crime, drama, and war movies - all of which were genres of some of this user's top rated movies.
Visualizing true and predicted ratings matrix
End of explanation
<END_TASK>